Image classification method, device and equipment for simulating radial field of view, and medium

Abstract

The application relates to the field of artificial intelligence, and discloses an image classification method simulating radial visual field, which comprises the following steps: performing image feature extraction on a noise-reduced image to be classified by using an encoding layer of a pre-trained image recognition model to obtain an initial image feature matrix; performing pooling operation on the initial image feature matrix in sequence by using a preset number of pooling layers in the image recognition model to obtain a minimum pooling feature matrix, wherein the preset number of pooling layers are arranged in the order of decreasing size of convolution kernels of the pooling layers; performing hole convolution on the minimum pooling feature matrix according to a preset receptive field coefficient to obtain a hole feature matrix; and classifying the image to be classified according to the hole feature matrix by using a decoding layer in the image recognition model. The application further discloses an image classification device simulating radial visual field, an electronic device and a computer readable storage medium. The application can improve the accuracy of medical image classification.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more particularly to an image classification method, apparatus, electronic device, and computer-readable storage medium that simulates radial field of view. Background Technology

[0002] In the medical field, the classification and processing of medical images is often involved, such as classifying target organs or lesions in medical images.

[0003] Currently, in the classification and processing of medical images, a common approach is to use convolutional neural network models to extract image features from medical images, then perform dimensionality reduction on the convolutional image features using traditional pooling operations, and finally use an image classification model to classify the medical images based on the dimensionality-reduced image features.

[0004] Traditional pooling operations include Max-pooling and Mean-pooling. While these two operations offer fast computation, they lack the hierarchical perception of image features. This is because human observation of an image involves a divergent field of vision. Unlike Max-pooling, which focuses solely on prominent pixels, or Mean-pooling, which averages prominent and non-prominent pixels to observe the entire image, human observation follows a radial field of vision. While focusing on a prominent pixel, attention gradually expands to surrounding pixels, although this focus diminishes with distance. Therefore, image features processed by pooling in image classification suffer information loss, thus requiring improvement in classification accuracy. Summary of the Invention

[0005] This invention provides an image classification method, apparatus, electronic device, and computer-readable storage medium that simulates radial field of view, with the main objective of improving the accuracy of medical image classification.

[0006] To achieve the above objectives, the present invention provides an image classification method simulating radial field of view, comprising:

[0007] Obtain the image to be classified, and perform noise reduction on the image to be classified;

[0008] Using the encoding layer of a pre-trained image recognition model, image feature extraction is performed on the denoised image to be classified to obtain the initial image feature matrix corresponding to the image to be classified.

[0009] Using a preset number of pooling layers in the image recognition model, the initial image feature matrix is ​​sequentially pooled to obtain a minimum pooling feature matrix. The preset number of pooling layers are arranged in descending order of their convolution kernels.

[0010] Based on the preset receptive field coefficients, dilated convolution is performed on the minimum pooling feature matrix to obtain the dilated feature matrix;

[0011] The image to be classified is classified using the decoding layer in the image recognition model based on the hole feature matrix, and the image classification result is obtained.

[0012] Optionally, the step of using the encoding layer of a pre-trained image recognition model to perform image feature extraction on the denoised image to be classified includes:

[0013] The image to be classified is divided into blocks using the encoding layer of the pre-trained image recognition model to obtain a block sequence;

[0014] The normalized image attention weight of each patch in the patch sequence is calculated sequentially, and the weighted image feature vector corresponding to each patch is generated using the normalized image attention weight;

[0015] By concatenating all weighted image feature vectors, the initial image feature matrix corresponding to the image to be classified is obtained.

[0016] Optionally, the step of sequentially calculating the normalized image attention weight for each patch in the patch sequence includes:

[0017] Image features are extracted from each patch in the patch sequence to obtain a patch feature matrix for each patch;

[0018] The patch feature matrix is ​​rearranged according to the size of the pixel values ​​in each column of the patch feature matrix;

[0019] Using the fully connected layer of the pre-trained image recognition model, the rearranged patch feature matrix is ​​converted into a one-dimensional patch feature vector;

[0020] The one-dimensional patch feature vector is activated and normalized using a preset activation function to obtain the normalized image attention weight for each patch.

[0021] Optionally, the step of activating and normalizing the one-dimensional patch feature vector using a preset activation function to obtain the normalized image attention weight for each patch includes:

[0022] The sum of the pixels in the one-dimensional feature vector of each of the aforementioned patches is used as the feature weight coefficient of the corresponding patch.

[0023] Nonlinear activation is applied to the feature weight coefficients of each of the aforementioned patches;

[0024] The feature weight coefficients after nonlinear activation are linearly normalized to obtain the normalized image attention weights corresponding to each patch.

[0025] Optionally, generating the weighted image feature vector corresponding to each patch using the normalized image attention weights includes:

[0026] The normalized image attention weights are multiplied by the rearranged patch feature matrix of the corresponding patch to obtain the weighted image feature matrix.

[0027] The weighted image feature vector is obtained by summing the values ​​of each column in the weighted image feature matrix.

[0028] Optionally, the step of performing dilated convolution on the minimum pooling feature matrix according to preset receptive field coefficients to obtain a dilated feature matrix includes:

[0029] Obtain the minimum kernel size of the preset number of pooling layers;

[0030] The minimum convolution kernel is zero-padded according to the preset receptive field coefficients to obtain an expanded convolution kernel;

[0031] The dilated convolution kernel is used to perform a convolution operation on the minimum pooling feature matrix to obtain the dilated feature matrix corresponding to the image to be classified.

[0032] Optionally, the noise reduction operation on the image to be classified includes:

[0033] Select one pixel in the image to be classified as the target point in sequence, and identify the average gray level difference between the target point and its surrounding neighboring points;

[0034] When the average gray-level difference is greater than or equal to a preset noise reduction threshold, the target point is smoothed.

[0035] When the average gray level difference is less than the preset noise reduction threshold, the pixel gray level of the target point remains unchanged;

[0036] Determine whether all pixels in the image to be classified have been selected;

[0037] If any pixel in the image to be classified is not selected, return to the steps described above of sequentially selecting one pixel in the image to be classified as the target point.

[0038] When all pixels in the image to be classified are selected, the denoised image to be classified is obtained.

[0039] To address the above problems, the present invention also provides an image classification device simulating radial field of view, the device comprising:

[0040] The image denoising module is used to acquire the image to be classified and perform denoising operations on the image to be classified.

[0041] The initial feature extraction module is used to perform image feature extraction on the denoised image to be classified using the encoding layer of a pre-trained image recognition model, so as to obtain the initial image feature matrix corresponding to the image to be classified.

[0042] The radial pooling module is used to sequentially perform pooling operations on the initial image feature matrix using a preset number of pooling layers in the image recognition model to obtain a minimum pooled feature matrix. The preset number of pooling layers are arranged in descending order of the convolution kernel size.

[0043] The dilated convolution module is used to perform dilated convolution on the minimum pooling feature matrix according to the preset receptive field coefficients to obtain the dilated feature matrix.

[0044] The feature classification module is used to classify the image to be classified based on the hole feature matrix using the decoding layer in the image recognition model, and obtain the image classification result.

[0045] To address the above problems, the present invention also provides an electronic device, the electronic device comprising:

[0046] Memory, storing at least one computer program; and

[0047] The processor executes the program stored in the memory to implement the image classification method for simulating radial field of view described above.

[0048] To address the aforementioned problems, the present invention also provides a computer-readable storage medium storing at least one computer program, which is executed by a processor in an electronic device to implement the image classification method for simulating radial field of view described above.

[0049] This invention simulates the radial field of vision of the human eye by performing pooling operations on the initial image feature matrix of the image to be classified, with the convolution kernel size decreasing. This gradually focuses on the prominent points of the image to be classified. Then, dilated convolution is performed on the minimum pooling feature matrix to prevent the loss of image feature data during the pooling process. Finally, the dilated feature matrix is ​​used to classify the image to be classified. Compared with traditional max-pooling and mean-pooling operations, the pooling operation provided in this application is more in line with the human eye's habit of recognizing image targets and can improve the accuracy of medical image classification. Attached Figure Description

[0050] Figure 1 A schematic flowchart of an image classification method simulating radial field of view provided in an embodiment of the present invention;

[0051] Figure 2 A detailed implementation flowchart of one step of the image classification method simulating radial field of view provided in an embodiment of the present invention is shown below;

[0052] Figure 3 A detailed implementation flowchart of one step of the image classification method simulating radial field of view provided in an embodiment of the present invention is shown below;

[0053] Figure 4 A functional block diagram of an image classification device simulating radial field of view provided in an embodiment of the present invention;

[0054] Figure 5 This is a schematic diagram of the structure of an electronic device that implements the simulated radial field of view image classification method according to an embodiment of the present invention.

[0055] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0056] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0057] This application provides an image classification method simulating radial field of view. The executing entity of the image classification method simulating radial field of view includes, but is not limited to, at least one of electronic devices, such as a server or a terminal, that can be configured to execute the method provided in this application. In other words, the image classification method simulating radial field of view can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0058] Reference Figure 1 The diagram shown is a flowchart illustrating an image classification method simulating radial field of view according to an embodiment of the present invention. In this embodiment, the image classification method simulating radial field of view includes:

[0059] S1. Obtain the image to be classified and perform noise reduction on the image to be classified;

[0060] In this embodiment of the invention, a simulated radial field of view image classification method is described using medical images as an example. The image to be classified may be a patient-authorized medical record image, such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), US (ultrasonic), X-ray images, electroencephalograms, and images generated by medical instruments using optical imaging.

[0061] In this embodiment of the invention, a preset half-means filtering algorithm can be used to perform noise reduction on the image to be classified.

[0062] The mean filtering algorithm refers to a method that selects a target pixel and its eight neighboring pixels (centered on the target pixel) in an image using a filtering template, and then replaces the target pixel with the average value of all pixels in the filtering template. In this embodiment of the invention, the semi-mean filtering algorithm selectively smooths each pixel in the image to be classified, applying the mean filtering algorithm only to pixels with small average gray-level differences.

[0063] Specifically, the noise reduction operation on the image to be classified includes:

[0064] Select one pixel in the image to be classified as the target point in sequence, and identify the average gray level difference between the target point and its surrounding neighboring points;

[0065] When the average gray level difference is greater than or equal to a preset noise reduction threshold, the target point is smoothed; when the average gray level difference is less than the preset noise reduction threshold, the pixel gray level of the target point remains unchanged.

[0066] Determine whether all pixels in the image to be classified have been selected;

[0067] If any pixel in the image to be classified is not selected, return to the steps described above of sequentially selecting one pixel in the image to be classified as the target point.

[0068] When all pixels in the image to be classified are selected, the denoised image to be classified is obtained.

[0069] In this embodiment of the invention, the image to be classified is affected by interference factors such as differences in the shooting instrument, ambient lighting, and individual differences in the patient's condition. The image to be classified may have difficulty in delineating the boundary of the target object and a lot of noise in the image. Therefore, this embodiment of the invention needs to use a semi-means filtering algorithm to smooth the pixels in the image to be classified in order to remove the noise in the image to be classified and ensure that the edge parts of each tissue in the image to be classified are not smoothed.

[0070] S2. Using the encoding layer of a pre-trained image recognition model, perform image feature extraction on the denoised image to be classified to obtain the initial image feature matrix corresponding to the image to be classified.

[0071] In this embodiment of the invention, the pre-trained image recognition model can be a ViT (VisionTransformer) network model.

[0072] For details, see S2, which includes:

[0073] S21. Using the encoding layer of the pre-trained image recognition model, the image to be classified is divided into blocks to obtain a block sequence;

[0074] S22. Calculate the normalized image attention weight for each patch in the patch sequence in sequence, and use the normalized image attention weight to generate the weighted image feature vector corresponding to each patch;

[0075] S23. Concatenate all weighted image feature vectors to obtain the initial image feature matrix corresponding to the image to be classified.

[0076] In this embodiment of the invention, the encoding layer adopts a Transaformer structure and cannot directly process images. Therefore, the image to be classified needs to be segmented into blocks beforehand.

[0077] For example, if the image to be classified, x ∈ H×W×C, is divided into P×P×C patches, then the image to be classified will be divided into N = HW / P² patches, and the dimensions of all patches can be written as N×P×P×C. Then, each patch is flattened, and the corresponding data dimension can be written as N×(P²×C). Here, N can be understood as the length of the sequence input to the Transformer, C is the number of channels in the input image, and P is the size of each patch.

[0078] It is understandable that, since each of the image patches contains different pixel features, the attention weights extracted by the coding layer for each image patch are also different. By normalizing the image attention weights of each image patch, the differences between large weights can be avoided from being amplified too quickly by the coding layer during convolution calculation.

[0079] For details, please refer to Figure 3 As shown, the step of sequentially calculating the normalized image attention weight for each patch in the patch sequence includes:

[0080] S221. Extract image features from each patch in the patch sequence to obtain a patch feature matrix for each patch;

[0081] S222. Rearrange the patch feature matrix according to the size of the pixel values ​​in each column of the patch feature matrix;

[0082] S223. Using the fully connected layer of the pre-trained image recognition model, the rearranged patch feature matrix is ​​converted into a one-dimensional patch feature vector;

[0083] S224. The one-dimensional patch feature vector is activated and normalized using a preset activation function to obtain the normalized image attention weight for each patch.

[0084] For example, the feature matrix of a certain tile is an (n*m)*768 matrix with n*m rows and 768 columns. The columns of the feature matrix are rearranged from largest to smallest to facilitate subsequent calculations. The rearranged feature matrix is ​​still (n*m)*768.

[0085] Preferably, the fully connected layer can be located at the end of the pre-trained image recognition model, and the fully connected layer can convert the patch feature matrix from a two-dimensional matrix into a one-dimensional patch feature vector.

[0086] In this embodiment of the invention, the preset activation function may be a common nonlinear activation function in deep learning, including but not limited to tanh activation function, sigmoid activation function, and ReLU activation function.

[0087] In detail, the step of activating and normalizing the one-dimensional patch feature vector using a preset activation function to obtain the normalized image attention weight for each patch includes:

[0088] The sum of the pixels in the one-dimensional feature vector of each of the aforementioned patches is used as the feature weight coefficient of the corresponding patch.

[0089] Nonlinear activation is applied to the feature weight coefficients of each of the aforementioned patches;

[0090] The feature weight coefficients after nonlinear activation are linearly normalized to obtain the normalized image attention weights corresponding to each patch.

[0091] Further, the step of generating a weighted image feature vector corresponding to each patch using the normalized image attention weights includes:

[0092] The normalized image attention weights are multiplied by the rearranged patch feature matrix of the corresponding patch to obtain the weighted image feature matrix.

[0093] The weighted image feature vector is obtained by summing the values ​​of each column in the weighted image feature matrix.

[0094] In this embodiment of the invention, by calculating the normalized image attention weight of the smallest unit of the image to be classified, the feature scale difference of each patch is preserved on the one hand, and the normalization of the image attention weight can reduce the further expansion of the feature scale difference of the patches in subsequent feature extraction.

[0095] S3. Using a preset number of pooling layers in the image recognition model, the initial image feature matrix is ​​sequentially pooled to obtain a minimum pooling feature matrix. The preset number of pooling layers are arranged in descending order of the convolution kernel size.

[0096] For example, the initial image feature matrix is ​​a 224*224 feature matrix. The initial image feature matrix is ​​divided into 14*14 patches, each patch having a pixel size of 16*16. Assuming the image recognition model has three pooling layers with corresponding convolutional kernel sizes of 3*3, 5*5, and 7*7, the initial image feature matrix is ​​pooled in the following order:

[0097] 1. A 7*7 pooling layer is used to perform a pooling operation on the initial 224*224 image feature matrix, and the pooled feature matrix is ​​32*32.

[0098] 2. Then, a 5*5 pooling layer is used to perform a pooling operation on the 32*32 feature matrix to obtain a 10*10 feature matrix.

[0099] 3. Finally, a 3x3 pooling layer is used to perform a pooling operation on the 10x10 feature matrix.

[0100] We obtain an 8*8 minimum pooling feature matrix.

[0101] In this embodiment of the invention, the initial image feature matrix is ​​progressively convolved by the aforementioned preset number of pooling layers. Although this increases the computation process, it can simulate the process by which the human eye gradually focuses on the key points of an image, making the image features after pooling more obvious.

[0102] S4. Based on the preset receptive field coefficients, perform dilated convolution on the minimum pooling feature matrix to obtain the dilated feature matrix.

[0103] It is understood that the receptive field refers to the size of the pixel range of the input image corresponding to one pixel in the convolution result. For example, in this embodiment of the invention, the convolution kernel of the minimum pooling layer is 3×3, that is, the value in the 3×3 range of the initial image feature matrix corresponds to a value in the minimum pooling feature matrix.

[0104] In this embodiment of the invention, since the initial image feature matrix has a relatively rich amount of information, after the pooling operation is performed by the preset number of pooling layers, the information content of the corresponding minimum pooling feature matrix may be lost. Therefore, dilated convolution is performed on the minimum pooling feature matrix to increase the information richness of the minimum pooling feature matrix and reduce the loss of information discarded by the features of the image to be classified during the pooling operation.

[0105] Specifically, the step of performing dilated convolution on the minimum pooling feature matrix according to preset receptive field coefficients to obtain a dilated feature matrix includes:

[0106] Obtain the minimum kernel size of the preset number of pooling layers;

[0107] The minimum convolution kernel is zero-padded according to the preset receptive field coefficients to obtain an expanded convolution kernel;

[0108] The dilated convolution kernel is used to perform a convolution operation on the minimum pooling feature matrix to obtain the dilated feature matrix corresponding to the image to be classified.

[0109] In this embodiment of the invention, the dilation rate parameter in the dilated convolution is configured, wherein the dilation rate defaults to 1, i.e., the original convolution kernel. In this embodiment, the dilated convolution kernel is obtained by padding the smallest convolution kernel with "0" values.

[0110] S5. Using the decoding layer in the image recognition model, classify the image to be classified according to the hole feature matrix to obtain the image classification result.

[0111] In this embodiment of the invention, the decoder in the image recognition model can adopt an MLPHead structure. The decoding layer in the image recognition model converts the hole feature matrix into a one-dimensional feature vector. Then, a preset activation function is used to calculate the relative probability value between the one-dimensional feature vector and the preset classification label. The classification label with the relative probability value greater than or equal to the preset threshold is selected as the classification result of the image to be classified.

[0112] In this embodiment of the invention, the preset classification labels can be set according to actual application needs. For example, in the application of organ identification and classification, the preset classification labels can be organ labels such as adrenal gland, liver and gallbladder, and lungs.

[0113] This invention simulates the radial field of vision of the human eye by performing pooling operations on the initial image feature matrix of the image to be classified, with the convolution kernel size decreasing. This gradually focuses on the prominent points of the image to be classified. Then, dilated convolution is performed on the minimum pooling feature matrix to prevent the loss of image feature data during the pooling process. Finally, the dilated feature matrix is ​​used to classify the image to be classified. Compared with traditional max-pooling and mean-pooling operations, the pooling operation provided in this application is more in line with the human eye's habit of recognizing image targets and can improve the accuracy of medical image classification.

[0114] like Figure 4 The diagram shown is a functional block diagram of an image classification device simulating radial field of view provided in an embodiment of the present invention.

[0115] The image classification device 100 simulating radial field of view described in this invention can be installed in an electronic device. Depending on the functions implemented, the image classification device 100 simulating radial field of view includes: an image denoising module 101, an initial feature extraction module 102, a radial pooling module 103, a dilated convolution module 104, and a feature classification module 105. The module described in this invention can also be referred to as a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, and which are stored in the memory of the electronic device.

[0116] In this embodiment, the functions of each module / unit are as follows:

[0117] The image noise reduction module 101 is used to acquire the image to be classified and perform noise reduction operation on the image to be classified.

[0118] The initial feature extraction module 102 is used to perform image feature extraction on the denoised image to be classified using the encoding layer of a pre-trained image recognition model, so as to obtain the initial image feature matrix corresponding to the image to be classified.

[0119] The radial pooling module 103 is used to perform pooling operations on the initial image feature matrix sequentially using a preset number of pooling layers in the image recognition model to obtain a minimum pooling feature matrix. The preset number of pooling layers are arranged in descending order of the convolution kernel size.

[0120] The dilated convolution module 104 is used to perform dilated convolution on the minimum pooling feature matrix according to the preset receptive field coefficients to obtain a dilated feature matrix.

[0121] The feature classification module 105 is used to classify the image to be classified based on the hole feature matrix using the decoding layer in the image recognition model, and obtain the image classification result.

[0122] In detail, the specific implementation methods of each module of the image classification device 100 simulating radial field of view are as follows:

[0123] Step 1: Obtain the image to be classified and perform noise reduction on the image to be classified;

[0124] In this embodiment of the invention, a simulated radial field of view image classification method is described using medical images as an example. The image to be classified may be a patient-authorized medical record image, such as CT (Computed Tomography), MRI (Magnetic Resonance Imaging), US (ultrasonic), X-ray images, electroencephalograms, and images generated by medical instruments using optical imaging.

[0125] In this embodiment of the invention, a preset half-means filtering algorithm can be used to perform noise reduction on the image to be classified.

[0126] The mean filtering algorithm refers to a method that selects a target pixel and its eight neighboring pixels (centered on the target pixel) in an image using a filtering template, and then replaces the target pixel with the average value of all pixels in the filtering template. In this embodiment of the invention, the semi-mean filtering algorithm selectively smooths each pixel in the image to be classified, applying the mean filtering algorithm only to pixels with small average gray-level differences.

[0127] Specifically, the noise reduction operation on the image to be classified includes:

[0128] Select one pixel in the image to be classified as the target point in sequence, and identify the average gray level difference between the target point and its surrounding neighboring points;

[0129] When the average gray level difference is greater than or equal to a preset noise reduction threshold, the target point is smoothed; when the average gray level difference is less than the preset noise reduction threshold, the pixel gray level of the target point remains unchanged.

[0130] Determine whether all pixels in the image to be classified have been selected;

[0131] If any pixel in the image to be classified is not selected, return to the steps described above of sequentially selecting one pixel in the image to be classified as the target point.

[0132] When all pixels in the image to be classified are selected, the denoised image to be classified is obtained.

[0133] In this embodiment of the invention, the image to be classified is affected by interference factors such as differences in the shooting instrument, ambient lighting, and individual differences in the patient's condition. The image to be classified may have difficulty in delineating the boundary of the target object and a lot of noise in the image. Therefore, this embodiment of the invention needs to use a semi-means filtering algorithm to smooth the pixels in the image to be classified in order to remove the noise in the image to be classified and ensure that the edge parts of each tissue in the image to be classified are not smoothed.

[0134] Step 2: Using the encoding layer of the pre-trained image recognition model, perform image feature extraction on the denoised image to be classified to obtain the initial image feature matrix corresponding to the image to be classified.

[0135] In this embodiment of the invention, the pre-trained image recognition model can be a ViT (VisionTransformer) network model.

[0136] In detail, the step of using the encoding layer of a pre-trained image recognition model to perform image feature extraction on the denoised image to be classified, to obtain the initial image feature matrix corresponding to the image to be classified, includes:

[0137] The image to be classified is divided into blocks using the encoding layer of the pre-trained image recognition model to obtain a block sequence;

[0138] The normalized image attention weight of each patch in the patch sequence is calculated sequentially, and the weighted image feature vector corresponding to each patch is generated using the normalized image attention weight;

[0139] By concatenating all weighted image feature vectors, the initial image feature matrix corresponding to the image to be classified is obtained.

[0140] In this embodiment of the invention, the encoding layer adopts a Transaformer structure and cannot directly process images. Therefore, the image to be classified needs to be segmented into blocks beforehand.

[0141] For example, if the image to be classified, x ∈ H×W×C, is divided into P×P×C patches, then the image to be classified will be divided into N = HW / P² patches, and the dimensions of all patches can be written as N×P×P×C. Then, each patch is flattened, and the corresponding data dimension can be written as N×(P²×C). Here, N can be understood as the length of the sequence input to the Transformer, C is the number of channels in the input image, and P is the size of each patch.

[0142] It is understandable that, since each of the image patches contains different pixel features, the attention weights extracted by the coding layer for each image patch are also different. By normalizing the image attention weights of each image patch, the differences between large weights can be avoided from being amplified too quickly by the coding layer during convolution calculation.

[0143] Specifically, the step of sequentially calculating the normalized image attention weight for each patch in the patch sequence includes:

[0144] Image features are extracted from each patch in the patch sequence to obtain a patch feature matrix for each patch;

[0145] The patch feature matrix is ​​rearranged according to the size of the pixel values ​​in each column of the patch feature matrix;

[0146] Using the fully connected layer of the pre-trained image recognition model, the rearranged patch feature matrix is ​​converted into a one-dimensional patch feature vector;

[0147] The one-dimensional patch feature vector is activated and normalized using a preset activation function to obtain the normalized image attention weight for each patch.

[0148] For example, the feature matrix of a certain tile is an (n*m)*768 matrix with n*m rows and 768 columns. The columns of the feature matrix are rearranged from largest to smallest to facilitate subsequent calculations. The rearranged feature matrix is ​​still (n*m)*768.

[0149] Preferably, the fully connected layer can be located at the end of the pre-trained image recognition model, and the fully connected layer can convert the patch feature matrix from a two-dimensional matrix into a one-dimensional patch feature vector.

[0150] In this embodiment of the invention, the preset activation function may be a common nonlinear activation function in deep learning, including but not limited to tanh activation function, sigmoid activation function, and ReLU activation function.

[0151] In detail, the step of activating and normalizing the one-dimensional patch feature vector using a preset activation function to obtain the normalized image attention weight for each patch includes:

[0152] The sum of the pixels in the one-dimensional feature vector of each of the aforementioned patches is used as the feature weight coefficient of the corresponding patch.

[0153] Nonlinear activation is applied to the feature weight coefficients of each of the aforementioned patches;

[0154] The feature weight coefficients after nonlinear activation are linearly normalized to obtain the normalized image attention weights corresponding to each patch.

[0155] Further, the step of generating a weighted image feature vector corresponding to each patch using the normalized image attention weights includes:

[0156] The normalized image attention weights are multiplied by the rearranged patch feature matrix of the corresponding patch to obtain the weighted image feature matrix.

[0157] The weighted image feature vector is obtained by summing the values ​​of each column in the weighted image feature matrix.

[0158] In this embodiment of the invention, by calculating the normalized image attention weight of the smallest unit of the image to be classified, the feature scale difference of each patch is preserved on the one hand, and the normalization of the image attention weight can reduce the further expansion of the feature scale difference of the patches in subsequent feature extraction.

[0159] Step 3: Using a preset number of pooling layers in the image recognition model, perform pooling operations on the initial image feature matrix sequentially to obtain the minimum pooling feature matrix. The preset number of pooling layers are arranged in descending order of the convolution kernel size.

[0160] For example, the initial image feature matrix is ​​a 224*224 feature matrix. The initial image feature matrix is ​​divided into 14*14 patches, each patch having a pixel size of 16*16. Assuming the image recognition model has three pooling layers with corresponding convolutional kernel sizes of 3*3, 5*5, and 7*7, the initial image feature matrix is ​​pooled in the following order:

[0161] 1. A 7*7 pooling layer is used to perform a pooling operation on the initial 224*224 image feature matrix, and the pooled feature matrix is ​​32*32.

[0162] 2. Then, a 5*5 pooling layer is used to perform a pooling operation on the 32*32 feature matrix to obtain a 10*10 feature matrix.

[0163] 3. Finally, a 3x3 pooling layer is used to perform a pooling operation on the 10x10 feature matrix.

[0164] We obtain an 8*8 minimum pooling feature matrix.

[0165] In this embodiment of the invention, the initial image feature matrix is ​​progressively convolved by the aforementioned preset number of pooling layers. Although this increases the computation process, it can simulate the process by which the human eye gradually focuses on the key points of an image, making the image features after pooling more obvious.

[0166] Step 4: Perform dilated convolution on the minimum pooling feature matrix according to the preset receptive field coefficients to obtain the dilated feature matrix;

[0167] It is understood that the receptive field refers to the size of the pixel range of the input image corresponding to one pixel in the convolution result. For example, in this embodiment of the invention, the convolution kernel of the minimum pooling layer is 3×3, that is, the value in the 3×3 range of the initial image feature matrix corresponds to a value in the minimum pooling feature matrix.

[0168] In this embodiment of the invention, since the initial image feature matrix has a relatively rich amount of information, after the pooling operation is performed by the preset number of pooling layers, the information content of the corresponding minimum pooling feature matrix may be lost. Therefore, dilated convolution is performed on the minimum pooling feature matrix to increase the information richness of the minimum pooling feature matrix and reduce the loss of information discarded by the features of the image to be classified during the pooling operation.

[0169] Specifically, the step of performing dilated convolution on the minimum pooling feature matrix according to preset receptive field coefficients to obtain a dilated feature matrix includes:

[0170] Obtain the minimum kernel size of the preset number of pooling layers;

[0171] The minimum convolution kernel is zero-padded according to the preset receptive field coefficients to obtain an expanded convolution kernel;

[0172] The dilated convolution kernel is used to perform a convolution operation on the minimum pooling feature matrix to obtain the dilated feature matrix corresponding to the image to be classified.

[0173] In this embodiment of the invention, the dilation rate parameter in the dilated convolution is configured, wherein the dilation rate defaults to 1, i.e., the original convolution kernel. In this embodiment, the dilated convolution kernel is obtained by padding the smallest convolution kernel with "0" values.

[0174] Step 5: Using the decoding layer in the image recognition model, classify the image to be classified according to the hole feature matrix to obtain the image classification result.

[0175] In this embodiment of the invention, the decoder in the image recognition model can adopt an MLPHead structure. The decoding layer in the image recognition model converts the hole feature matrix into a one-dimensional feature vector. Then, a preset activation function is used to calculate the relative probability value between the one-dimensional feature vector and the preset classification label. The classification label with the relative probability value greater than or equal to the preset threshold is selected as the classification result of the image to be classified.

[0176] In this embodiment of the invention, the preset classification labels can be set according to actual application needs. For example, in the application of organ identification and classification, the preset classification labels can be organ labels such as adrenal gland, liver and gallbladder, and lungs.

[0177] This invention simulates the radial field of vision of the human eye by performing pooling operations on the initial image feature matrix of the image to be classified, with the convolution kernel size decreasing. This gradually focuses on the prominent points of the image to be classified. Then, dilated convolution is performed on the minimum pooling feature matrix to prevent the loss of image feature data during the pooling process. Finally, the dilated feature matrix is ​​used to classify the image to be classified. Compared with traditional max-pooling and mean-pooling operations, the image classification device that simulates radial field of vision provided in this application is more in line with the human eye's habit of recognizing image targets and can improve the accuracy of medical image classification.

[0178] like Figure 5The diagram shown is a schematic representation of an electronic device that implements an image classification method simulating radial field of view, according to an embodiment of the present invention.

[0179] The electronic device 1 may include a processor 10, a memory 11 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as an image classification program that simulates radial field of view.

[0180] The memory 11 includes at least one type of readable storage medium, such as flash memory, portable hard drive, multimedia card, card-type memory (e.g., SD or DX memory), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a portable hard drive. In other embodiments, the memory 11 can be an external storage device of the electronic device 1, such as a plug-in portable hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device 1. Furthermore, the memory 11 can include both internal and external storage units of the electronic device 1. The memory 11 can be used not only to store application software and various types of data installed on the electronic device 1, such as code for image classification programs simulating radial field of view, but also to temporarily store data that has been output or will be output.

[0181] In some embodiments, the processor 10 may be composed of integrated circuits, such as a single packaged integrated circuit or multiple integrated circuits with the same or different functions, including combinations of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, connecting various components of the entire electronic device through various interfaces and lines. It executes programs or modules stored in the memory 11 (e.g., image classification programs simulating radial field of view) and calls data stored in the memory 11 to perform various functions of the electronic device 1 and process data.

[0182] The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable communication between the memory 11 and at least one processor 10, etc.

[0183] Figure 5 Only electronic devices with components are shown; those skilled in the art will understand that... Figure 5 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown, or combine certain components, or have different component arrangements.

[0184] For example, although not shown, the electronic device 1 may also include a power supply (such as a battery) to power various components. Preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, thereby enabling functions such as charging management, discharging management, and power consumption management. The power supply may also include one or more DC or AC power supplies, recharging devices, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components. The electronic device 1 may also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be described in detail here.

[0185] Furthermore, the electronic device 1 may also include a network interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a Wi-Fi interface, a Bluetooth interface, etc.), which is typically used to establish communication connections between the electronic device 1 and other electronic devices.

[0186] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired or wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display may also be appropriately referred to as a screen or display unit, used to display information processed in the electronic device 1 and to display a visual user interface.

[0187] It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

[0188] The image classification program for simulating radial field of view stored in the memory 11 of the electronic device 1 is a combination of multiple instructions, which, when run in the processor 10, can achieve the following:

[0189] Obtain the image to be classified, and perform noise reduction on the image to be classified;

[0190] Using the encoding layer of a pre-trained image recognition model, image feature extraction is performed on the denoised image to be classified to obtain the initial image feature matrix corresponding to the image to be classified.

[0191] Using a preset number of pooling layers in the image recognition model, the initial image feature matrix is ​​sequentially pooled to obtain a minimum pooling feature matrix. The preset number of pooling layers are arranged in descending order of their convolution kernels.

[0192] Based on the preset receptive field coefficients, dilated convolution is performed on the minimum pooling feature matrix to obtain the dilated feature matrix;

[0193] The image to be classified is classified using the decoding layer in the image recognition model based on the hole feature matrix, and the image classification result is obtained.

[0194] Furthermore, if the modules / units integrated in the electronic device 1 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile. For example, the computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0195] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor of an electronic device, can perform the following:

[0196] Obtain the image to be classified, and perform noise reduction on the image to be classified;

[0197] Using the encoding layer of a pre-trained image recognition model, image feature extraction is performed on the denoised image to be classified to obtain the initial image feature matrix corresponding to the image to be classified.

[0198] Using a preset number of pooling layers in the image recognition model, the initial image feature matrix is ​​sequentially pooled to obtain a minimum pooling feature matrix. The preset number of pooling layers are arranged in descending order of their convolution kernels.

[0199] Based on the preset receptive field coefficients, dilated convolution is performed on the minimum pooling feature matrix to obtain the dilated feature matrix;

[0200] The image to be classified is classified using the decoding layer in the image recognition model based on the hole feature matrix, and the image classification result is obtained.

[0201] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional modules.

[0202] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0203] Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be embraced within the invention. No appended diagram markings in the claims should be construed as limiting the scope of the claims.

[0204] The blockchain referred to in this invention is a novel application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product service layer, and an application service layer.

[0205] The embodiments of this application can acquire and process relevant data based on holographic projection technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0206] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a system claim may also be implemented by a single unit or device through software or hardware. The term "second class" is used to indicate names and does not indicate any specific order.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. An image classification method simulating radial field of view, characterized in that, The method includes: Obtain the image to be classified, and smooth the pixels in the image to be classified to obtain the noise-reduced image to be classified. Using the encoding layer of a pre-trained image recognition model, the denoised image to be classified is divided into blocks to obtain a block sequence. Image features are extracted from each block in the block sequence to obtain a block feature matrix for each block. The block feature matrix is ​​rearranged according to the size of the pixel values ​​in each column of the block feature matrix. The rearranged block feature matrix is ​​converted into a one-dimensional block feature vector using the fully connected layer of the image recognition model. The sum of pixels in the one-dimensional feature vector of each patch is used as the feature weight coefficient of the corresponding patch. Non-linear activation is performed on the feature weight coefficient of each patch, and linear normalization is performed on the non-linearly activated feature weight coefficient to obtain the normalized image attention weight corresponding to each patch. The normalized image attention weights are used to generate a weighted image feature vector for each patch. All weighted image feature vectors are concatenated to obtain the initial image feature matrix corresponding to the image to be classified. Using a preset number of pooling layers in the image recognition model, the initial image feature matrix is ​​sequentially pooled to obtain a minimum pooling feature matrix. The preset number of pooling layers are arranged in descending order of their convolution kernels. Based on the preset receptive field coefficients, dilated convolution is performed on the minimum pooling feature matrix to obtain the dilated feature matrix; The image to be classified is classified using the decoding layer in the image recognition model based on the hole feature matrix, and the image classification result is obtained.

2. The image classification method simulating radial field of view as described in claim 1, characterized in that, The step of generating a weighted image feature vector for each patch using the normalized image attention weights includes: The normalized image attention weights are multiplied by the rearranged patch feature matrix of the corresponding patch to obtain the weighted image feature matrix. The weighted image feature vector is obtained by summing the values ​​of each column in the weighted image feature matrix.

3. The image classification method simulating radial field of view as described in claim 1, characterized in that, The step of performing dilated convolution on the minimum pooling feature matrix according to preset receptive field coefficients to obtain a dilated feature matrix includes: Obtain the minimum kernel size of the preset number of pooling layers; The minimum convolution kernel is zero-padded according to the preset receptive field coefficients to obtain an expanded convolution kernel; The dilated convolution kernel is used to perform a convolution operation on the minimum pooling feature matrix to obtain the dilated feature matrix corresponding to the image to be classified.

4. The image classification method simulating radial field of view as described in claim 1, characterized in that, The step of smoothing the pixels in the image to be classified to obtain the noise-reduced image to be classified includes: Select one pixel in the image to be classified as the target point in sequence, and identify the average gray level difference between the target point and its surrounding neighboring points; When the average gray-level difference is greater than or equal to a preset noise reduction threshold, the target point is smoothed. When the average gray level difference is less than the preset noise reduction threshold, the pixel gray level of the target point remains unchanged; Determine whether all pixels in the image to be classified have been selected; If any pixel in the image to be classified is not selected, return to the steps described above of sequentially selecting one pixel in the image to be classified as the target point. When all pixels in the image to be classified are selected, the denoised image to be classified is obtained.

5. An image classification device simulating radial field of view, characterized in that, The device includes: An image denoising module is used to acquire an image to be classified and to smooth the pixels in the image to be classified to obtain a denoised image to be classified. The initial feature extraction module is used to divide the denoised image to be classified into blocks using the encoding layer of a pre-trained image recognition model to obtain a block sequence. Image features are extracted from each block in the block sequence to obtain a block feature matrix for each block. The block feature matrix is ​​rearranged according to the pixel values ​​of each column. The rearranged block feature matrix is ​​converted into a one-dimensional block feature vector using the fully connected layer of the image recognition model. The sum of pixels in the one-dimensional block feature vector of each block is used as the feature weight coefficient of the corresponding block. Non-linear activation is applied to the feature weight coefficient of each block. The non-linearly activated feature weight coefficient is then linearly normalized to obtain the normalized image attention weight for each block. The normalized image attention weight is used to generate a weighted image feature vector for each block. All weighted image feature vectors are concatenated to obtain the initial image feature matrix corresponding to the image to be classified. The radial pooling module is used to sequentially perform pooling operations on the initial image feature matrix using a preset number of pooling layers in the image recognition model to obtain a minimum pooled feature matrix. The preset number of pooling layers are arranged in descending order of the convolution kernel size. The dilated convolution module is used to perform dilated convolution on the minimum pooling feature matrix according to the preset receptive field coefficients to obtain the dilated feature matrix. The feature classification module is used to classify the image to be classified based on the hole feature matrix using the decoding layer in the image recognition model, and obtain the image classification result.

6. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the image classification method for simulating radial field of view as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the image classification method for simulating radial field of view as described in any one of claims 1 to 4.

Images