Image processing method, device, computer equipment and storage medium

Abstract

The present application relates to an image processing method, device, computer equipment and storage medium. The method comprises: obtaining an image to be classified; performing statistical processing on the image features of the image to be classified to obtain at least two image statistical features for the image to be classified; inputting the image statistical features into a pre-trained image classification neural network respectively; and determining the classification result for the image to be classified according to the output result of the pre-trained image classification neural network. The use of this method can improve the accuracy of image classification.

Description

Technical Field

[0001] The present application relates to the field of image recognition technology, and in particular to an image processing method, apparatus, computer equipment and storage medium. Background Art

[0002] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in this learning process is of great help in interpreting data such as text, images, and sounds. Deep learning is a complex machine learning algorithm that has achieved results in speech and image recognition that far exceed previous related technologies.

[0003] However, when using neural networks to classify images, the prior art often performs dimensionality reduction processing on the image feature map to avoid the problem of overfitting of the neural network due to too many neural network processing parameters, because the feature dimension of each pixel in the image feature map is too high. However, when performing dimensionality reduction processing on the image feature map, information loss often occurs, which reduces the classification accuracy of the neural network for the image.

[0004] Therefore, existing image classification methods have the problem of low classification accuracy. Summary of the invention

[0005] Based on this, it is necessary to provide an image processing method, device, computer equipment and storage medium that can improve the accuracy of image classification in response to the above technical problems.

[0006] An image processing method is applied to a pre-trained image classification neural network, wherein the image classification neural network includes a feature extraction layer and a fully connected layer, and the method includes:

[0007] Get the image to be classified;

[0008] By means of the feature extraction layer, statistical processing is performed on the image features of the image to be classified to separate at least two image statistical features for the image to be classified;

[0009] The image statistical features are fully connected through the fully connected layer to obtain a classification result for the image to be classified.

[0010] An image processing device is applied to a pre-trained image classification neural network, wherein the image classification neural network includes a feature extraction layer and a fully connected layer, and the device includes:

[0011] An acquisition module, used for acquiring images to be classified;

[0012] A feature extraction module, used to perform statistical processing on the image features of the image to be classified through the feature extraction layer, and separate at least two image statistical features for the image to be classified;

[0013] The classification module is used to perform full connection processing on the image statistical features through the full connection layer to obtain a classification result for the image to be classified.

[0014] A computer device is applied to a pre-trained image classification neural network, the image classification neural network includes a feature extraction layer and a fully connected layer, includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0015] Get the image to be classified;

[0016] By means of the feature extraction layer, statistical processing is performed on the image features of the image to be classified to separate at least two image statistical features for the image to be classified;

[0017] The image statistical features are fully connected through the fully connected layer to obtain a classification result for the image to be classified.

[0018] A computer-readable storage medium is applied to a pre-trained image classification neural network, the image classification neural network includes a feature extraction layer and a fully connected layer, and a computer program is stored thereon, and when the computer program is executed by a processor, the following steps are implemented:

[0019] Get the image to be classified;

[0020] By means of the feature extraction layer, statistical processing is performed on the image features of the image to be classified to separate at least two image statistical features for the image to be classified;

[0021] The image statistical features are fully connected through the fully connected layer to obtain a classification result for the image to be classified.

[0022] The above-mentioned image processing method, device, computer equipment and storage medium are applied to a pre-trained image classification neural network, which includes a feature extraction layer and a fully connected layer. The image to be classified is obtained; through the feature extraction layer, the image features of the image to be classified are statistically processed to split out at least two image statistical features for the image to be classified; through the fully connected layer, the image statistical features are fully connected to obtain a classification result for the image to be classified; in this way, data augmentation can be performed while reducing the feature dimension of the image to be classified, and more image feature information that can be used to express the image to be classified can be obtained, and then the image feature information can be used to accurately classify the image to be classified, thereby reducing the parameter processing amount of the image classification neural network, avoiding overfitting caused by a significant decrease in memory during training of the image classification neural network, and improving the classification accuracy of the image. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 A diagram of an application environment of an image processing method in an embodiment;

[0024] Figure 2 is a schematic flow chart of an image processing method in an embodiment;

[0025] Figure 3 is a schematic flow chart of an image processing method in another embodiment;

[0026] Figure 4 A diagram of a practical application scenario of an image processing method in an embodiment;

[0027] Figure 5 A schematic diagram of a network architecture of an image classification neural network of an image processing method in one embodiment;

[0028] Figure 6 A schematic diagram of a network architecture of an image classification neural network of an image processing method in another embodiment;

[0029] Figure 7 is a structural block diagram of an image processing device in one embodiment;

[0030] Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0032] The present application provides an image processing method, which is applied to a pre-trained image classification neural network, wherein the image classification neural network includes a feature extraction layer and a fully connected layer, and can be applied to Figure 1 The application environment diagram shown. The application environment diagram includes a classification server 110. The classification server 110 obtains the image to be classified; then, the classification server 110 performs statistical processing on the image features of the image to be classified through the feature extraction layer, and separates at least two image statistical features for the image to be classified; finally, the classification server 110 performs full connection processing on the image statistical features through the full connection layer to obtain a classification result for the image to be classified. In actual applications, the server 104 can be implemented as an independent server or a server cluster composed of multiple servers.

[0033] like Figure 2 As shown, in one embodiment, an image processing method is provided. This embodiment mainly applies the method to the above Figure 1 The classification server 110 in FIG. Figure 2 , the image processing method specifically comprises the following steps:

[0034] An image processing method is applied to a pre-trained image classification neural network, the image classification neural network includes a feature extraction layer and a fully connected layer, the method includes:

[0035] S210, obtaining an image to be classified.

[0036] The image classification neural network may refer to a neural network used for image classification. In practical applications, the image classification neural network may be a convolutional neural network.

[0037] The feature extraction layer may refer to a network layer used to extract image features of images to be classified in an image classification neural network.

[0038] Among them, the fully connected layer may refer to a network layer used to perform fully connected processing on input features in an image classification neural network.

[0039] The image to be classified may refer to a natural image that needs to be classified. In practical applications, the image to be classified may refer to a medical detection image that needs to determine the type of lesion. For example, a brain CT image used to diagnose cerebral hemorrhage, a lung CT image used to diagnose benign or malignant lung nodules, etc.

[0040] In a specific implementation, when it is necessary to determine the classification result of the image to be classified, the classification server 110 obtains the image to be classified and inputs the image to be classified into a pre-trained image classification neural network.

[0041] S220, performing statistical processing on the image features of the image to be classified through the feature extraction layer, and separating out at least two image statistical features for the image to be classified.

[0042] The statistical processing may refer to the process of performing image statistical calculations on the image features of the image to be classified. In practical applications, the statistical processing may be to calculate the weighted average, variance, correlation coefficient, entropy, skewness, kurtosis, coefficient of variation, etc.

[0043] The image statistical features may refer to features obtained by performing statistical calculations on image features of the image to be classified. In practical applications, different types of statistical calculation methods may obtain different types of image statistical features.

[0044] Among them, the image statistical features can be image weighted average features, image variance features, image correlation coefficient features, image entropy features, image skewness features, image kurtosis features, image coefficient of variation features and the like.

[0045] In a specific implementation, after the classification server 110 obtains the image to be classified and inputs the image to be classified into a pre-trained image classification neural network, the classification server 110 extracts features of the image to be classified through the feature extraction layer of the pre-trained image classification neural network to obtain image features of the image to be classified. Then, the classification server 110 also performs image statistical calculations on the image features of the image to be classified through the feature extraction layer, and then separates at least two image statistical features for the image to be classified from the image features of the image to be classified.

[0046] Specifically, the classification server 110 can perform convolution operations on the image to be classified through the feature extraction layer of the pre-trained image classification neural network to obtain image convolution features; then, the image convolution features are normalized, activated, max-pooled and residual operations are performed to obtain image features of the image to be classified. Then, different types of image statistical calculations are performed on the image features of the image to be classified to obtain at least two image statistical features for the image to be classified. For example, image weighted average features, image variance features, image skewness features, image kurtosis features, image coefficient of variation features, etc., for the image to be classified.

[0047] S230, performing full connection processing on the image statistical features through the fully connected layer to obtain a classification result for the image to be classified.

[0048] In a specific implementation, after the classification server 110 obtains at least two image statistical features for the image to be classified through a pre-trained image classification neural network, the classification server 110 inputs each image statistical feature into the fully connected layer of the pre-trained image classification neural network according to the corresponding channel, and performs fully connected processing on the image statistical features through the fully connected layer of the pre-trained image classification neural network, thereby obtaining a classification result for the image to be classified.

[0049] In the above-mentioned image processing method, a pre-trained image classification neural network is applied, and the image classification neural network includes a feature extraction layer and a fully connected layer, by obtaining an image to be classified; through the feature extraction layer, the image features of the image to be classified are statistically processed to split out at least two image statistical features for the image to be classified; through the fully connected layer, the image statistical features are fully connected to obtain a classification result for the image to be classified; in this way, data augmentation can be performed while reducing the feature dimension of the image to be classified, and more image feature information that can be used to express the image to be classified can be obtained, and then the image to be classified can be accurately classified according to the image feature information, reducing the parameter processing amount of the image classification neural network, avoiding overfitting caused by a significant decrease in memory of the image classification neural network during training, and improving the classification accuracy of the image.

[0050] In another embodiment, the image features of the image to be classified are statistically processed through a feature extraction layer to separate at least two image statistical features for the image to be classified, including: determining a region of interest in the image to be classified; performing statistical processing on features of each pixel point in the region of interest to obtain at least two first image statistical features for the region of interest; and determining image statistical features for the image to be classified based on the first image statistical features.

[0051] The region of interest may refer to a key region that needs to be focused on in the image to be classified when the image to be classified is classified.

[0052] For example, for a brain CT image used to diagnose brain hemorrhage, the focus area of ​​the brain hemorrhage in the brain CT image is the region of interest in the image to be classified.

[0053] In practical applications, the region of interest can also be called the segmentation region (within the Mask).

[0054] In a specific implementation, before the user uses the classification server 110 to identify the image to be classified, the user also needs to segment and annotate the image to be classified, divide the region of interest in the image to be classified, and obtain a segmented region marking feature map (original mask map). For example, for medical images used clinically, doctors can directly or use AI software to segment the lesion area on the medical image to obtain a segmented region marking feature map for the medical image. Among them, the segmented region marking feature map is composed of 0 and 1, the Mask value in the segmented area is 1, and the Mask value outside the segmented area is 0.

[0055] When the classification server 110 performs statistical processing on the image features of the image to be classified through the feature extraction layer, it separates out at least two image statistical features for the image to be classified, specifically including: determining the region of interest in the image to be classified; performing statistical processing on the features of each pixel point in the region of interest to obtain at least two first image statistical features for the region of interest; and determining the image statistical features for the image to be classified based on the first image statistical features.

[0056] It should be noted that the classification server 110 can perform average pooling processing on the segmented region marking feature map of the image to be classified during the feature extraction process of the image to be classified, and obtain the pooled segmented region marking feature map. Therefore, each pixel point of the pooled segmented region marking feature map is a value between 0 and 1. Therefore, in the process of statistically processing the features of each pixel point in the region of interest, each Mask value in the pooled segmented region marking feature map can be used as a weighted weight to act on the feature map of the image to be classified; that is, the weight map within the segmented region is represented by Mask. Finally, at least two first image statistical features for the region of interest, that is, the image statistical features within the segmented region, are calculated, and finally, based on the first image statistical features, the image statistical features for the image to be classified are determined.

[0057] Wherein, the first image statistics may include:

[0058] Weighted average within Mask:

[0059] Among them, Ex1 is the weighted average of the image in the segmented area; Mask is the marked feature map of the segmented area after pooling; sum() is the summation function; I X is any one of the feature maps.

[0060] Variance within Mask:

[0061] Among them, SD1 is the image variance of the segmented area; Ex1 is the weighted average of the image of the segmented area; Mask is the pooled segmented area marker feature map; sum() is the summation function; I X is any one of the feature maps.

[0062] Mask intra-coefficient of variation:

[0063] Among them, CV1 is the image variation coefficient of the segmented area; SD1 is the image variance of the segmented area; Ex1 is the weighted average value of the image of the segmented area.

[0064] Mask internal skewness:

[0065] Among them, Skew1 is the image skewness of the segmented area; SD1 is the image variance of the segmented area; Ex1 is the weighted average of the image of the segmented area; sum() is the summation function; I X is any one of the feature maps; Mask is the segmentation area marking feature map after pooling.

[0066] Mask internal kurtosis:

[0067] Among them, Avg1 is the image kurtosis of the segmented area; SD1 is the image variance of the segmented area; Ex1 is the weighted average of the image of the segmented area; sum() is the summation function; I X is any one of the feature maps; Mask is the segmentation area marking feature map after pooling.

[0068] The technical solution of this embodiment determines the region of interest in the image to be classified; performs statistical processing on the features of each pixel in the region of interest to obtain at least two first image statistical features for the region of interest; determines the image statistical features for the image to be classified based on the first image statistical features; and can achieve targeted identification of regions with classification value in the image to be classified, thereby reducing the amount of parameter processing of the image classification neural network, avoiding overfitting caused by a significant decrease in memory of the image classification neural network during training, and improving the classification accuracy of the image.

[0069] In another embodiment, the image features of the image to be classified are statistically processed through a feature extraction layer to separate at least two image statistical features for the image to be classified, including: determining a non-interest area in the image to be classified; performing statistical processing on features of each pixel point in the non-interest area to obtain at least two second image statistical features for the non-interest area; and determining the image statistical features for the image to be classified based on the first image statistical features and the second image statistical features.

[0070] The non-interest region may refer to an image region other than the interest region in the image to be classified.

[0071] In practical applications, the non-interest region can also be referred to as outside the segmentation region (outside the Mask).

[0072] In a specific implementation, when the classification server 110 performs statistical processing on the image features of the image to be classified through the feature extraction layer, it splits out at least two image statistical features for the image to be classified, specifically including: determining a non-interest area in the image to be classified; performing statistical processing on the features of each pixel point in the non-interest area to obtain at least two second image statistical features for the non-interest area; and determining the image statistical features for the image to be classified based on the first image statistical features and the second image statistical features.

[0073] It should be noted that the classification server 110 can perform average pooling processing on the segmented region marking feature map of the image to be classified along with the feature extraction process of the image to be classified, and obtain the original segmented region marking feature map after pooling. Therefore, each pixel in the segmented region marking feature map after pooling has a value between 0 and 1. In the process of statistically processing the features of each pixel in the non-interested area, each mask value in the segmented region marking feature map after pooling can be used as a weighted weight to act on the feature map of the image to be classified; that is, the weight map outside the segmented area is represented by 1-Mask. Finally, at least two second image statistical features for the non-interested area, that is, the image statistical features outside the segmented area, are calculated, and finally, based on the second image statistical features, the image statistical features for the image to be classified are determined.

[0074] Wherein, the second image statistics may include:

[0075] Weighted average outside Mask:

[0076] Among them, Ex2 is the weighted average of the image outside the segmented area; 1-Mask is the marked feature map outside the segmented area after pooling; sum() is the summation function; I X is any one of the feature maps.

[0077] Mask outer variance:

[0078] Among them, SD2 is the variance of the image outside the segmented area; Ex2 is the weighted average of the image outside the segmented area; 1-Mask is the marked feature map outside the segmented area after pooling; sum() is the summation function; I X is any one of the feature maps.

[0079] Mask outer coefficient of variation:

[0080] Among them, CV2 is the coefficient of variation of the image outside the segmented area; SD2 is the variance of the image outside the segmented area; Ex2 is the weighted average of the image outside the segmented area.

[0081] Mask external skewness:

[0082] Among them, Skew2 is the skewness of the image outside the segmented area; SD2 is the variance of the image outside the segmented area; Ex2 is the weighted average of the image outside the segmented area; sum() is the summation function; I X is any one of the feature maps; 1-Mask is the marked feature map outside the segmented area after pooling.

[0083] Mask outer kurtosis:

[0084] Where Avg2 is the kurtosis of the image outside the segmented area; SD2 is the variance of the image outside the segmented area; Ex2 is the weighted average of the image outside the segmented area; sum() is the summation function; I X is any one of the feature maps; 1-Mask is the marked feature map outside the segmented area after pooling.

[0085] The technical solution of this embodiment obtains at least two first image statistical features for the region of interest by performing statistical processing on the features of each pixel point in the region of interest in the image to be classified; and obtains at least two second image statistical features for the region of non-interest by performing statistical processing on the features of each pixel point in the region of non-interest in the image to be classified; and determines the image statistical features for the image to be classified based on the first image statistical features and the second image statistical features; in this way, data augmentation can be performed while reducing the feature dimension of the image to be classified, so as to obtain more image feature information that can be used to express the image to be classified, and then the image to be classified can be accurately classified based on the image feature information, thereby reducing the parameter processing amount of the image classification neural network, avoiding overfitting caused by a significant decrease in memory of the image classification neural network during training, and improving the classification accuracy of the image.

[0086] In another embodiment, it also includes: constructing an image classification neural network to be trained, and obtaining training sample images; using the training sample images to train the image classification neural network to be trained to obtain a pre-trained image classification neural network.

[0087] The training sample images may refer to sample images used to train the image classification neural network to be trained.

[0088] In practical applications, the training sample images may include multiple data sets; wherein the multiple data sets may be a CT plain scan brain hemorrhage cause judgment data set provided by a cooperative hospital, an MR data set including 4 modalities for the BraTS18 competition (a machine learning competition), a benign and malignant lung nodule CT classification data set of a cooperative hospital, and other data sets.

[0089] In a specific implementation, the classification server 110, in the process of obtaining a pre-trained image classification neural network, specifically includes: the classification server 110 constructs an image classification neural network to be trained, and the classification server 110 obtains a training sample image; then, the classification server 110 uses the training sample image to train the image classification neural network to be trained to obtain a pre-trained image classification neural network.

[0090] The technical solution of this embodiment is to construct an image classification neural network to be trained and obtain training sample images; then, use the training sample images to train the image classification neural network to be trained, so that the pre-trained image classification neural network can classify the image to be classified more accurately, thereby improving the classification accuracy of the image to be classified.

[0091] In another embodiment, obtaining a training sample image includes: obtaining an original sample image; performing data augmentation processing on the original sample image to obtain a processed original sample image; and using the processed original sample image as a training sample image.

[0092] In a specific implementation, before the classification server 110 uses the training sample images to train the image classification neural network to be trained, the classification server 110 can also obtain the original sample images; then, the classification server 110 performs data augmentation processing on the original sample images. Specifically, the original sample images can be flipped, rotated, scaled, contrast enhanced, and the like to obtain multiple processed original sample images; finally, the processed original sample images are used as training sample images.

[0093] The technical solution of this embodiment is to obtain the original sample images and perform data augmentation processing on the original sample images to obtain the training sample images before using the training sample images to train the image classification neural network to be trained, thereby increasing the number of training samples for the image classification neural network to be trained, and at the same time increasing the directionality and information value at different sizes, so that the obtained pre-trained image classification neural network can classify the images to be classified more accurately, thereby improving the classification accuracy of the images to be classified.

[0094] In another embodiment, a training sample image is used to train an image classification neural network to be trained to obtain a pre-trained image classification neural network, including: inputting the training sample image into the image classification neural network to be trained to obtain a first recognition result of the image classification neural network to be trained for the training sample image; obtaining a target recognition result for the training sample image, and calculating a loss parameter for the image classification neural network to be trained based on the first recognition result and the target recognition result; adjusting the network parameters of the image classification neural network to be trained based on the loss parameter to obtain a trained image classification neural network, and using the trained image classification neural network as the image classification neural network to be trained; repeating the above steps until a pre-trained image classification neural network is obtained.

[0095] In a specific implementation, the classification server 110 can use the training sample images to train the image classification neural network to be trained. Specifically, the training sample images can be input into the image classification neural network to be trained to obtain a first recognition result of the image classification neural network to be trained for the training sample images; a target recognition result for the training sample images is obtained, and the error between the first recognition result and the target recognition result is calculated as a loss parameter of the image classification neural network to be trained; based on the loss parameter, partial derivatives of the network parameters of the image classification neural network to be trained are taken to obtain partial derivatives of the network parameters for the image classification neural network to be trained; based on the partial derivatives, the network parameters of the image classification neural network to be trained are adjusted to obtain a trained image classification neural network, and the trained image classification neural network is used as the image classification neural network to be trained; the above steps are repeated until a pre-trained image classification neural network is obtained.

[0096] In practical applications, the network parameters of the image classification neural network to be trained can be updated based on the gradient descent method of Adam (a first-order optimization algorithm that can replace the traditional stochastic gradient descent (SGD) process); wherein the initial learning rate is 0.05, and betas in Adam = (0.95, 0.9995). The probability value, i.e., the first recognition result (predicted category, which is an N*1 vector, where N is the number of categories) is obtained through the neural network model, and the error between it and the label, i.e., the target recognition result (the true category), is calculated as the loss parameter of the image classification neural network to be trained to obtain the loss function; finally, by minimizing the loss function, the error gradient can be calculated and the gradient of the network can be updated through back propagation. After the above network training is completed, the classification result is obtained using the final predicted probability value.

[0097] The technical solution of this embodiment is to construct an image classification neural network to be trained and obtain training sample images; then, use the training sample images to train the image classification neural network to be trained, so that the pre-trained image classification neural network can classify the image to be classified more accurately, thereby improving the classification accuracy of the image to be classified.

[0098] In another embodiment, obtaining the image to be classified includes: obtaining the original image to be classified; and performing normalization processing on the original image to be classified to obtain the image to be classified.

[0099] The original image to be classified may refer to an image to be classified that has not been processed.

[0100] In a specific implementation, before the classification server 110 inputs the image to be classified into a pre-trained image classification neural network, the classification server 110 also needs to perform 0-1 normalization processing on the original image to be classified to obtain the normalized original image to be classified. Finally, the classification server 110 uses the normalized original image to be classified as the image to be classified and inputs it into the pre-trained image classification neural network for image classification processing.

[0101] The technical solution of this embodiment obtains the original image to be classified by acquiring and normalizing the original image to be classified, thereby reducing the parameter processing amount of the image classification neural network while avoiding reducing the amount of feature information of the image to be classified, improving the recognition efficiency of the image classification neural network, and further improving the accuracy of the classification results output by the image classification neural network.

[0102] like Figure 3 As shown, in another embodiment, an image processing method is provided, which is applied to a pre-trained image classification neural network, wherein the image classification neural network includes a feature extraction layer and a fully connected layer. The image processing method specifically includes the following steps:

[0103] S310, obtaining an original image to be classified;

[0104] S320, performing normalization processing on the original image to be classified to obtain the image to be classified.

[0105] S330: Determine a region of interest in the image to be classified.

[0106] S340: Perform statistical processing on features of each pixel point in the region of interest to obtain at least two first image statistical features for the region of interest.

[0107] S350: Determine a non-interest region in the image to be classified.

[0108] S360: Perform statistical processing on the features of each pixel point in the non-interest region to obtain at least two second image statistical features for the non-interest region.

[0109] S370: Determine image statistical features for the image to be classified according to the first image statistical features and the second image statistical features.

[0110] S380, performing full connection processing on the image statistical features through the fully connected layer to obtain a classification result for the image to be classified.

[0111] The specific definition of the above steps can be found in the above definition of an image processing method, which will not be repeated here.

[0112] The technical solution of this embodiment is applied to a pre-trained image classification neural network, which includes a feature extraction layer and a fully connected layer. The image to be classified is obtained; the image features of the image to be classified are statistically processed through the feature extraction layer to split out at least two image statistical features for the image to be classified; the image statistical features are fully connected through the fully connected layer to obtain a classification result for the image to be classified; in this way, data augmentation can be performed while reducing the feature dimension of the image to be classified, so as to obtain more image feature information that can be used to express the image to be classified, and then the image feature information can be used to accurately classify the image to be classified, thereby reducing the parameter processing amount of the image classification neural network, avoiding overfitting caused by a significant decrease in memory of the image classification neural network during training, and improving the classification accuracy of the image.

[0113] It should be understood that although Figure 2 and Figure 3 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 and Figure 3 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0114] To facilitate the understanding of those skilled in the art, Figure 4 A practical application scenario diagram of an image processing method in one embodiment is provided: Figure 4As shown, the user inputs a 3D medical image through the front end A; then, the front end A receives the data and performs data preprocessing on the data, wherein the data preprocessing may include but is not limited to data augmentation such as translation, rotation, symmetry, and segmentation of the 3D medical image and selection of the corresponding organ algorithm; the front end A then uploads it to the back end, and the back end obtains the image to be classified, and through the feature extraction layer, performs statistical processing on the image features of the image to be classified, and separates at least two image statistical features for the image to be classified; through the fully connected layer, the image statistical features are fully connected to obtain the classification results for the image to be classified. Finally, the back end uploads the classification results for the image to be classified to the front end B for the doctor's reference. In this way, more feature information that can be used to express the image features of the image to be classified can be obtained, and then the feature information can be used to accurately classify the image to be classified.

[0115] In order to facilitate the understanding of those skilled in the art, Figure 5 A schematic diagram of a network architecture of an image classification neural network of an image processing method in one embodiment is provided: Figure 5 As shown, before the user uses the classification server 110 to identify the image to be classified, the user needs to segment and annotate the image to be classified, divide the area of ​​interest in the image to be classified, obtain the original mask image, namely Input_mask, and input the image to be classified into the image classification neural network Input_image. The classification server 110 performs a convolution operation on the image to be classified through the convolution block (Conv) of the pre-trained image classification neural network to obtain the image convolution feature; then, the image convolution feature is normalized (BN, BatchNormalization), activated using the activation function ReLU, and the maximum pooling (Maxpooling) and residual block (ResNetblock) residual operation to obtain the feature map of the image to be classified. At the same time, the image classification neural network also needs to perform average pooling (Avgpooling) on ​​the original Mask image of the image to be classified to obtain the original Mask image after pooling. The feature extraction module (Feature extraction) of the image classification neural network uses each mask value in the pooled original mask map as a weighted weight to act on the feature map of the image to be classified, and splits the feature map of the image to be classified into at least two image statistical features, such as image weighted average feature, image variance feature, image skewness feature, image kurtosis feature, image coefficient of variation feature, etc. Finally, the at least two image statistical features mentioned above are input into the fully connected layer (FC) for full connection processing to obtain the classification result for the image to be classified. In this way, more feature information that can be used to express the image features of the image to be classified can be obtained, and then the feature information can be used to accurately classify the image to be classified.

[0116] In order to facilitate the understanding of those skilled in the art, Figure 6 A schematic diagram of a network architecture of an image classification neural network of an image processing method in another embodiment is provided: Figure 6 As shown, the feature extraction module (Feature extraction) of the image classification neural network (CNN, convolutional neural network) uses each mask value in the pooled original mask map as a weighted weight to act on the feature map of the image to be classified in a certain channel (Channel), and divides the channel into MaskChannel (channel within the segmented area) and Nonmask Channel (channel outside the segmented area), so as to achieve statistical processing of the features of each pixel in the region of interest (within the segmented area, within the Mask) in the image to be classified, and obtain at least two first image statistical features within the segmented area, and statistical processing of the features of each pixel in the non-interested area (outside the segmented area, outside the Mask) in the image to be classified, and obtain at least two second image statistical features outside the segmented area. Finally, the above-mentioned first image statistical features and second image statistical features are input into the fully connected layer of CNN according to independent input channels, so as to determine the classification result for the image to be classified. In this way, more feature information that can be used to express the image features of the image to be classified can be obtained, and then the feature information can be used to accurately classify the image to be classified.

[0117] In one embodiment, Figure 7 As shown, an image processing device is provided, comprising:

[0118] An acquisition module 710 is used to acquire an image to be classified;

[0119] A feature extraction module 720, configured to perform statistical processing on the image features of the image to be classified through the feature extraction layer, and separate at least two image statistical features for the image to be classified;

[0120] The classification module 730 is used to perform full connection processing on the image statistical features through the full connection layer to obtain a classification result for the image to be classified.

[0121] The above-mentioned image processing device is applied to a pre-trained image classification neural network, which includes a feature extraction layer and a fully connected layer. The image to be classified is obtained; through the feature extraction layer, the image features of the image to be classified are statistically processed to split at least two image statistical features for the image to be classified; through the fully connected layer, the image statistical features are fully connected to obtain a classification result for the image to be classified; in this way, data augmentation can be performed while reducing the feature dimension of the image to be classified, and more image feature information that can be used to express the image to be classified can be obtained, and then the image feature information can be used to accurately classify the image to be classified, reducing the parameter processing amount of the image classification neural network, avoiding overfitting caused by a significant decrease in memory of the image classification neural network during training, and improving the classification accuracy of the image.

[0122] In one embodiment, the feature extraction module 720 includes: a first region determination submodule, used to determine the region of interest in the image to be classified; a first statistical submodule, used to perform statistical processing on the features of each pixel point in the region of interest to obtain at least two first image statistical features for the region of interest; a first determination submodule, used to determine the image statistical features for the image to be classified based on the first image statistical features.

[0123] In one embodiment, the feature extraction module 720 includes: a second region determination submodule, used to determine a non-interest region in the image to be classified; a second statistical submodule, used to perform statistical processing on features of each pixel point in the non-interest region to obtain at least two second image statistical features for the non-interest region; and a second determination submodule, used to determine image statistical features for the image to be classified based on the first image statistical features and the second image statistical features.

[0124] In one embodiment, the above-mentioned image processing device also includes: a construction module, used to construct an image classification neural network to be trained, and to obtain training sample images; a training module, used to use the training sample images to train the image classification neural network to be trained to obtain the pre-trained image classification neural network.

[0125] In one embodiment, the above-mentioned construction module includes: an acquisition submodule for acquiring an original sample image; an augmentation submodule for performing data augmentation processing on the original sample image to obtain a processed original sample image; and an image generation submodule for using the processed original sample image as the training sample image.

[0126] In one embodiment, the training module includes: an input submodule, used to input the training sample image into the image classification neural network to be trained, and obtain a first recognition result of the image classification neural network to be trained for the training sample image; a loss calculation submodule, used to obtain a target recognition result for the training sample image, and calculate the loss parameter for the image classification neural network to be trained based on the first recognition result and the target recognition result; an adjustment submodule, used to adjust the network parameters of the image classification neural network to be trained based on the loss parameter, to obtain a trained image classification neural network, and use the trained image classification neural network as the image classification neural network to be trained; an iteration submodule, used to repeatedly execute the above steps until the pre-trained image classification neural network is obtained.

[0127] In one embodiment, the acquisition module 710 includes: an image acquisition submodule, used to acquire the original image to be classified; and a normalization submodule, used to perform normalization processing on the original image to be classified to obtain the image to be classified.

[0128] For the specific definition of an image processing device, please refer to the definition of an image processing method above, which will not be repeated here. Each module in the above-mentioned image processing device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0129] Figure 8 The internal structure diagram of a computer device in one embodiment is shown. The computer device may specifically be Figure 1 The classification server 110 in FIG. Figure 8 As shown, the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program, which, when executed by the processor, enables the processor to implement the image processing method. The internal memory may also store a computer program, which, when executed by the processor, enables the processor to execute the image processing method. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0130] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0131] In one embodiment, the image processing apparatus provided by the present application can be implemented in the form of a computer program. The computer program can be Figure 8 The computer device shown in the figure is run on the computer device. The memory of the computer device can store various program modules constituting the image processing device, such as: Figure 7 The acquisition module 710, feature extraction module 720 and classification module 730 are shown. The computer program composed of various program modules enables the processor to execute the steps of the image processing method of each embodiment of the present application described in this specification.

[0132] For example, Figure 8 The computer device shown can be Figure 7 The acquisition module 710 in the image processing apparatus shown in the figure acquires the image to be classified. The computer device can perform statistical processing on the image features of the image to be classified through the feature extraction layer through the feature extraction module 720, and separate at least two image statistical features for the image to be classified. The computer device can perform full connection processing on the image statistical features through the full connection layer through the classification module 730 to obtain a classification result for the image to be classified.

[0133] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above-mentioned image processing method. The steps of the image processing method here may be the steps of the image processing method in the above-mentioned various embodiments.

[0134] In one embodiment, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the above-mentioned image processing method. The steps of the image processing method here can be the steps in the image processing method of each of the above-mentioned embodiments.

[0135] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0136] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0137] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An image processing method, characterized in that: Applied to a pre-trained image classification neural network, the image classification neural network includes a feature extraction layer and a fully connected layer, the method includes: Get the image to be classified; The image features of the image to be classified are statistically processed by the feature extraction layer to separate at least two image statistical features for the image to be classified, including: statistically processing the features of each pixel point in the region of interest in the image to be classified to obtain at least two first image statistical features for the region of interest, and statistically processing the features of each pixel point in the non-interest region in the image to be classified to obtain at least two second image statistical features for the non-interest region; The image statistical features are fully connected through the fully connected layer to obtain a classification result for the image to be classified, including: the at least two first image statistical features and the at least two second image statistical features are input into the fully connected layer according to independent input channels to obtain a classification result for the image to be classified.

2. The method according to claim 1, characterized in that Also includes: Constructing an image classification neural network to be trained, and obtaining training sample images; The image classification neural network to be trained is trained using the training sample images to obtain the pre-trained image classification neural network.

3. The method according to claim 2, characterized in that The step of acquiring a training sample image comprises: Get the original sample image; Performing data augmentation processing on the original sample image to obtain a processed original sample image; The processed original sample image is used as the training sample image.

4. The method according to claim 2, characterized in that: The step of training the image classification neural network to be trained using the training sample images to obtain the pre-trained image classification neural network includes: Inputting the training sample image into the image classification neural network to be trained, and obtaining a first recognition result of the image classification neural network to be trained for the training sample image; Obtaining a target recognition result for the training sample image, and calculating a loss parameter for the image classification neural network to be trained based on the first recognition result and the target recognition result; According to the loss parameter, adjusting the network parameters of the image classification neural network to be trained to obtain a trained image classification neural network, and using the trained image classification neural network as the image classification neural network to be trained; The above steps are repeated until the pre-trained image classification neural network is obtained.

5. The method according to claim 1, characterized in that The step of obtaining the image to be classified comprises: Get the original image to be classified; The original image to be classified is normalized to obtain the image to be classified.

6. An image processing device, applied to a pre-trained image classification neural network, the image classification neural network comprising a feature extraction layer and a fully connected layer, characterized in that: The device comprises: An acquisition module, used for acquiring images to be classified; A feature extraction module, used for performing statistical processing on the image features of the image to be classified through the feature extraction layer, and separating out at least two image statistical features for the image to be classified, including: performing statistical processing on the features of each pixel point in the region of interest in the image to be classified, and obtaining at least two first image statistical features for the region of interest, and performing statistical processing on the features of each pixel point in the non-interest region in the image to be classified, and obtaining at least two second image statistical features for the non-interest region; A determination module is used to perform full connection processing on the image statistical features through the fully connected layer to obtain a classification result for the image to be classified, including: inputting the at least two first image statistical features and the at least two second image statistical features into the fully connected layer according to independent input channels to obtain a classification result for the image to be classified.

7. The device according to claim 6, characterized in that The image processing device also includes: a construction module, used to construct the image classification neural network to be trained, and to obtain training sample images; a training module, used to use the training sample images to train the image classification neural network to be trained to obtain the pre-trained image classification neural network.

8. The device according to claim 7, characterized in that The construction module includes: an acquisition submodule for acquiring an original sample image; an augmentation submodule for performing data augmentation processing on the original sample image to obtain a processed original sample image; and an image generation submodule for using the processed original sample image as the training sample image.

9. The device according to claim 7, characterized in that The training module includes: an input submodule, used to input the training sample image into the image classification neural network to be trained, and obtain a first recognition result of the image classification neural network to be trained for the training sample image; a loss calculation submodule, used to obtain a target recognition result for the training sample image, and calculate a loss parameter for the image classification neural network to be trained based on the first recognition result and the target recognition result; an adjustment submodule, used to adjust the network parameters of the image classification neural network to be trained based on the loss parameter, to obtain a trained image classification neural network, and use the trained image classification neural network as the image classification neural network to be trained; an iteration submodule, used to repeatedly execute the above steps until the pre-trained image classification neural network is obtained.

10. The device according to claim 6, characterized in that The acquisition module includes: an image acquisition submodule, which is used to acquire the original image to be classified; and a normalization submodule, which is used to perform normalization processing on the original image to be classified to obtain the image to be classified.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images