Image classification methods, apparatus, devices, and storage media

Abstract

This invention provides an image classification method, apparatus, device, and storage medium. The method includes: labeling acquired original images to obtain a training image set; the training image set includes a first image, a second image, and a third image; training a first neural network model using the first and third images as the training set to obtain a first classification model; training a second neural network model using the second and third images as the training set to obtain a second classification model; selecting the classification model with the minimum generalization error from the first and second classification models as the final model; and classifying the target image based on the final model. This invention can significantly improve the accuracy of image classification and avoid the decrease in accuracy caused by errors that easily occur during long-term manual judgment.

Description

Technical Field

[0001] This invention relates to the field of image classification technology, specifically to an image classification method, apparatus, device, and storage medium. Background Technology

[0002] Currently, image information screening and recognition can be achieved through various methods, including computer vision technology and manual screening. Computer vision technology has advantages such as high processing speed, but its classification accuracy is low for images containing significant noise. Manual screening has higher classification accuracy, but its efficiency is generally low. In certain specific fields, such as medicine, the identification and interpretation of patients' fundus images rely entirely on the doctor's experience, which not only suffers from low efficiency but also risks affecting accuracy over long periods of manual processing.

[0003] There is currently no effective solution to the problems of low accuracy in computer-based image classification and the impact of long-term manual judgment on accuracy. Summary of the Invention

[0004] This invention provides an image classification method, apparatus, device, and storage medium to overcome the problems of low accuracy in computer-based image classification and the impact of long-term manual judgment on accuracy in related technologies.

[0005] To achieve the above objectives, a first aspect of the present invention provides an image classification method, comprising:

[0006] The acquired original images are labeled to obtain a training image set; the training image set includes a first image, a second image, and a third image; the first image is an image that does not contain a region of interest, the second image is an image that contains a region of interest with a first level, and the third image is an image that contains a region of interest with a second level.

[0007] A first neural network model is trained using the first image and the third image as a training set to obtain a first classification model; and a second neural network model is trained using the second image and the third image as a training set to obtain a second classification model.

[0008] Select the classification model with the smallest generalization error from the first classification model and the second classification model as the final model;

[0009] The target image is classified based on the final model.

[0010] The image classification method provided by this invention obtains a first classification model by training a neural network model using a first image and a third image as training sets; obtains a second classification model by training the neural network model using a second image and a third image as training sets; and selects the classification model with the minimum generalization error to classify the target image, which can significantly improve the accuracy of image classification and avoid the situation where errors easily occur during long-term manual judgment, leading to a decrease in the accuracy of image classification.

[0011] Preferably, classifying the target image based on the final model includes:

[0012] Calculate the probability that each pixel value of the target image belongs to multiple categories;

[0013] The category to which all pixel values ​​of the target image belong with the highest probability is determined based on the probability that each pixel value belongs to each category;

[0014] The category to which the highest probability belongs is taken as the category of the target image.

[0015] The image classification method provided by this invention determines the category of a target image by calculating the category probability of each pixel value in the target image. This method can quickly and accurately determine the category of the image to be detected, thereby improving the efficiency of image classification.

[0016] Preferably, after classifying the target image based on the final model, the method further includes:

[0017] Obtain the category probabilities obtained by the final model in classifying the target image;

[0018] Determine the gradient of the class probability relative to the feature map output by the last convolutional layer of the final model;

[0019] The feature map is weighted according to the gradient to obtain a heatmap.

[0020] The heatmap is overlaid on the target image to obtain the final heatmap reflecting the region of interest.

[0021] The image classification method provided by this invention obtains a final heatmap by overlaying a heatmap on the target image. It can clearly identify which regions of the target image contain regions of interest and which do not, thereby providing efficient, timely and accurate assistance for multidisciplinary intervention.

[0022] Preferably, selecting the classification model with the minimum generalization error from the first classification model and the second classification model as the final model includes:

[0023] The training image set is randomly divided into multiple sub-image sets of equal size;

[0024] The first classification model is trained and validated multiple times to obtain multiple first error values; the average of the multiple first error values ​​is calculated to obtain the first generalization error value of the first classification model.

[0025] The second classification model is trained and validated multiple times to obtain multiple second error values; the average of the multiple second error values ​​is calculated to obtain the second generalization error value of the second classification model.

[0026] The first generalization error value and the second generalization error value are compared, and the classification model with the smallest generalization error value is selected as the target model. The model is then trained again using the training image set to obtain the final model.

[0027] In each model training and model validation, any sub-image set of the training image set is selected as the validation set, and the sub-image sets other than the validation set are used as the training set; the number of model training times and the number of model validation times are determined by the number of sub-image sets.

[0028] The image classification method provided by this invention calculates the generalization error values ​​of a first classification model and a second classification model, and selects the classification model with the smallest generalization error value as the final model for classifying target images. This method can determine the classification model with the highest classification accuracy as the final model, which can significantly improve the accuracy of image classification.

[0029] Preferably, training the first neural network model using the first image and the third image as a training set includes:

[0030] The first image and the third image are assigned category weights; the first image and the third image with the assigned category weights are used to form a training set for training the first neural network model.

[0031] Preferably, classifying the target image based on the final model includes:

[0032] The first feature map is obtained by progressively reducing the feature map of the target image layer by layer and extracting features at each scale based on multiple convolutional layers.

[0033] The first feature map is processed into a second feature map using a global average pooling layer; the first feature map is a two-dimensional feature map, and the second feature map is a one-dimensional feature map.

[0034] The second feature map is passed to a fully connected layer whose output dimension is the number of classification categories, so as to output the weights of each category;

[0035] The category with the highest weight is used as the classification result for the target image.

[0036] The image classification method provided by this invention, through multi-layer feature extraction processing and transmitting the extracted feature map to a fully connected layer whose output dimension is the number of classification categories, uses the category with the highest weight as the classification result of the target image, which can extract key features from the target image, thereby improving the accuracy of image classification.

[0037] Preferably, both the original image and the target image are patient fundus images;

[0038] Before training the first neural network model using the first image and the third image as training sets, and before training the second neural network model using the second image and the third image as training sets, the method further includes:

[0039] All images in the training image set are uniformly scaled to the target size.

[0040] A second aspect of the present invention provides an image classification apparatus, comprising:

[0041] The annotation module is used to annotate the acquired original images to obtain a training image set; the training image set includes a first image, a second image, and a third image; the first image is an image that does not contain a region of interest, the second image is an image that contains a region of interest with a first level, and the third image is an image that contains a region of interest with a second level.

[0042] The model training module is used to train a first neural network model using the first image and the third image as training sets to obtain a first classification model; and to train a second neural network model using the second image and the third image as training sets to obtain a second classification model.

[0043] The model selection module is used to select the classification model with the smallest generalization error from the first classification model and the second classification model as the final model;

[0044] The classification module is used to classify the target image based on the final model.

[0045] A third aspect of the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program executable on the processor, and the processor executes the computer program to implement the steps in the various method embodiments described above.

[0046] A fourth aspect of the present invention provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the steps of the method described in the first aspect of the present invention and various possible designs of the first aspect. Attached Figure Description

[0047] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the image classification method of Embodiment 1 of the present invention.

[0049] Figure 2 This is a schematic diagram of the image classification device according to Embodiment 2 of the present invention.

[0050] Figure 3 This is a structural diagram of the computer device in Embodiment 3 of the present invention. Detailed Implementation

[0051] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0052] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0053] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can also refer to the internal connection of two components; and they can refer to a wireless connection or a wired connection. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0054] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] Example 1

[0056] This embodiment provides an image classification method, such as Figure 1 As shown, it includes the following steps:

[0057] S1: Label the acquired original images to obtain the training image set.

[0058] Specifically, the original image can be a patient's fundus image, obtained using a specialized fundus image acquisition device (RetCam), and transmitted to a temporary storage unit within the device for storage. Image annotation can be done manually or by machine. The training image set includes a first image, a second image, and a third image; the first image does not contain a region of interest (ROI), the second image contains a ROI with a first-level ROI, and the third image contains a ROI with a second-level ROI. The ROI can be a lesion area; a first-level ROI can be understood as a minor lesion area (no intervention required), and a second-level ROI can be understood as a severe lesion area (requiring intervention), including but not limited to intravenous chemotherapy, intraretinal artery chemotherapy, and focal cryocoagulation / photocoagulation. The first image can be a lesion-free image, the second image can be an image of stable retinoblastoma, and the third image can be an image of active retinoblastoma.

[0059] Specifically, the training image set includes 19,045 images without lesions, 2,918 images of stable retinoblastoma, and 14,660 images of active retinoblastoma. Of course, it is not limited to this and can be set according to actual conditions based on the present invention.

[0060] S2: Using the first and third images as training sets, train the first neural network model to obtain the first classification model; and using the second and third images as training sets, train the second neural network model to obtain the second classification model.

[0061] Specifically, before training the first neural network model using the first and third images as training sets, and before training the second neural network model using the second and third images as training sets, the method further includes: uniformly scaling all images in the training image set to a target size, such as 256*256, to meet the requirements for subsequent training of the neural network model.

[0062] Specifically, the neural network model includes a first neural network model and a second neural network model; the first classification model is obtained by training the first neural network model using the first image and the third image as training sets; the second classification model is obtained by training the second neural network model using the second image and the third image as training sets.

[0063] Specifically, the first and second neural network models can adopt a deep residual neural network, specifically the ResNet-50 neural network structure. The output layer of the original ResNet50 neural network is removed and replaced with a binary classification output layer. The network parameters can be optimized using the default optimization parameters and stochastic batch gradient descent method in the Keras framework, an open-source artificial neural network library written in Python. The final model output represents the probability of the target image belonging to one of the two classes; therefore, the activation function of the final output layer is the logistic regression algorithm, specifically the Softmax function, where the sum of the probabilities of the two classes is 1.

[0064] Preferably, in step S2 above, two binary classification models can be constructed using a ResNet-50 neural network structure. The first classification model is trained using the first image (labeled as not containing a region of interest) and the third image (labeled as containing a second-level region of interest) as training sets, and the output category is whether a second-level region of interest exists. The second classification model is trained using the second image (labeled as containing a first-level region of interest) and the third image (labeled as containing a second-level region of interest) as training sets, and the output category is also whether a second-level region of interest exists.

[0065] Using the above model training method, two binary classification models are trained to identify the existence of second-level regions of interest. The model with the highest classification accuracy is then selected as the final model, which can significantly improve the accuracy of image classification.

[0066] Preferably, in the process of constructing two binary classification models, the first image (labeled as not containing a region of interest) and the second image (labeled as containing a region of interest) can be used as training sets to construct the first classification model, and the output category is whether a region of interest exists; the third image (labeled as containing a first-level region of interest) and the fourth image (labeled as containing a second-level region of interest) can be used as training sets to construct the second classification model, and the output category is whether the region of interest belongs to the first level or the second level.

[0067] The two binary classification models trained using the above model training method can not only accurately identify whether a target image has a region of interest, but also identify the level of the region of interest, providing efficient, timely and accurate assistance for multidisciplinary comprehensive intervention.

[0068] Preferably, considering the uneven distribution of images between the two categories in the training set of the binary classification model, category weights are set. That is, for the first classification model, category weights are set for the first image and the third image. ； The first neural network model is trained using a training set comprised of the first and third images with adjusted class weights; the weights for the first and third images are set to 10 and 0.01, respectively; the first classification model undergoes 50 iterations. The second classification model does not use class weights and undergoes 10 iterations. The number of first and third images after weighting is balanced.

[0069] By using the above category weighting method, different weights can be assigned to the first and third images, thereby overcoming the problem of unbalanced distribution of image types in the training set.

[0070] S3: Select the classification model with the smallest generalization error from the first classification model and the second classification model as the final model.

[0071] Specifically, after obtaining the first classification model and the second classification model, the model with the highest recognition accuracy is determined as the final model through cross-validation results. The specific steps are as follows:

[0072] The training image set is randomly divided into multiple sub-image sets of equal size;

[0073] The first classification model is trained and validated multiple times to obtain multiple first error values; the average of the multiple first error values ​​is calculated to obtain the first generalization error value of the first classification model.

[0074] The second classification model is trained and validated multiple times to obtain multiple second error values; the average of the multiple second error values ​​is calculated to obtain the second generalization error value of the second classification model.

[0075] The first generalization error value and the second generalization error value are compared, and the classification model with the smallest generalization error value is selected as the target model. The model is then trained again using the training image set to obtain the final model.

[0076] In each model training and model validation, any sub-image set of the training image set is selected as the validation set, and the sub-image sets other than the validation set are used as the training set; the number of model training and model validation times is determined by the number of sub-image sets; multiple sub-image sets of equal size may include, but are not limited to, multiple sub-image sets of equal size with the number of patients.

[0077] Specifically, the model can be trained and validated five times. The training image set is divided into five equal sub-image sets. One sub-image set is used as the training set, and the other four are used as the validation set. The model is then used to train either the first or second classification model and to validate its classification performance, resulting in one error value. Each sub-image set is then used as a training set, and the above steps are performed to obtain five error values. The average of these five error values ​​(i.e., the generalization error value) is calculated. The generalization error values ​​of the first and second classification models are compared, and the model with the smaller generalization error value is selected as the target model. This target model is then retrained using the training image set to obtain the final model.

[0078] Specifically, the output categories of both the first and second classification models are images of whether there is a second-level region of interest (i.e., an area with severe lesions requiring intervention). Therefore, the neural network model with the highest recognition accuracy can be determined as the final model through the cross-validation method described above.

[0079] The image classification method provided by this invention calculates the generalization error values ​​of a first classification model and a second classification model, and selects the classification model with the smallest generalization error value as the final model for classifying target images. This method can determine the classification model with the highest classification accuracy as the final model, which can significantly improve the accuracy of image classification.

[0080] S4: Classify the target image based on the final model.

[0081] Preferably, step S4 includes: calculating the probability that each pixel value of the target image belongs to multiple categories; determining the category to which all pixel values ​​of the target image belong with the highest probability based on the probability that each pixel value belongs to each category; and taking the category to which the highest probability belongs as the category of the target image.

[0082] Specifically, the target image is the image to be detected, i.e., the patient's fundus image. Categories include: Category 1, where a second-level region of interest exists; and Category 2, where a second-level region of interest does not exist. When determining the category of the image to be detected, first, the category of each pixel in the image is determined, i.e., whether the pixel belongs to Category 1 or Category 2. For example, if pixel 1 has a 75% probability of belonging to Category 1 and a 25% probability of belonging to Category 2, then pixel 1 is determined to belong to Category 1. Then, the categories of all pixels are combined to determine the category of the entire image to be detected. For example, if 40% of the pixels in the image belong to Category 1 and 60% belong to Category 2, then the image to be detected is determined to belong to Category 2, meaning that a second-level region of interest does not exist in the image.

[0083] The image classification method provided by this invention determines the category of a target image by calculating the category probability of all pixel values ​​in the target image. This method can quickly and accurately determine the category of the image to be detected, thereby improving the efficiency of image classification.

[0084] Preferably, step S4 includes: progressively reducing the feature map of the target image layer by layer according to multiple convolutional layers and extracting features at each scale to obtain a first feature map; processing the first feature map into a second feature map using a global average pooling layer; the first feature map is a two-dimensional feature map, and the second feature map is a one-dimensional feature map; transmitting the second feature map to a fully connected layer whose output dimension is the number of classification categories to output the weights of each category; and taking the category with the highest weight as the classification result of the target image. The number of classification categories can be two types: a first category (i.e., a region of interest with a second level exists) and a second category (i.e., a region of interest with no second level exists).

[0085] Specifically, through a series of convolutional layers, activation functions, and downsampling, the input image to be detected is progressively reduced in size and features are extracted to obtain a feature map; each convolutional layer produces a corresponding feature map; the feature map is the result of the input image being convolved by the neural network, representing a feature in the neural space; its resolution depends on the stride of the previous convolutional kernel.

[0086] The image classification method provided by this invention, through multi-layer feature extraction processing and transmitting the extracted feature map to a fully connected layer whose output dimension is the number of classification categories, uses the category with the highest weight as the classification result of the target image, which can extract key features from the target image, thereby improving the accuracy of image classification.

[0087] Preferably, after step S4, the method further includes: obtaining the class probabilities obtained by the final model classifying the target image; determining the gradient of the class probabilities relative to the feature map output by the last convolutional layer of the final model; weighting each channel in the feature map according to the gradient to obtain a heatmap; and superimposing the heatmap with the target image to obtain a final heatmap that reflects the region of interest.

[0088] Specifically, when the output feature map of the last convolutional layer of the neural network is obtained, each channel of the output feature map can be weighted by the gradient of the output category (input to the channel) to obtain a heatmap. This heatmap is then superimposed on the image to be detected to obtain the final heatmap. The final heatmap indicates which regions in the image to be detected partially activate which categories. For example, regions A, B, and C of the image to be detected activate the first category (i.e., there is a second-level region of interest), while other regions of the image to be detected activate the second category (i.e., there is no second-level region of interest).

[0089] The image classification method provided by this invention obtains a final heatmap by overlaying a heatmap on the target image. It can clearly identify which regions of the target image contain regions of interest and which do not, thereby providing efficient, timely and accurate assistance for multidisciplinary intervention.

[0090] Preferably, the method further includes: labeling the image to be detected 1 (which may be a case image that has not been clinically diagnosed) as a screening group and inputting it into the trained first classification model, and outputting whether the image to be detected in the screening group contains a newly emerging second-level region of interest; labeling the image to be detected 2 (which may be a case image that has been clinically diagnosed) as a monitoring group and inputting it into the trained second classification model, and outputting whether the image to be detected in the monitoring group contains a second-level region of interest.

[0091] The image classification method provided by this invention trains a first neural network model using a first image and a third image as training sets to obtain a first classification model; trains a second neural network model using a second image and a third image as training sets to obtain a second classification model; and selects the classification model with the minimum generalization error to classify the target image, which can significantly improve the accuracy of image classification and avoid the situation where errors easily occur during long-term manual judgment, leading to a decrease in the accuracy of image classification.

[0092] Example 2

[0093] This embodiment provides an image classification device, such as Figure 2 As shown, it includes:

[0094] The annotation module is used to annotate the acquired raw images to obtain a training image set. The training image set includes a first image, a second image, and a third image. The first image is an image that does not contain a region of interest, the second image is an image that contains a region of interest with a first level, and the third image is an image that contains a region of interest with a second level.

[0095] The model training module is used to train the first neural network model using the first image and the third image as training sets to obtain the first classification model; and to train the second neural network model using the second image and the third image as training sets to obtain the second classification model.

[0096] The model selection module is used to select the classification model with the smallest generalization error from the first classification model and the second classification model as the final model;

[0097] The classification module is used to classify target images based on the final model.

[0098] Specifically, for image classification devices, hospitals can configure the devices on designated computers, granting them access to the hospital's network and connecting them to image acquisition devices via appropriate gateways to capture and transmit feedback data in real time. The specific usage method is as follows:

[0099] Use an image acquisition device (such as RetCam), install the image capture plugin, and connect it to the hospital's local area network; adjust the focal length and illumination parameters of the image acquisition device so that a clear and well-lit image to be detected (such as a retinal image) can be transmitted to the monitor. This device transmits the image to the annotation module in real time through the hospital's network.

[0100] After the acquired image to be detected is imported into the image classification device, the image to be detected is identified and classified by the classification model, that is, based on the trained neural network model.

[0101] After feature extraction and recognition calculation by the trained neural network model, the classification result of the image to be detected is finally determined, and its classification label is presented on the image to be detected. Based on the label result on the image, the presence of a second-level region of interest in the image to be detected can be correctly classified.

[0102] Preferably, the classification module includes:

[0103] The first category probability calculation unit is used to calculate the probability that each pixel value of the target image belongs to multiple categories;

[0104] The second category probability calculation unit is used to determine the category to which all pixel values ​​of the target image belong with the highest probability based on the probability that each pixel value belongs to each category;

[0105] The category determination unit is used to determine the category of the target image based on the category with the highest probability.

[0106] Preferably, the device further includes:

[0107] The category probability acquisition unit is used to acquire the category probabilities obtained by the final model in classifying the target image;

[0108] The gradient calculation unit is used to determine the gradient of the class probability with respect to the feature map output by the last convolutional layer of the final model;

[0109] The weighted processing unit is used to weight each channel in the feature map according to the gradient to obtain a heatmap;

[0110] The overlay processing unit is used to overlay the heatmap with the target image to obtain the final heatmap that reflects the region of interest.

[0111] Preferably, the model selection module includes:

[0112] A partitioning unit is used to randomly divide the training image set into multiple sub-image sets of equal size;

[0113] The first error value calculation unit is used to perform multiple model training and model validations on the first classification model to obtain multiple first error values; the average of the multiple first error values ​​is calculated to obtain the first generalization error value of the first classification model.

[0114] The second error value calculation unit is used to perform multiple model training and model validations on the second classification model to obtain multiple second error values; the average of the multiple second error values ​​is calculated to obtain the second generalization error value of the second classification model;

[0115] The comparison unit is used to compare the first generalization error value and the second generalization error value, select the classification model with the smallest generalization error value as the target model, and retrain it using the training image set to obtain the final model.

[0116] In each model training and validation cycle, any sub-image set of the training image set is selected as the validation set, and the sub-image sets other than the validation set are used as the training set; the number of model training cycles and model validation cycles is determined by the number of sub-image sets.

[0117] Preferably, the model training module includes:

[0118] Class weights are set for the first and third images; the first and third images with the adjusted class weights form a training set for training the first neural network model. The weights for the first and third images are set to 10 and 0.01, respectively; the first classification model iterates 50 times.

[0119] Preferably, the classification module includes:

[0120] The first processing unit is used to progressively reduce the feature map of the target image based on multiple convolutional layers and extract features at each scale to obtain the first feature map.

[0121] The second processing unit is used to process the first feature map into a second feature map using a global average pooling layer; the first feature map is a two-dimensional feature map, and the second feature map is a one-dimensional feature map;

[0122] The category weight output unit is used to transmit the second feature map to a fully connected layer whose output dimension is the number of categories, so as to output the weights of each category;

[0123] The classification result output unit is used to take the category with the highest weight as the classification result of the target image.

[0124] Preferably, both the original image and the target image are patient fundus images; before training the first neural network model using the first image and the third image as training sets, and before training the second neural network model using the second image and the third image as training sets, the method further includes: uniformly scaling all images in the training image set to the target size.

[0125] Example 3

[0126] The present invention also provides a computer device, such as Figure 3 As shown, it includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the methods provided in the various embodiments described above.

[0127] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the methods provided in the various embodiments described above.

[0128] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0130] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0131] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0132] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An image classification method characterized by, The method comprises the following steps: annotating the obtained original images to obtain a training image set; the training image set comprises a first image, a second image and a third image; the first image is an image not containing a region of interest, the second image is an image containing a region of interest with a first level, and the third image is an image containing a region of interest with a second level; training a first neural network model by taking the first image and the third image as a training set to obtain a first classification model, and training a second neural network model by taking the second image and the third image as a training set to obtain a second classification model; selecting a classification model with a minimum generalization error from the first classification model and the second classification model as a final model; classifying a target image based on the final model and outputting a category of whether a region of interest with the second level exists; wherein the training of the first neural network model by taking the first image and the third image as a training set comprises: performing category weight setting on the first image and the third image; forming a training set by taking the first image and the third image after the category weight setting, and training the first neural network model; after the classification of the target image based on the final model, the method further comprises the following steps: obtaining a category probability obtained by classifying the target image by using the final model; determining a gradient of the category probability with respect to a feature map output by a last convolutional layer of the final model; performing weighted processing on each channel in the feature map according to the gradient to obtain a heat map; superimposing the heat map and the target image to obtain a final heat map for reflecting a region of interest.

2. The image classification method of claim 1, wherein, The classification of the target image based on the final model comprises: calculating a probability of each pixel value of the target image belonging to multiple categories; determining a category to which a maximum probability of all pixel values of the target image belongs according to the probability of each pixel value belonging to each category; taking the category to which the maximum probability belongs as the category of the target image.

3. The image classification method of claim 1, wherein, The selection of the classification model with the minimum generalization error from the first classification model and the second classification model as the final model comprises: randomly dividing the training image set into multiple sub-image sets with equal sizes; performing multiple model training and model verification on the first classification model to obtain multiple first error values; calculating an average value of the multiple first error values to obtain a first generalization error value of the first classification model; performing multiple model training and model verification on the second classification model to obtain multiple second error values; calculating an average value of the multiple second error values to obtain a second generalization error value of the second classification model; comparing the first generalization error value and the second generalization error value, selecting a classification model with a minimum generalization error value as a target model, and performing retraining on the training image set to obtain a final model; Wherein, each time the model is trained and the model is verified, any one of the sub-image sets of the training image set is selected as a verification set, and the sub-image set other than the verification set is selected as a training set; the number of model training times and the number of model verification times are determined by the number of sub-image sets.

4. The image classification method of claim 1, wherein, The classification of the target image based on the final model comprises: The feature maps of the target image are reduced layer by layer and features at each scale are extracted by using a plurality of convolutional layers to obtain a first feature map; The first feature map is processed into a second feature map by using a global average pooling layer; the first feature map is a two-dimensional feature map, and the second feature map is a one-dimensional feature map; The second feature map is transmitted to a fully connected layer with an output dimension of a classification category number to output weights of each category; The category with the highest weight is taken as the classification result of the target image.

5. The image classification method of claim 1, wherein, The original image and the target image are both fundus images of a patient; The first neural network model is trained by using the first image and the third image as a training set; Before the second neural network model is trained by using the second image and the third image as a training set, the method further comprises: All images of the training image set are uniformly scaled to a target size.

6. An image classification apparatus characterized by comprising: Comprise: An annotation module is configured to annotate the obtained original image to obtain a training image set; the training image set comprises a first image, a second image and a third image; the first image is an image without a region of interest, the second image is an image containing a region of interest with a first level, and the third image is an image containing a region of interest with a second level; A model training module is configured to train a first neural network model by using the first image and the third image as a training set to obtain a first classification model, and train a second neural network model by using the second image and the third image as a training set to obtain a second classification model; A model selection module is configured to select a classification model with a minimum generalization error from the first classification model and the second classification model as a final model; A classification module is configured to classify a target image based on the final model and output a category of whether a region of interest with a second level exists; The model training module is specifically configured to set a category weight for the first image and the third image; the first image and the third image after the category weight setting form a training set, and the first neural network model is trained by using the training set. The image classification device further comprises: A category probability acquisition unit is configured to acquire a category probability obtained by classifying a target image by using the final model; A gradient calculation unit is configured to determine a gradient of the category probability with respect to a feature map output by a last convolutional layer of the final model; A weighting processing unit is configured to perform weighting processing on each channel in the feature map according to the gradient to obtain a heat map; A superimposition processing unit is configured to superimpose the heat map with the target image to obtain a final heat map reflecting a region of interest.

7. A computer device comprising a memory and a processor, the memory storing a computer program operable on the processor, characterized in that, The processor executes the computer program to implement the steps of the image classification method in any one of claims 1 to 5.

8. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program, which is executed by a processor, implements the steps of the image classification method according to any one of claims 1 to 5.

Images