Image classification method, device, equipment and computer storage medium

Abstract

The embodiment of the present invention provides an image classification method, device, equipment and computer storage medium. The method includes: obtaining a training image; performing convolution calculation on the training image based on the first convolutional neural network layer in the convolutional neural network to obtain a first feature map; performing convolution calculation on the first feature map based on the second convolutional neural network layer in the convolutional neural network to obtain a second feature map; splicing the second feature map, the pre-acquired naked eye vision vector and the pre-acquired corrected vision vector based on the third convolutional neural network layer in the convolutional neural network to obtain a five-dimensional vector; training the convolutional neural network based on the five-dimensional vector to obtain an image classification model. The model can not only extract feature information from the image, but also use the patient's corrected vision and naked eye vision as auxiliary diagnostic factors, integrate the relevant data of the image and the patient, and improve the classification accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to an image classification method, device, equipment and computer storage medium. Background Art

[0002] The optic disc, whose full name is the optic nerve disc, is a light red disc-shaped structure with a diameter of about 1.5 mm and clear boundaries located about 3 mm from the macula to the nasal side of the retina. It is called the optic nerve disc, or optic disc for short.

[0003] When patients seek medical treatment for eye diseases, doctors mainly use their own medical experience to manually detect the patient's symptoms. This detection method is not only inefficient, but the doctor's medical experience also has a certain degree of influence on the diagnosis results. Especially for some extremely rare optic disc diseases, the probability of misdiagnosis will increase accordingly.

[0004] In order to assist doctors to better diagnose eye diseases, in the prior art, the optic disc of the eye is segmented and located, and the relative position of the optic disc is found from the image, but the optic disc disease is not classified and diagnosed, which cannot well assist doctors to diagnose the disease.

[0005] Therefore, an algorithm is needed to improve detection efficiency and assist doctors in better diagnosing related diseases. Summary of the invention

[0006] The embodiments of the present invention provide an image classification method, apparatus, device and computer storage medium, which can classify optic disc diseases by acquiring optic disc images, better assist doctors in diagnosing optic disc diseases, improve detection efficiency and reduce misdiagnosis rate.

[0007] In a first aspect, an embodiment of the present invention provides an image classification model training method, the method comprising:

[0008] Get training images;

[0009] Performing convolution calculation on the training image based on a first convolutional neural network layer in the convolutional neural network to obtain a first feature map;

[0010] Performing a convolution calculation on the first feature map based on a second convolutional neural network layer in the convolutional neural network to obtain a second feature map, wherein the convolution calculation is used to perform feature map cropping on the first feature map, and the second feature map is a feature map after the first feature map is cropped;

[0011] splicing the second feature map, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector based on the third convolutional neural network layer in the convolutional neural network to obtain a five-dimensional vector;

[0012] The convolutional neural network is trained based on the five-dimensional vector to obtain the image classification model.

[0013] In an optional implementation, the third convolutional neural network layer includes a global average pooling layer, a fully connected layer, a softmax function, and a sigmoid function; based on the third convolutional neural network layer in the convolutional neural network, the second feature map, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector are spliced ​​to obtain a five-dimensional vector, including:

[0014] Extracting a second feature map vector of the second feature map based on a third convolutional neural network layer in the convolutional neural network;

[0015] The second feature map vector, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector are concatenated to obtain a first vector;

[0016] The first vector is passed into the global average pooling layer for global average pooling processing to obtain a second vector;

[0017] The second vector is successively passed into the fully connected layer, the softmax function, and the sigmoid function to obtain the five-dimensional vector.

[0018] In an optional implementation, the second convolutional neural network layer includes at least one first convolutional layer with a first preset step size, and at least one second convolutional layer with a second preset step size.

[0019] In an optional implementation, the first convolutional neural network layer includes at least one third convolutional layer with a third preset step size, and at least one maximum pooling layer with a preset size, and the third convolutional layer and the maximum pooling layer are alternately connected.

[0020] In an optional implementation, the training image is an image of a size of 224*224*3, and the first feature map is a feature map of a size of 28*28*1024.

[0021] In an optional implementation, the second feature map includes two feature maps of a size of 28*28*128.

[0022] In a second aspect, an embodiment of the present invention provides an image classification method, the method comprising:

[0023] Get the image to be processed;

[0024] The image to be processed is classified based on the image classification model to obtain a classification result, wherein the image classification model is obtained based on the image classification model training method in the first aspect or any one of the implementations of the first aspect.

[0025] In an optional implementation, classifying the image to be processed based on the image classification model to obtain a classification result includes:

[0026] Processing the image to be processed by bilinear interpolation and Gaussian low-pass filter to obtain a first image to be processed;

[0027] Performing size normalization preprocessing on the first image to be processed to obtain a second image to be processed;

[0028] Performing convolution calculation on the second image to be processed to obtain a cropped second image to be processed;

[0029] The cropped second image to be processed is classified based on the image classification model to obtain a classification result.

[0030] In a third aspect, an embodiment of the present invention provides an image classification model training device, the device comprising:

[0031] An acquisition module, used for acquiring training images;

[0032] A first convolution module, used to perform convolution calculation on the training image based on a first convolutional neural network layer in the convolutional neural network to obtain a first feature map;

[0033] A second convolution module, used for performing convolution calculation on the first feature map based on a second convolution neural network layer in the convolution neural network to obtain a second feature map, wherein the convolution calculation is used to perform feature map cropping on the first feature map, and the second feature map is a feature map after the first feature map is cropped;

[0034] A splicing module, used for splicing the second feature map, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector based on the third convolutional neural network layer in the convolutional neural network to obtain a five-dimensional vector;

[0035] A training module is used to train the convolutional neural network based on the five-dimensional vector to obtain an image classification model;

[0036] Among them, the first convolutional neural network layer in the convolutional neural network is connected to the second convolutional neural network layer, and the second convolutional neural network layer is connected to the third convolutional neural network layer.

[0037] In a fourth aspect, an embodiment of the present invention provides an image classification device, the device comprising:

[0038] An acquisition module, used for acquiring an image to be processed;

[0039] A classification module is used to classify the image to be processed based on an image classification model to obtain a classification result, wherein the image classification model is obtained based on the image classification model training method in the first aspect or any one of the implementations of the first aspect.

[0040] In a fifth aspect, an embodiment of the present invention provides an image classification device, the image classification device comprising: a processor, and a memory storing computer program instructions;

[0041] The processor reads and executes computer program instructions to implement the image classification model training method in the first aspect or any one of the implementations of the first aspect, or the processor reads and executes the computer program instructions to implement the image classification method in the second aspect or any one of the implementations of the second aspect.

[0042] In a sixth aspect, an embodiment of the present invention provides a computer storage medium having computer program instructions stored thereon, wherein when the computer program instructions are executed by a processor, the image classification model training method in the first aspect or any one of the implementations of the first aspect is implemented; or, when the computer program instructions are executed by a processor, the image classification method in the second aspect or any one of the implementations of the second aspect is implemented.

[0043] The embodiment of the present invention provides an image classification model training method, an image classification method, an apparatus, a device and a computer storage medium. Among them, the image classification model training method and apparatus of the embodiment of the present invention can perform convolution calculation on the training image based on the first convolutional neural network layer in the convolutional neural network to obtain a first feature map, perform convolution calculation on the first feature map based on the second convolutional neural network layer in the convolutional neural network to obtain a second feature map, wherein the convolution calculation is used to perform feature map cropping on the first feature map, and the second feature map is a feature map after the first feature map is cropped, and the second feature map, a pre-acquired naked eye vision vector and a pre-acquired corrected vision vector are spliced ​​based on the third convolutional neural network layer in the convolutional neural network to obtain a five-dimensional vector, and the convolutional neural network is trained based on the five-dimensional vector to obtain an image classification model, which can not only extract feature information in the image, but also use the patient's corrected vision and naked eye vision as auxiliary diagnostic factors, and integrate the relevant data of the image and the patient to improve the classification accuracy. The image classification method and device of the embodiments of the present invention can classify the image to be processed based on the image classification model to obtain the classification result. The classification result of the image to be processed is generated by the model, which can better assist doctors in diagnosing optic disc diseases, improve detection efficiency, and reduce misdiagnosis rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solution of the embodiment of the present invention, the following is a brief introduction to the drawings required for use in the embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1It is a flowchart of an image classification model training method provided by an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of the structure of a convolutional neural network provided by an embodiment of the present invention;

[0047] Figure 3 is a flowchart of an image classification method provided by an embodiment of the present invention;

[0048] Figure 4 is a schematic diagram of a process of image saliency detection provided by an embodiment of the present invention;

[0049] Figure 5 It is a structural schematic diagram of an image classification model training device provided by an embodiment of the present invention;

[0050] Figure 6 is a structural schematic diagram of an image classification device provided by an embodiment of the present invention;

[0051] Figure 7 It is a structural schematic diagram of an image classification device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0052] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0053] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0054] At present, the classification and screening algorithms for fundus image diseases in the existing technology are mainly aimed at common eye diseases such as fundus macular degeneration, diabetic retinopathy, glaucoma, etc., but cannot diagnose well the optic disc diseases such as optic disc edema, macular membrane deformation, morning glory syndrome, invisible optic disc, pre-optic disc membrane defect, etc. Moreover, the feature extraction of optic disc images is mainly carried out through target detection or semantic segmentation methods, which is easily interfered by the fundus vascular area.

[0055] In order to solve the problems of the prior art, the embodiments of the present invention provide an image classification model training method, an image classification method, an apparatus, a device and a computer storage medium.

[0056] First, the image classification model training method of the embodiment of the present invention can perform convolution calculation on the training image based on the first convolutional neural network layer in the convolutional neural network to obtain a first feature map, and perform convolution calculation on the first feature map based on the second convolutional neural network layer in the convolutional neural network to obtain a second feature map, wherein the convolution calculation is used to perform feature map cropping on the first feature map, and the second feature map is the feature map after the first feature map is cropped. Based on the third convolutional neural network layer in the convolutional neural network, the second feature map, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector are spliced ​​to obtain a five-dimensional vector, and the convolutional neural network is trained based on the five-dimensional vector to obtain an image classification model. This model can not only extract feature information from the image, but also use the patient's corrected vision and naked eye vision as auxiliary diagnostic factors, and integrate the relevant data of the image and the patient to improve the classification accuracy.

[0057] Moreover, the image classification method of the embodiment of the present invention can classify the image to be processed based on the image classification model to obtain the classification result. The classification result of the image to be processed is generated by the model, which can better assist doctors in diagnosing optic disc diseases, improve detection efficiency, and reduce misdiagnosis rate.

[0058] Since the image classification method provided by the embodiment of the present invention requires the use of an image classification model, the image classification model training method provided by the embodiment of the present invention is first introduced below.

[0059] Figure 1 FIG. 1 is a flow chart of an image classification model training method provided by an embodiment of the present invention. Figure 1 As shown, the method may include the following steps:

[0060] S110: Obtain training images.

[0061] The original image may be preprocessed, and the preprocessed image may be used as a training image for subsequent training. The training image is a three-channel red, green, and blue (RGB) image.

[0062] S120. Perform convolution calculation on the training image based on the first convolutional neural network layer in the convolutional neural network to obtain a first feature map.

[0063] The image classification model is trained by convolutional neural network, such as Figure 2 As shown, the convolutional neural network includes three convolutional neural network layers, namely the first convolutional neural network layer, the second convolutional neural network layer, and the third convolutional neural network layer.

[0064] In one embodiment, the first convolutional neural network layer includes at least one third convolutional layer with a third preset step size, and at least one maximum pooling layer with a preset size, and the third convolutional layer and the maximum pooling layer are alternately connected.

[0065] Among them, at least one third convolutional layer with a third preset step size includes at least one convolutional layer with a step size of 1, and at least one convolutional layer with a step size of 2, and the maximum pooling layer with a preset size is a maximum pooling layer with a size of 2*2.

[0066] It can be understood that the convolution layer (conv)-XY convolution layer represents a convolution layer including Y convolution kernels, each of which has a size of X*X.

[0067] Specifically, in one embodiment, Figure 2 As shown in the figure, the first convolutional neural network layer includes 2 conv-364, 1 first maximum pooling layer of 2*2 size, 2 conv-3 128, 1 second maximum pooling layer of 2*2 size, 3 conv-3 256, 1 third maximum pooling layer of 2*2 size, 3 conv-3 512, and 1 first conv-3 1024. Among them, the step size of conv-3 64, conv-3128, conv-3 256, and conv-3 512 are all 1, and the step size of the first conv-3 1024 is 2.

[0068] In one embodiment, the training image is an image of size 224*224*3, and the first feature map is a feature map of size 28*28*1024.

[0069] When the convolutional neural network inputs a 224*224 RGB image, it is first passed into conv-364, in which 64 convolution kernels are sampled. After two consecutive layers of the same convolution layer sampling, the output feature map is 224*224*64.

[0070] After two conv-3 64 convolution calculations, it is passed to the first maximum pooling layer of size 2*2, and a 2*2 pooling operation is performed to output a feature map of size 112*112*64, which is passed to the next layer of conv-3 128.

[0071] When a feature map of size 112*112*64 is passed into conv-3 128, 128 convolution kernels are sampled in this layer. After two consecutive layers of the same convolution layer sampling, a feature map of size 112*112*128 is output.

[0072] After two conv-3 128 convolution calculations, it is passed to the second maximum pooling layer of size 2*2, and a 2*2 pooling operation is performed to output a feature map of size 56*56*128, which is passed to the next layer of conv-3 256.

[0073] When a feature map of size 56*56*128 is passed into conv-3 256, 256 convolution kernels are sampled in this layer. After three consecutive layers of the same convolution layer sampling, a feature map of size 56*56*256 is output.

[0074] After three conv-3 256 convolution calculations, it is passed to the third maximum pooling layer of size 2*2, and a 2*2 pooling operation is performed to output a feature map of size 28*28*256, which is passed to the next layer of conv-3 512.

[0075] When the feature map of size 28*28*256 is passed into the conv-3 512 convolution layer, 512 convolution kernels are sampled in this layer. After three consecutive layers of the same convolution layer sampling, the feature map of size 56*56*512 is output and passed into the first conv-3 1024 of the next layer.

[0076] When the feature map of size 28*28*512 is passed into the first conv-3 1024, 1024 convolution kernels are sampled in this layer, and the first feature map of size 28*28*1024 is output.

[0077] S130. Perform convolution calculation on the first feature map based on the second convolutional neural network layer in the convolutional neural network to obtain a second feature map, wherein the convolution calculation is used to perform feature map cropping on the first feature map, and the second feature map is a feature map after the first feature map is cropped.

[0078] After obtaining the first feature map, the first convolutional neural network layer transmits the first feature map to the second convolutional neural network layer, and the second convolutional neural network layer performs convolution calculation on the first feature map to obtain the second feature map. The convolution calculation in this step mainly performs feature map cropping on the first feature, reduces the size of the feature map, and completes the extraction of image feature information. The second feature map is the feature map after the first feature map is cropped.

[0079] In one embodiment, the second feature map includes two feature maps of size 28*28*128.

[0080] The second feature map also includes a feature map with a size of 28*28*1024, a feature map with a size of 28*28*512, and a feature map with a size of 28*28*256.

[0081] In one embodiment, the second convolutional neural network layer includes at least one first convolutional layer with a first preset step size, and at least one second convolutional layer with a second preset step size.

[0082] The first preset step size is 1, and the second preset step size is 2. Figure 2 As shown, the first convolutional layer includes the second conv-3 1024, the second conv-3 512, the second conv-3 256, and the second conv-3 128, all of which have a step size of 1, and the second convolutional layer includes the first conv-3 512, the first conv-3256, and the first conv-3 128, all of which have a step size of 2.

[0083] The first convolutional neural network layer inputs the first feature map into the second convolutional neural network layer, which is then input into conv-31024 for convolution calculation to obtain a feature map of size 28*28*1024, which is then passed into conv-3 512 and the third convolutional neural network layer respectively.

[0084] The feature map of size 28*28*1024 is passed into the first conv-3 512 for convolution calculation, and the obtained feature map is further passed into the second conv-3 512 for convolution calculation to obtain a feature map of size 28*28*512, which is passed into the first conv-3 256 and the third convolutional neural network layer respectively.

[0085] The feature map of size 28*28*512 is passed into the first conv-3 256 for convolution calculation, and the obtained feature map is further passed into the second conv-3 256 for convolution calculation to obtain a feature map of size 28*28*256, which is passed into the first conv-3 128 and the third convolutional neural network layer respectively.

[0086] The feature map of size 28*28*256 is passed into the first conv-3 128 for convolution calculation, and the fourth feature map of size 28*28*128 is obtained, which is passed into the second conv-3 128 and the third convolutional neural network respectively. The fourth feature map is convolved by the second conv-3128 to obtain the fifth feature map of size 28*28*128, which is passed into the third convolutional neural network. In other words, after the feature map of size 28*28*256 is passed into the first conv-3 128 and the second conv-3 128 respectively, two feature maps of size 28*28*128 are obtained.

[0087] S140, based on the third convolutional neural network layer in the convolutional neural network, concatenate the second feature map, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector to obtain a five-dimensional vector.

[0088] The second convolutional neural network inputs the output second feature map into the third convolutional neural network layer, and at the same time, inputs the pre-acquired naked eye vision vector and the pre-acquired corrected vision vector into the third convolutional neural network layer. The third convolutional neural network layer concatenates the second feature map, the naked eye vision vector, and the corrected vision vector to obtain a five-dimensional vector. Since the naked eye vision and the corrected vision are used as auxiliary diagnostic factors, the accuracy of classification is improved.

[0089] In one embodiment, the third convolutional neural network layer includes a global average pooling layer, a fully connected layer, a softmax function, and a sigmoid function; based on the third convolutional neural network layer in the convolutional neural network, the second feature map, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector are spliced ​​to obtain a five-dimensional vector, including:

[0090] Extracting a second feature map vector of the second feature map based on a third convolutional neural network layer in the convolutional neural network;

[0091] The second feature map vector, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector are concatenated to obtain a first vector;

[0092] The first vector is passed into the global average pooling layer for global average pooling processing to obtain a second vector;

[0093] The second vector is successively passed into the fully connected layer, the softmax function, and the sigmoid function to obtain the five-dimensional vector.

[0094] like Figure 2 As shown, the third convolutional neural network layer also includes a Flatten function and a Concat function.

[0095] After the second feature map is passed into the third convolutional neural network layer, first, the feature vector of the second feature map is extracted through the Flatten function to obtain the N-dimensional vector of the second feature map, where N is the size of the second feature map. For example, if the size of the second feature map is X*Y*Z, then N=X*Y*Z.

[0096] The vision vector is input to the third convolutional neural network layer, where the vision vector includes the pre-acquired naked eye vision vector and the pre-acquired corrected vision vector. The obtained N-dimensional vector and the vision vector are concatenated by the concat function to form an N+2-dimensional vector, which is passed to the global average pooling layer for global average pooling to obtain a new vector, which is then passed to the fully connected layer, softmax function, and sigmoid function. After activation by the sigmoid function, a five-dimensional vector can be obtained, which is the confidence of five types of optic disc abnormalities, including optic disc edema, spotty membrane deformation, morning glory syndrome, invisible optic disc, and pre-optic disc membrane defect.

[0097] S150. Train the convolutional neural network based on the five-dimensional vector to obtain an image classification model.

[0098] The image classification model trained by the method of steps S110-S150 not only performs relevant classification based on the feature information of the image, but also considers the auxiliary role of the patient's corrected vision and naked eye vision in the classification of optic disc abnormalities, and adds the patient's corrected vision and naked eye vision as parameters to the feature vector of the image. After the spliced ​​feature vector is activated by the sigmoid function, the confidence for disease classification is obtained. The relevant data of the comprehensive image and the patient's vision are added as training data to the training of the convolutional neural network, which improves the accuracy of image classification.

[0099] The image classification model training method of the embodiment of the present invention can perform convolution calculation on the training image based on the first convolutional neural network layer in the convolutional neural network to obtain a first feature map, and perform convolution calculation on the first feature map based on the second convolutional neural network layer in the convolutional neural network to obtain a second feature map, wherein the convolution calculation is used to perform feature map cropping on the first feature map, and the second feature map is the feature map after the first feature map is cropped. Based on the third convolutional neural network layer in the convolutional neural network, the second feature map, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector are spliced ​​to obtain a five-dimensional vector, and the convolutional neural network is trained based on the five-dimensional vector to obtain an image classification model. This model can not only extract feature information from the image, but also use the patient's corrected vision and naked eye vision as auxiliary diagnostic factors, and integrate the relevant data of the image and the patient to improve the classification accuracy.

[0100] Based on the image classification model training method of the embodiment of the present invention, the embodiment of the present invention also provides an image classification method. Figure 3 FIG. 1 is a flow chart of an image classification method provided by an embodiment of the present invention. Figure 3 As shown, the method may include the following steps:

[0101] S310: Obtain an image to be processed.

[0102] S320: Classify the image to be processed based on the image classification model to obtain a classification result.

[0103] During the image acquisition process, the image is often affected by external conditions, resulting in uneven grayscale distribution in the acquired image. In addition, the contrast between the optic disc image and the background is low, so preprocessing is required before feature extraction. In the process of image feature extraction, the quality of this step can directly affect the effect of subsequent processing.

[0104] In one embodiment, classifying the image to be processed based on the image classification model to obtain a classification result includes:

[0105] Processing the image to be processed by bilinear interpolation and Gaussian low-pass filter to obtain a first image to be processed;

[0106] Performing size normalization preprocessing on the first image to be processed to obtain a second image to be processed;

[0107] Performing convolution calculation on the second image to be processed to obtain a cropped second image to be processed;

[0108] The cropped second image to be processed is classified based on the image classification model to obtain a classification result.

[0109] Since different imaging devices may have random errors due to imaging angle, perspective relationship and even the lens itself during shooting, the acquired images to be processed are preprocessed to remove images with lens stains and poor exposure quality. After obtaining a relatively clear image, saliency detection is performed on the optic disc to better extract image feature information.

[0110] Firstly, bilinear interpolation and Gaussian low-pass filtering are used to pre-process the image to be processed, so as to highlight the main body of the image and obtain the first image to be processed.

[0111] Secondly, by performing size normalization processing on the image to be processed, a second image to be processed with a size reduced to 336*336*3 can be obtained.

[0112] Then, if Figure 4As shown, the second image to be processed is passed into the convolution layer for convolution calculation. After the feature extraction of the convolution layer, the feature map of the second image to be processed is passed into the candidate region of interest (ROI) pooling layer. The ROI pooling layer uses the maximum pooling to convert the features of any region of the ROI in the feature map of the second image to be processed into a fixed feature map of size 224*224. After passing through the fully connected layer and the softmax function, the optic disc saliency image of size 224*224*3 is output. That is to say, after the optic disc saliency detection, the input image of size 336*336*3 is cropped to an image of size 224*224*3.

[0113] Finally, the image classification model classifies the input optic disc saliency image of size 224*224*3 to obtain the classification result. Since the image classification method of the embodiment of the present invention is mainly used to diagnose diseases in the optic disc area, including 5 types of optic disc abnormalities, namely optic disc edema, macular membrane deformation, morning glory syndrome, invisible optic disc, and pre-optic disc membrane defect, it is different from the existing method for classifying eye diseases such as diabetic retinopathy, macular coloboma, glaucoma, and pathological myopia, and the types of diseases targeted are different. In addition, focusing only on the classification of optic disc diseases can obtain higher specificity and sensitivity.

[0114] The image classification method of the embodiment of the present invention can classify the image to be processed based on the image classification model to obtain the classification result. The classification result of the image to be processed is generated by the model, which can better assist doctors in diagnosing optic disc diseases, improve detection efficiency, and reduce misdiagnosis rate.

[0115] Figure 5 Schematic diagram of the structure of an image classification model training device provided by an embodiment of the present invention. Figure 5 As shown, the image classification model training device 500 may include an acquisition module 510, a first convolution module 520, a second convolution module 530, a splicing module 540 and a training module 550.

[0116] The acquisition module 510 is used to acquire a training image.

[0117] The first convolution module 520 is used to perform convolution calculation on the training image based on the first convolutional neural network layer in the convolutional neural network to obtain a first feature map.

[0118] The second convolution module 530 is used to perform convolution calculation on the first feature map based on the second convolutional neural network layer in the convolutional neural network to obtain a second feature map, wherein the convolution calculation is used to perform feature map cropping on the first feature map, and the second feature map is the feature map after the first feature map is cropped.

[0119] The splicing module 540 is used to splice the second feature map, the pre-acquired naked eye vision vector and the pre-acquired corrected vision vector based on the third convolutional neural network layer in the convolutional neural network to obtain a five-dimensional vector.

[0120] The training module 550 is used to train the convolutional neural network based on the five-dimensional vector to obtain an image classification model.

[0121] It is understandable that the image classification model training device 500 of the embodiment of the present invention may correspond to the embodiment of the present invention. Figure 1 The execution subject of the image classification model training method in the image classification model training device 500, the specific details of the operation and / or function of each module of the image classification model training device 500 can refer to the above-mentioned embodiment of the present invention. Figure 1 The description of the corresponding parts in the image classification model training method is not repeated here for the sake of brevity.

[0122] Figure 6 is a schematic diagram of the structure of an image classification device provided by an embodiment of the present invention, such as Figure 6 As shown, the image classification device 600 may include an acquisition module 610 and a classification module 620 .

[0123] The acquisition module 610 is used to acquire the image to be processed.

[0124] The classification module 620 is used to classify the image to be processed based on the image classification model to obtain a classification result. Figure 1 The image classification model training method is obtained.

[0125] The image classification method of the embodiment of the present invention can classify the image to be processed based on the image classification model to obtain the classification result. The classification result of the image to be processed is generated by the model, which can better assist doctors in diagnosing optic disc diseases, improve detection efficiency, and reduce misdiagnosis rate.

[0126] It can be understood that the image classification device 600 of the embodiment of the present invention may correspond to the image classification device 600 of the embodiment of the present invention. Figure 3 The specific details of the operations and / or functions of each module of the image classification device 600 can be found in the above-mentioned embodiment of the present invention. Figure 3 The description of the corresponding parts in the image classification method is not repeated here for the sake of brevity.

[0127] Figure 7 A schematic diagram of the hardware structure of an image classification device provided by an embodiment of the present invention is shown.

[0128] The image classification device may include a processor 701 and a memory 702 storing computer program instructions.

[0129] Specifically, the processor 701 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.

[0130] The memory 702 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 702 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In one example, the memory 702 may include a removable or non-removable (or fixed) medium, or the memory 702 is a non-volatile solid-state memory. The memory 702 may be inside or outside the integrated gateway disaster recovery device.

[0131] In one example, the memory 702 may be a read-only memory (ROM). In one example, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0132] The processor 701 reads and executes the computer program instructions stored in the memory 702 to implement Figure 1 Steps S110 to S150 in the embodiment shown in the figure are performed, and Figure 1 The corresponding technical effects achieved by executing the steps of the example shown, or by realizing Figure 3 Steps S310 to S320 in the embodiment shown in the figure are performed, and Figure 3 The corresponding technical effects achieved by executing the steps of the example shown are not repeated here for the sake of brevity.

[0133] In one example, the image classification device may further include a communication interface 703 and a bus 710. Figure 7 As shown, the processor 701, the memory 702, and the communication interface 703 are connected via a bus 710 and communicate with each other.

[0134] The communication interface 703 is mainly used to implement the communication between the modules, devices, units and / or equipment in the embodiment of the present invention.

[0135] Bus 710 includes hardware, software or both, and the components of online data flow billing equipment are coupled to each other. For example, but not limitation, the bus may include Accelerated Graphics Port (AGP) or other graphics bus, Enhanced Industry Standard Architecture (EISA) bus, Front Side Bus (FSB), Hyper Transport (HT) interconnection, Industry Standard Architecture (ISA) bus, InfiniBand interconnection, Low Pin Count (LPC) bus, Memory bus, Micro Channel Architecture (MCA) bus, Peripheral Component Interconnect (PCI) bus, PCI-Express (PCI-X) bus, Serial Advanced Technology Attachment (SATA) bus, Video Electronics Standards Association Local (VLB) bus or other suitable bus or two or more of these combinations. In appropriate cases, bus 710 may include one or more buses. Although the embodiment of the present invention describes and shows a specific bus, the present invention considers any suitable bus or interconnection.

[0136] The image classification device can execute the image classification model training method or image classification method in the embodiment of the present invention, thereby realizing the combination of Figure 1 The image classification model training method described in this paper can be combined with Figure 3 Describe the image classification method.

[0137] In addition, in combination with the image classification model training method or image classification method in the above embodiments, the embodiment of the present invention can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, any one of the image classification model training method or image classification method in the above embodiments is implemented.

[0138] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0139] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), appropriate firmware, plug-in, function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (Radio Frequency, RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0140] It should also be noted that the exemplary embodiments mentioned in the present invention describe some methods or systems based on a series of steps or devices. However, the present invention is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiments, or in a different order from the embodiments, or several steps can be performed simultaneously.

[0141] The above is only a specific implementation of the present invention. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the system, module and unit described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be covered within the protection scope of the present invention.

Claims

1. A method for training an image classification model, characterized in that: The method comprises: Get training images; Performing convolution calculation on the training image based on a first convolutional neural network layer in the convolutional neural network to obtain a first feature map; Performing a convolution calculation on the first feature map based on a second convolutional neural network layer in the convolutional neural network to obtain a second feature map, wherein the convolution calculation is used to perform feature map cropping on the first feature map, the second feature map is a feature map after the first feature map is cropped, and the second feature map is a three-dimensional feature map; splicing the second feature map, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector based on the third convolutional neural network layer in the convolutional neural network to obtain a five-dimensional vector; Training the convolutional neural network based on the five-dimensional vector to obtain an image classification model; The third convolutional neural network layer includes a global average pooling layer, a fully connected layer, a softmax function, and a sigmoid function; the third convolutional neural network layer based on the convolutional neural network splices the second feature map, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector to obtain a five-dimensional vector, including: Extracting a second feature map vector of the second feature map based on a third convolutional neural network layer in the convolutional neural network; The second feature map vector, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector are concatenated to obtain a first vector; Passing the first vector into the global average pooling layer for global average pooling processing to obtain a second vector; The second vector is successively passed into the fully connected layer, the softmax function, and the sigmoid function to obtain the five-dimensional vector.

2. The method according to claim 1, characterized in that The second convolutional neural network layer includes at least one first convolutional layer with a first preset step size, and at least one second convolutional layer with a second preset step size.

3. The method according to claim 1, characterized in that The first convolutional neural network layer includes at least one third convolutional layer with a third preset step size, and at least one maximum pooling layer with a preset size, and the third convolutional layer is alternately connected with the maximum pooling layer.

4. The method according to claim 1, characterized in that: The training image is an image of size 224*224*3, and the first feature map is a feature map of size 28*28*1024.

5. The method according to claim 1, characterized in that The second feature map includes two feature maps of size 28*28*128.

6. An image classification method, characterized in that: The method comprises: Get the image to be processed; The image to be processed is classified based on the image classification model to obtain a classification result, wherein the image classification model is obtained based on the image classification model training method described in any one of claims 1 to 5.

7. The method according to claim 6, characterized in that The step of classifying the image to be processed based on the image classification model to obtain a classification result includes: Processing the image to be processed by bilinear interpolation and Gaussian low-pass filter to obtain a first image to be processed; Performing size normalization preprocessing on the first image to be processed to obtain a second image to be processed; Performing convolution calculation on the second image to be processed to obtain a cropped second image to be processed; The cropped second image to be processed is classified based on the image classification model to obtain a classification result.

8. An image classification model training device, characterized in that: The device comprises: An acquisition module, used for acquiring training images; A first convolution module, used for performing convolution calculation on the training image based on a first convolutional neural network layer in a convolutional neural network to obtain a first feature map; A second convolution module, used for performing convolution calculation on the first feature map based on the second convolution neural network layer in the convolution neural network to obtain a second feature map, wherein the convolution calculation is used to perform feature map cropping on the first feature map, the second feature map is the feature map after the first feature map is cropped, and the second feature map is a three-dimensional feature map; A splicing module, configured to splice the second feature map, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector based on the third convolutional neural network layer in the convolutional neural network to obtain a five-dimensional vector; A training module, used for training the convolutional neural network based on the five-dimensional vector to obtain an image classification model; The first convolutional neural network layer in the convolutional neural network is connected to the second convolutional neural network layer, and the second convolutional neural network layer is connected to the third convolutional neural network layer; The third convolutional neural network layer includes a global average pooling layer, a fully connected layer, a softmax function, and a sigmoid function; the third convolutional neural network layer based on the convolutional neural network splices the second feature map, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector to obtain a five-dimensional vector, including: Extracting a second feature map vector of the second feature map based on a third convolutional neural network layer in the convolutional neural network; The second feature map vector, the pre-acquired naked eye vision vector, and the pre-acquired corrected vision vector are concatenated to obtain a first vector; Passing the first vector into the global average pooling layer for global average pooling processing to obtain a second vector; The second vector is successively passed into the fully connected layer, the softmax function, and the sigmoid function to obtain the five-dimensional vector.

9. An image classification device, characterized in that: The device comprises: An acquisition module, used for acquiring an image to be processed; A classification module is used to classify the image to be processed based on the image classification model to obtain a classification result, wherein the image classification model is obtained based on the image classification model training method described in any one of claims 1-5.

10. An image classification device, characterized in that: The image classification device includes: a processor, and a memory storing computer program instructions; The processor reads and executes the computer program instructions to implement the image classification model training method as described in any one of claims 1-5, or the processor reads and executes the computer program instructions to implement the image classification method as described in any one of claims 6-7.

11. A computer storage medium, characterized in that: The computer storage medium stores computer program instructions, which, when executed by a processor, implement the image classification model training method as described in any one of claims 1 to 5, or, when executed by a processor, implement the image classification method as described in claims 6 to 7.

Images