Training method and recognition method of image recognition model, equipment and storage medium

Abstract

The invention discloses a training method, a recognition method and equipment of an image recognition model and a storage medium. The training method comprises the following steps: acquiring a plurality of training eye fundus images; performing data amplification on the training eye fundus image to obtain an amplified image; and in each iteration, inputting the amplified image into the image recognition model to obtain a corresponding prediction result comprising feature mapping, obtaining a first loss function based on the prediction results obtained by the historical model and the current model, and optimizing the image recognition model by using the first loss function, the historical model is an image recognition model obtained by any previous optimization of the current model; a first loss function is calculated based on feature maps of at least two augmented images of the historical model and the current model, the feature maps consisting of a plurality of feature maps with the augmented images. Therefore, the training method, the recognition method, the equipment and the storage medium of the image recognition model, which can improve the recognition consistency of the fundus images, are provided.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing based on artificial intelligence, and in particular to a training method, a recognition method, a device and a storage medium for an image recognition model. Background Art

[0002] With the development of artificial intelligence technology, it is becoming more and more common in the medical field to analyze fundus images of patients through fundus recognition systems. Fundus recognition systems generally refer to image recognition models that can identify lesions in fundus images using artificial intelligence technology.

[0003] At present, the training method of the image recognition model generally uses the fundus image and the real label corresponding to the fundus image as the supervision signal to train the image recognition model. Specifically, the parameters of the image recognition model are optimized by calculating the difference between the predicted result of the fundus image and the result corresponding to the real label.

[0004] However, during the acquisition of fundus images, some factors (such as the acquisition environment or the experience of the acquisition personnel, etc.) have a great impact on the acquisition of fundus images, resulting in large variations in the fundus images of the same fundus (such as different shooting angles or shooting brightness of the fundus images, etc.). The image recognition model obtained by the existing training method may obtain different results for the recognition of fundus images of the same fundus obtained under different factors. Therefore, the recognition consistency of the recognition algorithm for images obtained under different factors of the same fundus needs to be improved. Summary of the invention

[0005] The present disclosure is proposed in view of the above-mentioned situation, and its purpose is to provide a training method, a recognition method, a device and a storage medium of an image recognition model that can improve the recognition consistency of fundus images.

[0006] To this end, the first aspect of the present disclosure provides a training method for an image recognition model, comprising: acquiring multiple training fundus images; performing data augmentation on the training fundus images to obtain augmented images; and in each iteration, inputting the augmented images into the image recognition model to obtain corresponding prediction results including feature maps, obtaining a first loss function based on the prediction results obtained by a historical model and a current model, and optimizing the image recognition model using the first loss function, wherein the historical model is an image recognition model obtained by any previous optimization of the current model; calculating the first loss function based on feature maps of at least two augmented images corresponding to the same training fundus image obtained by the historical model and the current model, the feature maps consisting of multiple feature maps with the augmented images.

[0007] In the first aspect of the present disclosure, by performing data augmentation on the acquired training fundus images, the data diversity of the fundus images in the training set and the generalization ability of the image recognition model can be improved, thereby improving the recognition consistency of the image recognition model for fundus images of different forms based on the same fundus. In addition, by calculating the first loss function through the feature map related to the same training fundus image to optimize the image recognition model, the similarity of the feature map related to the same training fundus image can be improved, thereby improving the consistency of the prediction results of the related images. In addition, by optimizing the image recognition model based on the feature map determined by the historical model and the multiple augmented images corresponding to the current model, the robustness of the image recognition model can be improved, and the recognition consistency of the image recognition model for fundus images of different forms based on the same fundus can be improved.

[0008] In addition, in the training method involved in the first aspect of the present disclosure, optionally, the prediction result also includes a prediction category, and also includes: obtaining the true label corresponding to each of the training fundus images; obtaining a second loss function based on the true label and the predicted category obtained by the current model; and optimizing the image recognition model using the first loss function and the second loss function. In this case, optimizing the parameters of the image recognition model based on the second loss function can make the predicted category and the true label close. In addition, by optimizing the image recognition model in combination with the first loss function and the second loss function, the ability of the image recognition model to extract features based on different forms of training fundus images of the same fundus can be improved.

[0009] In addition, in the training method involved in the first aspect of the present disclosure, optionally, the parameters of the current model are updated based on the first loss function and the second loss function using a back propagation algorithm to optimize the image recognition model; the historical model does not participate in back propagation. In this case, by using a back propagation algorithm to update the parameters of the current model (that is, the most recently acquired image recognition model), the accuracy of acquiring the parameters of the image recognition model can be improved, thereby making the predicted category and the true label closer.

[0010] In addition, in the training method involved in the first aspect of the present disclosure, optionally, the two groups of feature maps in the feature map of the two augmented images corresponding to the same training fundus image are compared for similarity to obtain the first loss function. In this case, based on the first loss function, the consistency of the feature map extracted by the image recognition model can be improved, and the recognition consistency of different forms of training fundus images based on the same fundus can be further improved.

[0011] In addition, in the training method involved in the first aspect of the present disclosure, optionally, in each iteration process, for one of the training fundus images: a random number is obtained; and based on the random number, whether to perform the data augmentation and the data augmentation operation is determined. In this case, by determining the update method with a random number, the balance of the distribution of the training fundus images can be improved, the domain shift of the training samples with a large proportion can be suppressed, and the generalization ability of the image recognition model can be improved.

[0012] In addition, in the training method involved in the first aspect of the present disclosure, optionally, the data augmentation operation includes at least one of image copying, image flipping, image rotation, image distortion, image scaling, image compression, image random cropping, image random edge filling, image contrast adjustment, image brightness adjustment, image chromaticity adjustment, image saturation adjustment, image color jittering, image noise addition, image blurring, random erasing of image areas, background color transformation, and image style conversion. In this case, by performing different data augmentation operations on the training fundus images, the generalization ability and robustness of the image recognition model can be improved, and the occurrence of domain shift can be reduced.

[0013] In addition, in the training method involved in the first aspect of the present disclosure, optionally, the training fundus image is preprocessed before the data augmentation process. In this case, irrelevant information in the training fundus image can be eliminated, the detectability of relevant information can be enhanced, and the data can be simplified to the maximum extent, thereby improving the reliability of recognition.

[0014] The second aspect of the present disclosure provides a fundus image recognition method, comprising: obtaining a fundus image to be recognized; and inputting the fundus image to be recognized into a trained model to obtain a recognition result corresponding to the fundus image to be recognized; wherein the trained model is a current model obtained based on the training method involved in the first aspect of the present disclosure. Thus, the recognition consistency of the trained model for fundus images of different forms based on the same fundus can be improved.

[0015] The third aspect of the present disclosure provides an electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, wherein when the processor executes the program, it implements the training method of the image recognition model involved in the first aspect of the present disclosure, or implements the fundus image recognition method involved in the second aspect of the present disclosure.

[0016] The fourth aspect of the present disclosure provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it implements the training method of the image recognition model involved in the first aspect of the present disclosure, or implements the fundus image recognition method involved in the second aspect of the present disclosure.

[0017] According to the present disclosure, a training method, a recognition method, a device and a storage medium of an image recognition model capable of improving the recognition consistency of fundus images are provided. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present disclosure will now be explained in further detail by way of example only with reference to the accompanying drawings, in which:

[0019] Figure 1 It is a schematic diagram showing the application scenario of the training method of the image recognition model involved in the example of the present disclosure.

[0020] Figure 2 is a flowchart showing a method for training an image recognition model involved in the example of the present disclosure.

[0021] Figure 3 is a flow chart showing a data augmentation method involved in an example of the present disclosure.

[0022] Figure 4A is a schematic diagram showing a training fundus image involved in the example of the present disclosure.

[0023] Figure 4B is a schematic diagram showing an amplified image involved in the examples of the present disclosure.

[0024] Figure 5 is a flow chart showing the training process involved in the examples of the present disclosure.

[0025] Figure 6 is a schematic diagram showing the acquisition of the first loss function involved in the example of the present disclosure.

[0026] Figure 7 is a schematic diagram showing the method of obtaining the second loss function involved in the example of the present disclosure.

[0027] Figure 8 is a flowchart showing the image recognition model optimization involved in the examples of the present disclosure.

[0028] Fig. 9 is a schematic diagram showing a fundus image recognition method involved in the example of the present disclosure. DETAILED DESCRIPTION

[0029] Hereinafter, the preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the following description, the same symbols are assigned to the same components, and repeated descriptions are omitted. In addition, the accompanying drawings are only schematic diagrams, and the ratio of the dimensions of the components or the shapes of the components may be different from the actual ones.

[0030] It should be noted that the terms "including" and "having" and any variations thereof in the present disclosure, such as a process, method, system, product or device including or having a series of steps or units, are not necessarily limited to those steps or units clearly listed, but may include or have other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. All methods described in the present disclosure may be performed in any suitable order, unless otherwise indicated herein or clearly contradicted by the context.

[0031] The training method of the image recognition model involved in the present disclosure example may sometimes also be referred to as a training method, an image learning method, an image recognition training method, an optimization method or an iterative method, etc.

[0032] The training method involved in the example of the present disclosure can improve the recognition consistency of the image recognition model for the image. The training method involved in the example of the present disclosure can be applied to the application scenario of training any image recognition model.

[0033] The disclosed example also provides a fundus image recognition method, which may also be referred to as a recognition method, which is based on a trained model obtained after training by the training method involved in the disclosed example. Thus, the recognition consistency of the trained model for fundus images of different forms based on the same fundus 10 can be improved.

[0034] In some examples, the training fundus images in different forms based on the same fundus may be training fundus images acquired under the same fundus with different presentation forms, for example, training fundus images acquired multiple times on the same fundus or training fundus images acquired on the same fundus using different image acquisition devices.

[0035] In some examples, when evaluating the recognition consistency of the image recognition model involved in the examples of the present disclosure, the robustness of the model can be used for evaluation. For example, in the training fundus images of different forms based on the same fundus 10, the lower the robustness of the image recognition model, the lower the recognition consistency of the image recognition model for the fundus images of different forms based on the same fundus.

[0036] The image recognition model involved in the examples of the present disclosure may be a model based on machine learning. In some examples, the model based on machine learning may be an artificial neural network. In some examples, the artificial neural network may be a ResNet (residual network) model, a DenseNet (densely connected network) model, an Inception (multi-scale convolutional network) model, or an EfficientNet (efficient network) model, etc.

[0037] In some examples, the image of the input image recognition model involved in the examples of the present disclosure may be a fundus image, thereby enabling the fundus image to be recognized based on the trained model.

[0038] In some examples, the image input to the image recognition model may be an image of another tissue object (e.g., a vascular ultrasound image). In other examples, the image input to the image recognition model may be a natural image. The natural image may be an image observed or photographed in a natural scene.

[0039] Hereinafter, an example of the present disclosure is described by taking the image of the input model as a fundus image collected by a fundus camera as an example. It should be noted that such description does not limit the scope of the present disclosure, and the relevant description is also applicable to other images in the absence of contradiction.

[0040] Figure 1 It is a schematic diagram showing the application scenario of the training method of the image recognition model 13 involved in the example of the present disclosure.

[0041] In some examples, the training method involved in the examples of the present disclosure can be applied to Figure 1 In the application scenario shown in Figure 1 In the application scenario shown, the following steps may be performed: first, a training fundus image 30 of the fundus 10 may be acquired by an image acquisition device 11, and the acquired multiple training fundus images 30 may be used as input 12 of an electronic device 14 including an image recognition model 13, so as to train the image recognition model 13 in the electronic device 14, and then obtain the trained image recognition model 13 as the output 15 of the electronic device 14. That is, the output 15 may be the trained image recognition model 13.

[0042] In some examples, the image acquisition device 11 may be at least one of a fundus camera, a high-definition digital camera, a video camera, and a scanner. In some examples, preferably, the image acquisition device 11 may be a fundus camera.

[0043] In some examples, the fundus camera may include a light source, a lens assembly through which a light beam reflected from the fundus passes, an imaging element for receiving the light beam passing through the lens assembly, and a recording module for recording an imaging position. The fundus camera may adjust the relative positions of the lens assembly and the imaging element based on the imaging position to obtain a relatively clear training fundus image 30.

[0044] In some examples, the electronic device 14 may be a personal terminal desktop computer, a personal terminal notebook computer, or a cloud server, etc.

[0045] In some examples, the training fundus images 30 may include fundus images of diabetic retinopathy, fundus images of microaneurysm lesions, fundus images of hemorrhagic lesions, fundus images of neovascular lesions, or fundus images of cotton wool spots.

[0046] The following combination Figure 2 Describe the training method involved in the examples of the present disclosure. Figure 2 is a flowchart showing a method for training the image recognition model 13 involved in the example of the present disclosure.

[0047] refer to Figure 2 In some examples, the training method may include acquiring a plurality of training fundus images 30 (step S100 ).

[0048] In some examples, in step S100, it is also possible to obtain the true label corresponding to each training fundus image 30. In addition, the true label can be used to determine the second loss function (described later).

[0049] In some examples, the true labels of the training fundus images 30 may include diabetic retina labels, microaneurysm labels, hemorrhage labels, exudation labels, cotton wool spot labels, neovascularization labels, fundus labels, or non-fundus labels, etc.

[0050] In some examples, the true label can be used to confirm whether the image recognition model 13 has misjudged. For example, in the image recognition model 13 used to identify whether it is a fundus, the true label can include a fundus label and a non-fundus label. The true label of the training fundus image 30 is the fundus label. However, after the image recognition model 13 recognizes the training fundus image 30, the recognition result is a non-fundus label, then it can be considered that the image recognition model 13 misjudged the training fundus image 30.

[0051] In some examples, the true label of the training fundus image 30 can be determined by medical workers after detailed inspection and analysis. For example, a lesion judgment result can be obtained based on whether there is a lesion in the training fundus image 30, and the lesion judgment result can be a negative result and a positive result. It can also be other results. For example, for diabetic retinopathy screening, the lesion judgment result can be a retinopathy grade in the retinopathy grading system used by the British National Retinopathy Screening Program, including R0, R1, R2 and R3.

[0052] refer to Figure 2 In some examples, the training method may include performing data augmentation on the same training fundus image 30 (step S101 ). In some examples, the training fundus image 30 acquired in step S100 may be data augmented to obtain a corresponding augmented image 31 .

[0053] The following, combined Figure 3 The data augmentation process of the image recognition model 13 involved in the example of the present disclosure (hereinafter referred to as the data augmentation method) is described. The data augmentation method involved in the example of the present disclosure is applied in step S101. Therefore, the relevant steps of the data augmentation method can be executed by step S101. Figure 3 is a flow chart showing a data augmentation method involved in an example of the present disclosure.

[0054] refer to Figure 3 In some examples, the data augmentation method may include obtaining a random number (step S201). In some examples, the random number may be any positive integer from 0 to 10. In other examples, the random number may be any numerical value.

[0055] In some examples, the random number can be generated by a computer program. In other examples, the random number can be set manually.

[0056] refer to Figure 3 In some examples, the data augmentation method may include determining whether to perform data augmentation based on a random number and performing a data augmentation operation (step S202).

[0057] In some examples, the training fundus image 30 may be preprocessed before data augmentation. In this case, irrelevant information in the training fundus image 30 can be eliminated, the detectability of relevant information can be enhanced, and the data can be simplified to the maximum extent, thereby improving the reliability of recognition. In some examples, the preprocessing operation may include cropping the training fundus image 30, image format conversion, or size adjustment.

[0058] In some examples, after the data augmentation process of the training fundus image 30, a pre-processing operation may be performed on the augmented image 31. For example, the augmented image 31 may be cropped, converted into an image format, or resized.

[0059] The following combination Figure 4A and Figure 4B The examples of the present disclosure described involve a training fundus image 30 and an augmented image 31 . Figure 4A is a schematic diagram showing a training fundus image 30 involved in the example of the present disclosure, Figure 4B is a schematic diagram showing an enlarged image 31 according to an example of the present disclosure.

[0060] In some examples, the data augmentation operation may include image duplication, image flipping, image rotation, image distortion, image scaling, image compression, random image cropping, random image edge filling, image contrast adjustment, image brightness adjustment, image chromaticity adjustment, image saturation adjustment, image color jitter, image noise addition, image blur, random erasing of image areas, background color transformation, and image style conversion, etc. In this case, by performing different data augmentation operations on the training fundus images 30, the generalization ability and robustness of the image recognition model 13 can be improved, and the occurrence of domain shift phenomenon can be reduced.

[0061] In some examples, data augmentation may be performed in the form of image flipping, and the data augmentation operation may be performed by flipping the training fundus image 30 upside down or left to right.

[0062] In some examples, data augmentation may be performed in the form of image rotation, where the data augmentation operation may be performed by rotating the training fundus image 30 by 30, 40, 50, 60, 70, 80, or 90 degrees, etc., along any direction with respect to the center point.

[0063] In some examples, data augmentation may be performed in the form of image distortion, where the data augmentation operation may be performed by rotating all pixels of the training fundus image 30 by 30, 40, 50, 60, 70, 80, or 90 degrees in any direction with the center point as the origin.

[0064] In some examples, data augmentation may be performed in the form of image scaling, and the data augmentation operation may be performed by reducing the entire training fundus image 30 to one-half, one-third, or one-quarter of the original size.

[0065] In some examples, data augmentation may be performed in the form of image compression, where the data augmentation operation may be performed by reducing the quality of the compressed image according to different quality factors between 0 and 100.

[0066] In some examples, data augmentation may be performed in the form of random image cropping, and the data augmentation operation may be performed by cropping the training fundus image 30 by one-half, one-third, one-quarter, etc. along any direction.

[0067] In some examples, data augmentation may be performed in the form of random edge filling of the image, and the operation of data augmentation may be performed by filling the edge portion of the training fundus image 30. In this case, the edge of the training fundus image 30 can be made clear and continuous.

[0068] In some examples, data augmentation may be performed in the form of image contrast adjustment, where the data augmentation operation may be performed to adjust the differences in color and brightness in the training fundus image 30 .

[0069] In some examples, data augmentation may be performed in the form of image brightness adjustment, and the data augmentation operation may be performed by increasing the brightness of the training fundus image 30. In this case, features such as edges or textures of the training fundus image 30 may be made more obvious.

[0070] In some examples, data augmentation may be performed in the form of image chromaticity adjustment, and the data augmentation operation may be performed by adjusting the chromaticity in the training fundus image 30 to 30%, 50%, or 100%, etc.

[0071] In some examples, data augmentation may be performed in the form of image saturation adjustment. The data augmentation operation may be performed by adjusting the saturation of the training fundus image 30 to a maximum value, a minimum value, an intermediate value, or the like.

[0072] In some examples, data augmentation may be performed in the form of image color dithering, and the data augmentation operation may be performed by shifting the hue constituting the training fundus image 30. Thus, the contrast and vividness of the training fundus image 30 may be enhanced.

[0073] In some examples, data augmentation may be performed in the form of adding noise to the image. The data augmentation operation may be performed by randomly adding black and white pixels to the training fundus image 30 , wherein the ratio of black pixels to white pixels is half each.

[0074] In some examples, data augmentation may be performed in the form of image blurring. The data augmentation operation may be performed by taking the average value of surrounding pixels as the pixel of the training fundus image 30 .

[0075] In some examples, data augmentation may be performed in the form of random erasing of an image region. The data augmentation operation may be performed by randomly selecting a rectangular region in the training fundus image 30 and replacing pixel values ​​of the rectangular region with random values.

[0076] In some examples, data augmentation may be performed in the form of image style conversion, and the data augmentation operation may be performed by converting the training fundus image 30 into a training fundus image 30 with an oil painting style.

[0077] In some examples, data amplification can be performed in the form of background color transformation. The data amplification operation can be expressed as, in the three-primary color mode, converting the first background color with a color value of 0 into a second background color with a color value of 255, wherein the first background color is the background color before data amplification (that is, the background color of the training fundus image 30), and the second background color is the background color after data amplification (that is, the background color of the amplified image 31).

[0078] In other examples, data amplification can be performed in the form of background color transformation. The data amplification operation can be expressed as, in the three-primary color mode, converting the first background color with a color value of 255 into a second background color with a color value of 0, wherein the first background color is the background color before data amplification (that is, the background color of the training fundus image 30), and the second background color is the background color after data amplification (that is, the background color of the amplified image 31).

[0079] In some examples, when performing background color transformation on the training fundus image 30, a preset span of color values ​​may be set between the first background color and the second background color, wherein, in the three-primary color mode, the color value may range from 0 to 255, and the preset span may range from 0 to 254. In addition, when performing background color transformation on the training fundus image 30, the span of color values ​​between the first background color and the second background color is greater than the preset span. For example, in the three-primary color mode, the preset span may be 254, and when the color value of the first background color is 0, the color value of the second background color is 255.

[0080] In some examples, the preset span may be determined by a manually specified value. In other examples, the preset span may be determined by a random number obtained by a computer program.

[0081] In some examples, the background color may be transformed on the training fundus image 30. For example, in a three-primary color mode, a first background color with a color value of 0 may be transformed into a second background color with a color value of 255.

[0082] In some examples, the background color transformation may be to transform the first background color of the training fundus image 30 into the second background color, and the span of the color value between the first background color and the second background color is greater than a preset span, wherein the preset span is within the color value range. In this case, by performing background color transformation on the training fundus image 30 and the color value of the transformed background color is greater than the preset span, the contrast of the training fundus image 30 can be enhanced, and the complexity of the training fundus image 30 can be reduced, thereby improving the robustness of the image recognition model 13.

[0083] In some examples, when data augmentation is performed on the training fundus image 30, one data augmentation operation may be used to perform data augmentation. In other examples, multiple data augmentation operations may be combined to perform data augmentation. For example, two data augmentation operations, image random cropping and image noise addition, may be used to perform data augmentation on the training fundus image 30.

[0084] In some examples, the generalization ability of the image recognition model 13 can be improved by performing data augmentation on the training fundus images 30 .

[0085] In some examples, the generalization ability can evaluate the recognition ability of the image recognition model 13. Specifically, if the generalization ability of the image recognition model 13 is higher, it means that the image recognition model 13 has a stronger recognition ability for data that has never been recognized, and the recognition result is more accurate. If the generalization ability of the image recognition model 13 is lower, it means that the image recognition model 13 has a weaker recognition ability for data that has never been recognized, and the recognition result is less accurate.

[0086] In some examples, for a training fundus image 30, in each iteration process: a random number is obtained, and whether to perform data augmentation and data augmentation operations are determined based on the random number. In this case, by determining the update method with a random number, the distribution balance of the training fundus image 30 can be improved, and the domain shift of the training samples with a large proportion can be suppressed, thereby improving the generalization ability of the image recognition model 13.

[0087] Return to reference Figure 2 In some examples, the training method may include optimizing the image recognition model 13 based on the augmented image 31 (step S102).

[0088] In some examples, in step S102, the process may further include inputting the augmented image 31 into the image recognition model 13 so that the image recognition model 13 outputs a prediction result, and adjusting the parameters of the image recognition model 13 based on the prediction result. In addition, a first loss function (described later) may be obtained based on the prediction result.

[0089] In some examples, the prediction result output by the image recognition model 13 may include at least one of a feature map, a prediction category, and a prediction probability distribution map. In some examples, the prediction result output by the image recognition model 13 may include a feature map, wherein the feature map may be represented by a feature vector.

[0090] In some examples, in step S102, a first loss function may be obtained based on the prediction result, and the image recognition model 13 may be optimized using the first loss function. In some examples, the first loss function may be a similarity loss function. For example, a cosine similarity function or a KL divergence function. In this case, the similarity between the training fundus image 30 and the augmented image 31 can be measured using the similarity loss function, so that the similarity between the training fundus image 30 and the augmented image 31 can be made closer.

[0091] The following combination Figure 5 The training process of the training method of the image recognition model 13 involved in the example of the present disclosure is described. The training process involved in the example of the present disclosure can be applied to step S102. Therefore, the relevant steps of the training process involved in the example of the present disclosure can be executed by step S102. Figure 5 is a flow chart showing the training process involved in the examples of the present disclosure.

[0092] refer to Figure 5 In some examples, the training process may include inputting the augmented image 31 into the image recognition model 13 and outputting a prediction result (step S301).

[0093] refer to Figure 5 In some examples, the training process may include adjusting parameters of the image recognition model 13 based on the prediction results (step S302).

[0094] In some examples, after the training process is completed, the parameters of the image recognition model 13 can be adjusted based on the first loss function (described later) and the second loss function (described later), thereby optimizing the recognition capability of the image recognition model 13.

[0095] In some examples, the first loss function can be obtained based on the first feature map and the second feature map, wherein the first feature map is a feature map obtained by the historical model and related to the same training fundus image 30, and the second feature map is a feature map obtained by the current model and related to the same training fundus image 30. In addition, the first feature map and the second feature map can be composed of multiple feature maps of training samples related to the same training fundus image 30, and the first loss function can be obtained by comparing the similarity between multiple feature maps in the first feature map and the second feature map.

[0096] In some examples, the first loss function can be obtained by comparing the similarities between multiple feature maps in the feature map. In this case, based on the first loss function, the consistency of the feature maps extracted by the image recognition model 13 can be improved, and the recognition consistency of the image recognition model 13 for different forms of training fundus images 30 based on the same fundus 10 can be further improved.

[0097] In some examples, the similarity between multiple feature maps in the feature map may be negatively correlated with the first loss function. Specifically, if the similarity between multiple feature maps in the feature map is higher, the first loss function obtained may be smaller; if the similarity between multiple feature maps in the feature map is lower, the first loss function obtained may be larger.

[0098] The following combination Figure 6 The process of obtaining the first loss function involved in the example of the present disclosure is described. Figure 6 is a schematic diagram showing the acquisition of the first loss function involved in the example of the present disclosure.

[0099] refer to Figure 6 In some examples, the training process of the historical model 20 may include training fundus images 30, augmented images 31, the historical model 20, and feature maps 32. In addition, the training process of the current model 21 may include training fundus images 30, augmented images 31, the current model 21, and feature maps 32.

[0100] In some examples, a first loss function can be obtained based on the prediction results obtained from the historical model 20 and the current model 21 and the image recognition model 13 can be optimized using the first loss function, wherein the historical model 20 can be the image recognition model 13 obtained by the previous optimization of the current model 21.

[0101] refer to Figure 6 In some examples, during the training of the historical model 20, the training fundus image 30 may be data augmented to obtain an augmented image 31, and the augmented image 31 may be input into the historical model 20 to output a feature map 32. In addition, during the training of the current model 21, the training fundus image 30 may be data augmented to obtain an augmented image 31, and the augmented image 31 may be input into the current model 21 to output a feature map 32. By comparing the feature map 32 and the similarity between multiple feature maps in the feature map 32, a first loss function may be obtained. The training fundus image 30 may be acquired based on the same fundus 10.

[0102] In some examples, the reference Figure 2 In step S102, a predicted category obtained based on the true label and the current model 21 may also be included to obtain a second loss function.

[0103] In some examples, the second loss function may be a cross entropy loss function. In addition, the cross entropy loss function may use a deep neural network with multiple fully connected layers as a classifier and use the output generated by the last fully connected layer to convert it into a probability distribution (for example, using a Softmax function).

[0104] In some examples, the gap between the output probability distribution of the image recognition model 13 and the probability distribution of the true label can be positively correlated with the second loss function. Specifically, if the gap between the output probability distribution of the image recognition model 13 and the probability distribution of the true label is smaller, the second loss function can be smaller; if the gap between the output probability distribution of the image recognition model 13 and the probability distribution of the true label is larger, the second loss function can be larger.

[0105] In some examples, step S102 may also include optimizing the image recognition model 13 using a first loss function and a second loss function.

[0106] In some examples, the prediction result may also include a predicted category, and the training method may also include: obtaining the true label corresponding to each training fundus image 30; obtaining a second loss function based on the true label of the training sample and the predicted category obtained by the current model 21; and optimizing the image recognition model 13 using the first loss function and the second loss function. In this case, the parameters of the image recognition model 13 can be optimized by the second loss function, so that the predicted category and the true label can be close. In addition, by optimizing the image recognition model 13 in combination with the first loss function and the second loss function, the ability of the image recognition model 13 to extract features based on different forms of training fundus images 30 of the same fundus 10 can be improved.

[0107] In some examples, the image recognition model 13 can be optimized by updating the parameters of the image recognition model 13 based on the first loss function and the second loss function and using the back propagation algorithm. In some examples, the image recognition model 13 can be optimized by updating the parameters of the current model 21 based on the first loss function and the second loss function and using the back propagation algorithm. In some examples, the historical model 20 may not participate in back propagation. In this case, by updating the parameters of the current model 21 (that is, the most recently acquired image recognition model 13) using the back propagation algorithm, the accuracy of acquiring the parameters of the image recognition model 13 can be improved, thereby making the predicted category and the true label closer.

[0108] In some examples, after obtaining the first loss function and the second loss function and optimizing the image recognition model 13 using the back propagation algorithm, the next training process can be performed.

[0109] The following combination Figure 7 The process of obtaining the second loss function involved in the example of the present disclosure is described. Figure 7 is a schematic diagram showing the process of obtaining the second loss function involved in the example of the present disclosure.

[0110] refer to Figure 7In some examples, the training process of the historical model 20 may include augmenting the image 31, the historical model 20, and the prediction result 33. In addition, the training process of the current model 21 may include augmenting the image 31, the current model 21, and the prediction result 33.

[0111] refer to Figure 7 In some examples, a first loss function may be obtained based on the prediction result 33 of the historical model 20 and the prediction result 33 of the current model 21. In addition, a second loss function may be obtained based on the prediction result 33 of the current model 21.

[0112] refer to Figure 7 In some examples, the current model 21 can be optimized based on the first loss function and the second loss function.

[0113] The following combination Figure 8 Describe the optimization method of the image recognition model 13 (hereinafter referred to as the model optimization method) of the training method of the image recognition model 13 involved in the example of the disclosure. The model optimization method involved in the example of the disclosure is applied in step S102. Therefore, the relevant steps of the model optimization method can be executed by step S102. Figure 8 is a flowchart showing the optimization of the image recognition model 13 involved in the example of the present disclosure.

[0114] refer to Figure 8 In some examples, the model optimization method may include calculating based on the first loss function and the second loss function and using a back propagation algorithm (step S401).

[0115] refer to Figure 8 In some examples, the model optimization method may include optimizing and adjusting the parameters of the image recognition model 13 based on the calculation results of the back propagation algorithm (step S402). Specifically, the back propagation algorithm may be used to calculate the gradients of all trainable parameters in the image recognition model 13 so as to optimize the image recognition model 13 by updating these parameters.

[0116] In some examples, during the training of the image recognition model 13, it can be determined whether the optimized image recognition model 13 has reached a preset training end condition. If the preset training end condition has been reached, the training of the image recognition model 13 is terminated; if the preset training end condition has not been reached, the iterative process is continued to optimize the image recognition model 13. The training end condition may include loss convergence and reaching a maximum iteration number threshold.

[0117] In some examples, during the training process of the image recognition model 13 , the training can be performed based on different forms of training fundus images 30 corresponding to a plurality of fundus images 10 .

[0118] In some examples, the training method of the image recognition model 13 may include: obtaining multiple training fundus images 30; performing data augmentation on the training fundus images 30 to obtain augmented images 31; and in each iteration, the augmented images 31 may be input into the image recognition model 13 to obtain corresponding prediction results 33 including feature mappings, obtaining a first loss function based on the prediction results 33 obtained by the historical model 20 and the current model 21, and optimizing the image recognition model 13 using the first loss function; wherein the historical model 20 may be an image recognition model 13 obtained by the previous optimization of the current model 21; the first loss function may be calculated based on feature maps 32 of at least two augmented images 31 corresponding to the same training fundus image 30 obtained in the historical model 20 and the current model 21, and the feature map 32 may be composed of multiple feature maps with the augmented image 31. In this case, by performing data augmentation on the acquired training fundus images 30, the data diversity of the fundus images in the training set and the generalization ability of the image recognition model 13 can be improved, thereby improving the recognition consistency of the image recognition model 13 for fundus images of different forms based on the same fundus 10. In addition, by calculating the first loss function based on the feature map 32 related to the same training fundus image 30 to optimize the image recognition model 13, the similarity of the feature map related to the same training fundus image 30 can be improved, and thus the consistency of the prediction result 33 of the related image can be improved. In addition, by optimizing the image recognition model 13 based on the feature map 32 determined based on the historical model 20 and the multiple augmented images 31 corresponding to the current model 21, the robustness of the image recognition model 13 can be improved, and the recognition consistency of the image recognition model 13 for fundus images of different forms based on the same fundus 10 can be improved.

[0119] The disclosed example also relates to a fundus image recognition method, which can recognize fundus images using an image recognition model 13 obtained after training using the above-mentioned training method.

[0120] Fig. 9 is a schematic diagram showing a fundus image recognition method involved in the example of the present disclosure.

[0121] In some examples, the fundus image recognition method may include acquiring a fundus image 50 to be recognized, and inputting the fundus image 50 to be recognized into a trained model 51 to obtain a recognition result 52 of the fundus image 50 to be recognized.

[0122] In some examples, the fundus image 50 to be identified may be a fundus image obtained by an operator collecting the fundus 10 of a patient in actual work.

[0123] In some examples, the trained model 51 may be the current model 21 obtained based on the training method of the image recognition model 13 involved in the examples of the present disclosure. In addition, the trained model 51 may be the current model 21 obtained based on a large number of training fundus images 30 and multiple training processes.

[0124] In some examples, the fundus image recognition method may include: obtaining a fundus image 50 to be recognized; and inputting the fundus image 50 to be recognized into a trained model 51 to obtain a recognition result 52 corresponding to the fundus image 50 to be recognized; wherein the trained model 51 may be a trained image recognition model 13 obtained based on the training method involved in the present disclosure. Thus, the recognition consistency of the trained model for fundus images of different forms based on the same fundus 10 can be improved.

[0125] The present disclosure also relates to an electronic device 14, which may also be referred to as a device, and may include a memory, a processor, and a computer program stored in the memory and running on the processor. In some examples, the training method of the above-mentioned image recognition model 13 may be implemented when the processor executes the computer program. In addition, the above-mentioned fundus image recognition method may be implemented when the processor executes the computer program.

[0126] The present disclosure also relates to a non-transitory computer-readable storage medium, on which a computer program may be stored (also referred to as a storage medium). In some examples, the computer program may implement the training method of the above-mentioned image recognition model 13 when executed by a processor. In addition, the computer program may implement the above-mentioned fundus image recognition method when executed by a processor.

[0127] Although the present disclosure is specifically described above in conjunction with the accompanying drawings and examples, it is to be understood that the above description does not limit the present disclosure in any form. Those skilled in the art may modify and change the present disclosure as needed without departing from the essential spirit and scope of the present disclosure, and these modifications and changes all fall within the scope of the present disclosure.

Claims

1. A training method for an image recognition model, characterized in that: include: Acquire multiple training fundus images; Perform data augmentation on the training fundus image to obtain an augmented image; and in each iteration, input the augmented image into the image recognition model to obtain a corresponding prediction result including a feature map, obtain a first loss function based on the prediction results obtained by the historical model and the current model, and optimize the image recognition model using the first loss function, wherein the historical model is an image recognition model obtained by any previous optimization of the current model; calculate the first loss function based on feature maps of at least two augmented images corresponding to the same training fundus image obtained by the historical model and the current model, wherein the feature map is composed of multiple feature maps with the augmented image.

2. The method for training an image recognition model according to claim 1, characterized in that: The prediction result also includes a prediction category, and also includes: obtaining the true label corresponding to each of the training fundus images; obtaining a second loss function based on the true label and the prediction category obtained by the current model; and optimizing the image recognition model using the first loss function and the second loss function.

3. The method for training an image recognition model according to claim 2, characterized in that: Based on the first loss function and the second loss function, the parameters of the current model are updated using a back propagation algorithm to optimize the image recognition model; the historical model does not participate in back propagation.

4. The method for training an image recognition model according to claim 1, characterized in that: The first loss function is obtained by performing a similarity comparison on two groups of feature maps in the feature maps of the two augmented images corresponding to the same training fundus image.

5. The method for training an image recognition model according to claim 1, characterized in that: In each iteration process, for one of the training fundus images: a random number is obtained; and based on the random number, whether to perform the data augmentation and the data augmentation operation are determined.

6. The method for training an image recognition model according to claim 1, characterized in that: The data augmentation operation includes at least one of image copying, image flipping, image rotation, image distortion, image scaling, image compression, random image cropping, random image edge filling, image contrast adjustment, image brightness adjustment, image chromaticity adjustment, image saturation adjustment, image color jittering, image noise addition, image blurring, random erasing of image areas, background color transformation, and image style conversion.

7. The method for training an image recognition model according to claim 1, characterized in that: The training fundus image is preprocessed before the data augmentation process.

8. A fundus image recognition method, characterized in that: include: Acquire a fundus image to be identified; and input the fundus image to be identified into a trained model to obtain a recognition result corresponding to the fundus image to be identified; wherein the trained model is a current model obtained based on the training method described in any one of claims 1 to 7.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, it implements the training method of the image recognition model as described in any one of claims 1 to 7, or implements the fundus image recognition method as described in claim 8.

10. A non-transitory computer readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the training method of the image recognition model as described in any one of claims 1 to 7, or implements the fundus image recognition method as described in claim 8.

Images