Model training method and device, electronic equipment and storage medium

Abstract

The application provides a model training method and device, electronic equipment and storage medium. The method comprises: obtaining fundus image samples, including a first number of ROP fundus images with staging labels and a second number of ROP fundus images without staging labels; inputting the first number of fundus image samples into a fundus image staging prediction model to obtain a staging prediction output and calculate a classification loss value; based on the prediction output of the fundus image staging prediction model for the first number and the second number of fundus image samples, a prediction consistency loss value is calculated; based on the features extracted from the fundus image samples by the fundus image staging prediction model, a semantic association consistency loss value is calculated; based on the classification loss value, the prediction consistency loss value and the semantic association consistency loss value, a target loss value is calculated; and in the case that the target loss value is within a preset range, a final staging prediction model is obtained. The application can improve the classification performance and recognition accuracy of the model.

Description

Technical Field

[0001] This application relates to the field of fundus image processing technology, and in particular to a model training method, apparatus, electronic device and storage medium. Background Technology

[0002] Retinopathy of prematurity (ROP) is a proliferative disorder that occurs during the development of the retina in premature infants. It remains a leading cause of childhood blindness worldwide. Ophthalmological examinations of premature infants require frequent and close monitoring, resulting in a significant workload of manual image interpretation. However, in clinical practice, the diagnosis of stages 1, 2, and 3 of ROP is subjective and involves diagnostic variability. Failure to detect and treat ROP in a timely manner can lead to late-stage ROP, resulting in visual impairment or even blindness. Furthermore, experienced pediatric ophthalmologists are in short supply, with most concentrated in large cities or major medical centers. Premature infants living in remote areas must travel long distances for referral, delaying treatment. Therefore, telemedicine and computer-assisted interpretation of ROP fundus images are of great significance.

[0003] Currently, many deep learning models have been proposed for computer-aided screening and diagnosis in the field of ROP image analysis. However, deep learning methods often require a large amount of annotated data for model training. In clinical settings, annotating large amounts of data is often time-consuming and labor-intensive, even for experienced physicians. For the annotation of ROP fundus images, clinicians typically determine the ROP stage based on the shape and size of the spine in the ROP fundus image. Stage 4 or 5 ROP fundus images are easily distinguishable, while normal fundus images, stage 1 ROP images, or stage 2 ROP images, being in the early stages of the disease, may not be very obvious and could be misinterpreted by clinicians, thus increasing their workload. Summary of the Invention

[0004] This application provides a model training method, apparatus, electronic device, and storage medium to address the problems in the prior art where deep learning methods require a large amount of annotation data, which is time-consuming and labor-intensive, and where early-stage fundus images may be misinterpreted by clinicians, leading to annotation errors and increasing the burden on clinicians.

[0005] To solve the above-mentioned technical problems, the embodiments of this application are implemented as follows:

[0006] In a first aspect, embodiments of this application provide a model training method, the method comprising:

[0007] Acquire fundus image samples; the fundus image samples include: a first number of ROP fundus images with stage labels and a second number of ROP fundus images without stage labels, wherein the first number is less than the second number;

[0008] The first number of fundus image samples are input into the fundus image staging prediction model to obtain the staging prediction output of the first number of fundus images, and the classification loss value is calculated.

[0009] The fundus image staging prediction model is used to process the first number and the second number of fundus image samples to obtain the prediction output, and the prediction consistency loss value of the fundus image staging prediction model is calculated.

[0010] Based on the fundus image staging prediction model, the semantic association consistency loss value of the fundus image staging prediction model is calculated according to the features extracted from the fundus image samples.

[0011] Based on the classification loss value, prediction consistency loss value, and semantic association consistency loss value, the target loss value of the fundus image staging prediction model is calculated.

[0012] If the target loss value is within a preset range, the converged fundus image staging prediction model is used as the final staging prediction model.

[0013] Optionally, acquiring fundus image samples includes:

[0014] Obtain raw fundus image samples;

[0015] The size of the original fundus image sample is adjusted to a set size to obtain the processed fundus image sample;

[0016] Data augmentation is performed on the processed fundus image sample to obtain the fundus image sample.

[0017] Optionally, the fundus image staging prediction model includes: a first staging prediction submodule and a second staging prediction submodel. The first staging prediction submodel includes: a first feature extraction layer and a first fully connected layer with N output units. The second staging prediction submodel includes: a second feature extraction layer and a second fully connected layer with N output units, where N is a positive integer.

[0018] The process of obtaining a prediction output by processing the first number and the second number of fundus image samples based on the fundus image staging prediction model includes:

[0019] A first random noise is applied to the ROP fundus image without stage label to obtain a first fundus image sample;

[0020] A second random noise is applied to the unstaging ROP fundus image to obtain a second fundus image sample;

[0021] The first fundus image sample is input into the first staging prediction sub-model, and the second fundus image sample is input into the second staging prediction sub-model;

[0022] The first feature extraction layer is invoked to extract high-level semantic features from the first fundus image sample, and the first fully connected layer is invoked to perform stage prediction on the first fundus image sample based on the high-level semantic features, and the first stage prediction label of the first fundus image sample is output.

[0023] The second feature extraction layer is invoked to extract high-level semantic features from the second fundus image sample, and the second fully connected layer is invoked to perform stage prediction on the second fundus image sample based on the high-level semantic features, and output the second stage prediction label of the second fundus image sample.

[0024] Optionally, the calculation of the prediction consistency loss value of the fundus image staging prediction model includes:

[0025] Obtain the first model parameters of the first staged prediction sub-model and the second model parameters of the second staged prediction sub-model;

[0026] Based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, and the second random noise, the prediction consistency loss value between the first staged prediction sub-model and the second staged prediction sub-model is calculated.

[0027] Optionally, the step of calculating the semantic association consistency loss value of the fundus image staging prediction model based on features extracted from the fundus image samples includes:

[0028] Obtain the similarity between the high-level semantic features of the first fundus image sample and the second fundus image sample;

[0029] Based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, the second random noise, and the similarity, the semantic association consistency loss value between the first stage prediction sub-model and the second stage prediction sub-model is calculated.

[0030] Optionally, calculating the target loss value of the fundus image staging prediction model based on the classification loss value, the prediction consistency loss value, and the semantic association consistency loss value includes:

[0031] The target loss value is calculated based on the classification loss value, the prediction consistency loss value, the semantic association consistency loss value, and the loss balance coefficient.

[0032] Secondly, embodiments of this application provide a model training apparatus, the apparatus comprising:

[0033] The image sample acquisition module is used to acquire fundus image samples; the fundus image samples include: a first number of ROP fundus images with stage labels and a second number of ROP fundus images without stage labels, wherein the first number is less than the second number;

[0034] The classification loss calculation module is used to input a first number of fundus image samples into the fundus image staging prediction model, obtain the staging prediction output of the first number of fundus images, and calculate the classification loss value.

[0035] The prediction consistency loss calculation module is used to process the first number and the second number of fundus image samples based on the fundus image staging prediction model to obtain the prediction output, and to calculate the prediction consistency loss value of the fundus image staging prediction model.

[0036] The semantic consistency loss calculation module is used to calculate the semantic association consistency loss value of the fundus image staging prediction model based on the features extracted from the fundus image samples.

[0037] The target loss value calculation module is used to calculate the target loss value of the fundus image staging prediction model based on the classification loss value, prediction consistency loss value and semantic association consistency loss value.

[0038] The staging prediction model acquisition module is used to use the converged fundus image staging prediction model as the final staging prediction model when the target loss value is within a preset range.

[0039] Optionally, the image sample acquisition module includes:

[0040] The raw image sample acquisition unit is used to acquire raw fundus image samples;

[0041] The image sample acquisition unit is used to adjust the size of the original fundus image sample to a set size to obtain the processed fundus image sample.

[0042] The fundus image sample acquisition unit is used to perform data augmentation operations on the processed fundus image sample to obtain the fundus image sample.

[0043] Optionally, the fundus image staging prediction model includes: a first staging prediction submodule and a second staging prediction submodel. The first staging prediction submodel includes: a first feature extraction layer and a first fully connected layer with N output units. The second staging prediction submodel includes: a second feature extraction layer and a second fully connected layer with N output units, where N is a positive integer.

[0044] The classification loss calculation module includes:

[0045] The first image sample acquisition unit is used to apply first random noise to the ROP fundus image without stage label to obtain a first fundus image sample.

[0046] The second image sample acquisition unit is used to apply a second random noise to the ROP fundus image without stage label to obtain a second fundus image sample.

[0047] The fundus image sample input unit is used to input the first fundus image sample into the first staging prediction sub-model and the second fundus image sample into the second staging prediction sub-model;

[0048] The first prediction label output unit is used to call the first feature extraction layer to extract high-level semantic features from the first fundus image sample, and call the first fully connected layer to perform stage prediction on the first fundus image sample based on the high-level semantic features, and output the first stage prediction label of the first fundus image sample.

[0049] The second prediction label output unit is used to call the second feature extraction layer to extract high-level semantic features from the second fundus image sample, and call the second fully connected layer to perform stage prediction on the second fundus image sample based on the high-level semantic features, and output the second stage prediction label of the second fundus image sample.

[0050] Optionally, the prediction consistency loss calculation module includes:

[0051] The model parameter acquisition unit is used to acquire the first model parameters of the first staged prediction sub-model and the second model parameters of the second staged prediction sub-model.

[0052] The prediction consistency loss calculation unit is used to calculate the prediction consistency loss value between the first staged prediction sub-model and the second staged prediction sub-model based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, and the second random noise.

[0053] Optionally, the semantic consistency loss calculation module includes:

[0054] A semantic similarity acquisition unit is used to acquire the similarity between the high-level semantic features of the first fundus image sample and the second fundus image sample;

[0055] The semantic consistency loss calculation unit is used to calculate the semantic association consistency loss value between the first stage prediction sub-model and the second stage prediction sub-model based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, the second random noise, and the similarity.

[0056] Optionally, the target loss value calculation module includes:

[0057] The target loss value calculation unit is used to calculate the target loss value based on the classification loss value, the prediction consistency loss value, the semantic association consistency loss value, and the loss balance coefficient.

[0058] Thirdly, embodiments of this application provide an electronic device, including:

[0059] A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the model training method described in any of the preceding claims.

[0060] Fourthly, embodiments of this application provide a readable storage medium that, when the instructions in the storage medium are executed by the processor of an electronic device, enables the electronic device to perform any of the model training methods described above.

[0061] In this embodiment, fundus image samples are acquired, including: a first number of ROP fundus images with staging labels and a second number of ROP fundus images without staging labels, wherein the first number is less than the second number. The first number of fundus image samples is input into a fundus image staging prediction model to obtain staging prediction outputs for the first number of fundus images, and a classification loss value is calculated. Based on the fundus image staging prediction model, the first and second number of fundus image samples are processed to obtain prediction outputs, and a prediction consistency loss value for the fundus image staging prediction model is calculated. Based on the features extracted from the fundus image samples, a semantic association consistency loss value for the fundus image staging prediction model is calculated. Based on the classification loss value, prediction consistency loss value, and semantic association consistency loss value, a target loss value for the fundus image staging prediction model is calculated. If the target loss value is within a preset range, the converged fundus image staging prediction model is used as the final staging prediction model. This application's embodiments can simultaneously utilize a small number of labeled ROP fundus images and a large number of unlabeled ROP fundus images for automatic ROP staging, helping to alleviate the annotation burden and cost for doctors. Simultaneously, by employing predictive consistency loss, useful discriminative information is fully extracted from unlabeled data, thereby significantly improving the classification performance of the deep learning model. Furthermore, by using disease semantic association consistency loss, the evolutionary relationship of disease features at different ROP stages is considered, improving the model's recognition accuracy.

[0062] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0063] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 A flowchart illustrating the steps of a model training method provided in this application embodiment;

[0065] Figure 2 A flowchart illustrating the steps of a method for acquiring fundus image samples provided in this application embodiment;

[0066] Figure 3 A flowchart illustrating the steps of a classification loss calculation method provided in this application embodiment;

[0067] Figure 4 A flowchart illustrating the steps of a method for calculating a predicted consistency loss value, as provided in an embodiment of this application;

[0068] Figure 5 A flowchart illustrating the steps of a method for calculating semantic association consistency loss value provided in this application embodiment;

[0069] Figure 6 A flowchart illustrating the steps of a target loss value calculation method provided in this application embodiment;

[0070] Figure 7 A schematic diagram of a ROP fundus image provided in an embodiment of this application;

[0071] Figure 8 A schematic diagram of the overall structure of a semi-supervised classification model provided in an embodiment of this application;

[0072] Figure 9 A schematic diagram of a ResNet-50 model structure provided in an embodiment of this application;

[0073] Figure 10 This is a schematic diagram of the structure of a model training device provided in an embodiment of this application;

[0074] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0075] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0076] Reference Figure 1 The flowchart illustrates the steps of a model training method provided in an embodiment of this application, as follows: Figure 1 As shown, the device control method may include steps 101, 102, 103, 104, 105 and 106.

[0077] Step 101: Obtain fundus image samples; the fundus image samples include: a first number of ROP fundus images with staging labels and a second number of ROP fundus images without staging labels, wherein the first number is less than the second number.

[0078] In this embodiment, the fundus image samples are the ROP fundus images used to train the fundus image staging prediction model. In this example, the fundus image samples may include: a first number of ROP fundus images with staging labels and a second number of ROP fundus images without staging labels, where the first number is less than the second number. That is, in this embodiment, a small number of labeled ROP fundus images and a large number of unlabeled ROP fundus images are used for model training.

[0079] In practical implementation, when training the fundus image staging prediction model, a predetermined number of fundus image samples can be obtained for model training. Specifically, raw fundus image samples can be obtained first, and then preprocessed and data augmented to obtain processed fundus images. This implementation process can be combined with... Figure 2 The following is a detailed description.

[0080] Reference Figure 2 The flowchart illustrates the steps of a method for acquiring fundus image samples according to an embodiment of this application. Figure 2 As shown, the method for acquiring fundus image samples may include steps 201, 202 and 203.

[0081] Step 201: Obtain raw fundus image samples.

[0082] In this embodiment, raw fundus images can be acquired during the training of the fundus image staging prediction model. Specifically, the raw fundus images can be ROP fundus images pre-stored in a medical database within a medical institution, or ROP fundus images downloaded from a medical website, etc. The method for acquiring the raw fundus images can be determined according to business needs, and this embodiment does not impose any restrictions on it.

[0083] After obtaining the original fundus image sample, proceed to step 202.

[0084] Step 202: Adjust the size of the original fundus image sample to a set size to obtain the processed fundus image.

[0085] After obtaining the original fundus image sample, the size of the original fundus image sample is adjusted to a set size to obtain the processed fundus image sample. For example, the size of the original fundus image can be adjusted to 256×256 pixels.

[0086] After adjusting the pixels of the original fundus image sample to the set size to obtain the processed fundus image sample, step 203 is executed.

[0087] Step 203: Perform data augmentation on the processed fundus image sample to obtain the fundus image sample.

[0088] After resizing the original fundus image sample to a set size to obtain the processed fundus image sample, data augmentation operations can be performed on the processed fundus image sample to obtain the final fundus image sample. Specifically, data augmentation operations such as random rotation and random horizontal / vertical flipping can be performed on the processed fundus image sample to improve the generalization performance of the model.

[0089] In this example, the fundus image samples used for model training can simultaneously include fundus image samples from different stages. In practical applications, the staging of ROP fundus images can include: normal images (such as...) Figure 7 (a)), ROP phase I images (e.g.) Figure 7 (b)), ROP phase II images (e.g.) Figure 7 (c) ROP phase III images (e.g.) Figure 7 (d))), ROP phase IV images (such as Figure 7 (e) and ROP phase 5 images (e) Figure 7 (f)). During each round of model training, the fundus images input into the model should simultaneously include the ROP fundus images from the above six periods to enable the model to learn fully.

[0090] After obtaining the fundus image sample, proceed to step 102.

[0091] Step 102: Input the first number of fundus image samples into the fundus image staging prediction model to obtain the staging prediction output of the first number of fundus images, and calculate the classification loss value.

[0092] After obtaining fundus image samples, a fundus image staging prediction model can be trained based on the fundus image samples to obtain the staging prediction labels corresponding to the fundus image samples output by the fundus image staging prediction model.

[0093] After obtaining fundus image samples, a first set of fundus image samples can be input into the fundus image staging prediction model to obtain the staging prediction output of the first set of fundus images, and the classification loss value can be calculated. Specifically, the first set of fundus image samples can be input into the fundus image staging prediction model, which processes the first set of fundus image samples to obtain the corresponding staging prediction labels. Combining the staging prediction labels and the labeled staging labels, the classification loss value can be calculated. In this example, the classification loss value can be the cross-entropy loss value.

[0094] Step 103: Based on the fundus image staging prediction model, process the first number and the second number of fundus image samples to obtain the prediction output, and calculate the prediction consistency loss value of the fundus image staging prediction model.

[0095] After obtaining fundus image samples, the first and second number of fundus image samples can be processed based on the fundus image staging prediction model to obtain the prediction output, and the prediction consistency loss value of the fundus image staging prediction model can be calculated.

[0096] In this example, the fundus image staging prediction model may include: a first staging prediction submodule and a second staging prediction submodel, wherein the first staging prediction submodule and the second staging prediction submodel have the same model structure but different model parameters. The first staging prediction submodule and the second staging prediction submodel can be trained simultaneously. Figure 8 As shown, in the training phase, the model consists of two parts: the Student model (i.e., the first stage prediction sub-model) and the Teacher model (i.e., the second stage prediction sub-model).

[0097] The process of processing fundus image samples by the first stage prediction submodule and the second stage prediction submodel can be combined. Figure 3 The following is a detailed description.

[0098] Reference Figure 3 The flowchart illustrates the steps of a phased prediction tag acquisition method provided in an embodiment of this application, as shown below. Figure 3 As shown, the method for obtaining the phased prediction label may include steps 301, 302, 303, 304, and 305.

[0099] Step 301: Apply first random noise to the ROP fundus image without stage label to obtain the first fundus image sample.

[0100] Step 302: Apply second random noise to the ROP fundus image without stage label to obtain a second fundus image sample.

[0101] In this embodiment, after obtaining the fundus image samples, a first random noise can be applied to the ROP fundus images without staging labels in the fundus image samples. Then, a second random noise can be applied to the ROP fundus images without staging labels in the fundus image samples to obtain the second fundus image samples.

[0102] After obtaining the first fundus image sample and the second fundus image sample, step 303 is executed.

[0103] Step 303: Input the first fundus image sample into the first staging prediction sub-model, and input the second fundus image sample into the second staging prediction sub-model.

[0104] After obtaining the first fundus image sample and the second fundus image sample, the first fundus image sample can be input into the first stage prediction sub-model, and the second fundus image sample can be input into the second stage prediction sub-model.

[0105] In this example, the model structures of the first stage prediction sub-model and the second stage prediction sub-model are the same. The first stage prediction sub-model may include: a first feature extraction layer and a first fully connected layer with N class output units. The second stage prediction sub-model may include: a second feature extraction layer and a second fully connected layer with N class output units, where N is a positive integer.

[0106] The feature extraction layers (i.e., the first and second feature extraction layers) can be ResNet-50, and the ResNet-50 model structure can be as follows: Figure 9 As shown, it alleviates the problem of decreased learning efficiency and ineffective improvement of accuracy caused by increasing the number of layers in deep learning models through residual learning. At the same time, it has better feature extraction capabilities and better model classification performance due to the increase in the number of layers. This network performs well in many fields such as image classification, segmentation, and detection.

[0107] Of course, the feature extraction layer is not limited to the ResNet-50 model mentioned above, and other model structures can also be used. This embodiment does not impose any restrictions on this.

[0108] Fully connected layers (i.e., the first fully connected layer and the second fully connected layer) can connect to feature extraction layers; that is, the first fully connected layer connects to the first feature extraction layer, and the second fully connected layer connects to the second feature extraction layer. A fully connected layer can include several output units, the number of which is the same as the number of classification categories, i.e., each fully connected layer has 6 output units.

[0109] After inputting the first fundus image sample into the first stage prediction sub-model, step 304 is executed.

[0110] After inputting the second fundus image sample into the second stage prediction sub-model, step 305 is executed.

[0111] Step 304: Call the first feature extraction layer to extract high-level semantic features from the first fundus image sample, and call the first fully connected layer to perform stage prediction on the first fundus image sample based on the high-level semantic features, and output the first stage prediction label of the first fundus image sample.

[0112] The first stage prediction label refers to the stage label of the first fundus image sample predicted by the first stage prediction sub-model.

[0113] After inputting the first fundus image sample into the first stage prediction sub-model, the feature extraction layer can be called to extract the high-level semantic features in the first fundus image sample, and the first fully connected layer can be called to perform stage prediction on the first fundus image sample based on the high-level semantic features, and obtain the first stage prediction label of the first fundus image sample.

[0114] Step 305: Call the second feature extraction layer to extract high-level semantic features from the second fundus image sample, and call the second fully connected layer to perform stage prediction on the second fundus image sample based on the high-level semantic features, and output the second stage prediction label of the second fundus image sample.

[0115] The second stage prediction label refers to the stage label of the second fundus image sample predicted by the second stage prediction sub-model.

[0116] After inputting the second fundus image sample into the second stage prediction sub-model, the feature extraction layer can be called to extract the high-level semantic features in the second fundus image sample, and the second fully connected layer can be called to perform stage prediction on the second fundus image sample based on the high-level semantic features, and output the second stage prediction label of the second fundus image sample.

[0117] After training the fundus image staging prediction model based on fundus image samples and obtaining the staging prediction labels corresponding to the fundus image samples output by the fundus image staging prediction model, the prediction consistency loss value can be calculated based on the staging prediction labels output by the model.

[0118] The calculation process for predicting the consistency loss value can be combined with... Figure 4 The following is a detailed description.

[0119] Reference Figure 4 The flowchart illustrates the steps of a method for calculating the predicted consistency loss value provided in an embodiment of this application. Figure 4 As shown, the method for calculating the prediction consistency loss value may include steps 401 and 402.

[0120] Step 401: Obtain the first model parameters of the first staged prediction sub-model and the second model parameters of the second staged prediction sub-model.

[0121] In this embodiment, the first model parameter refers to the model parameter of the first staged prediction sub-model. The second model parameter refers to the model parameter of the second staged prediction sub-model.

[0122] When calculating the prediction consistency loss value, the first model parameters of the first stage prediction sub-model and the second model parameters of the second stage prediction sub-model can be obtained.

[0123] After obtaining the first model parameters of the first stage prediction sub-model and the second model parameters of the second stage prediction sub-model, step 402 is executed.

[0124] Step 402: Based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, and the second random noise, calculate the prediction consistency loss value between the first staged prediction sub-model and the second staged prediction sub-model.

[0125] The prediction consistency loss value refers to the degree of consistency between the period prediction labels output by the first period prediction sub-model and the second period prediction sub-model.

[0126] After obtaining the first model parameters of the first stage prediction sub-model and the second model parameters of the second stage prediction sub-model, the prediction consistency loss value between the first stage prediction sub-model and the second stage prediction sub-model can be calculated based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, and the second random noise. The formula for calculating the prediction consistency loss value is as shown in the following formula (1):

[0127]

[0128] In the above formula (1), L c To predict the consistency loss value, τ s and τ t It is random data noise applied to unlabeled ROP fundus images. s and f t These represent the first and second stage prediction sub-models, respectively. θ s and θ t N represents the model parameters of the first and second stage prediction sub-models, respectively. l N is the first quantity. u The second quantity. x i Let i be the i-th fundus image sample.

[0129] Step 104: Based on the features extracted from the fundus image samples, calculate the semantic association consistency loss value of the fundus image staging prediction model.

[0130] The semantic association consistency loss value can be used to indicate the degree of semantic consistency between the period predictions output by the first and second period prediction sub-models.

[0131] After obtaining the high-level semantic features of the fundus image staging prediction model, the semantic association consistency loss value of the fundus image staging prediction model can be calculated based on the high-level semantic features. Specifically, the semantic association consistency loss value can be calculated by combining the high-level semantic features output by the first staging prediction sub-model and the second staging prediction sub-model, respectively. The calculation method for the semantic association consistency loss value can be combined with... Figure 5 The following is a detailed description.

[0132] Reference Figure 5 The flowchart illustrates the steps of a semantic association consistency loss value calculation method provided in an embodiment of this application, as follows: Figure 5 As shown, the method for calculating the semantic association consistency loss value may include steps 501 and 502.

[0133] Step 501: Obtain the similarity between the high-level semantic features of the fundus image samples output by the first model and the second model.

[0134] In this embodiment, after obtaining the output of the feature extraction layer, the similarity between the high-level semantic features of the fundus image samples output by the first model and the second model can be obtained. Specifically, after obtaining the output of the feature extraction network, the gram matrix between the data in each batch is calculated: G = Q·Q T G i,j The similarity between samples i and j is represented and then normalized:

[0135]

[0136] After obtaining the similarity between the high-level semantic features of the fundus image samples output by the first model and the second model, step 502 is executed.

[0137] Step 502: Based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, the second random noise, and the similarity, calculate the semantic association consistency loss value between the first stage prediction sub-model and the second stage prediction sub-model.

[0138] After obtaining the similarity between the high-level semantic features of the fundus image samples output by the first and second models, the semantic association consistency loss value between the first and second staging prediction sub-models can be calculated using the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, the second random noise, and the similarity. The formula for calculating the semantic association consistency loss value is shown in the following formula (3):

[0139]

[0140] In the above formula (3), L s P represents the semantic association consistency loss value, B is the number of fundus image samples, and P is the number of samples. s P t This represents the similarity of high-level semantic features.

[0141] This application's embodiments utilize predictive consistency loss to fully extract useful discriminative information from unlabeled data, thereby significantly improving the classification performance of deep learning models. Through semantic association consistency loss, the evolutionary relationships of disease features at different stages of ROP are additionally considered, and this relationship is used to improve the recognition accuracy of the classification model.

[0142] After calculating the prediction consistency loss value and the semantic association consistency loss value, proceed to step 105.

[0143] Step 105: Based on the classification loss value, prediction consistency loss value, and semantic association consistency loss value, calculate the target loss value of the fundus image staging prediction model.

[0144] After calculating the prediction consistency loss and semantic association consistency loss, the target loss value of the fundus image staging prediction model can be calculated based on these three values. In this embodiment, the corresponding cross-entropy loss can be calculated using fundus image samples with staging labels. The target loss value can then be calculated based on the cross-entropy loss, prediction consistency loss, and semantic association consistency loss value. The calculation process for the target loss value can be combined with... Figure 6 The following is a detailed description.

[0145] Reference Figure 6 The flowchart illustrates the steps of a target loss value calculation method provided in an embodiment of this application, as follows: Figure 6 As shown, the method for calculating the target loss value may include: step 601.

[0146] Step 601: Calculate the target loss value based on the classification loss value, the prediction consistency loss value, the semantic association consistency loss value, and the loss balance coefficient.

[0147] In this embodiment, the target loss value can be calculated based on the cross-entropy loss value, the prediction consistency loss value, the semantic association consistency loss value, and the loss balance coefficient. The formula for calculating the target loss value is as shown in the following formula (4):

[0148] L total =L ce +λ(L c +L s (4)

[0149] In the above formula (4), L total L is the target loss value. ce λ is the cross-entropy loss value, and λ is the balance coefficient.

[0150] After calculating the target loss value, proceed to step 106.

[0151] Step 106: If the target loss value is within a preset range, the converged fundus image staging prediction model is used as the final staging prediction model.

[0152] After calculating the target loss value, it can be determined whether the target loss value is within the preset range.

[0153] If the target loss value is not within the preset range, it means that the fundus image staging prediction model has not converged, and the fundus image staging prediction model will continue to be trained for the next round.

[0154] If the target loss value is within the preset range, it means that the fundus image staging prediction model has converged. At this time, the converged fundus image staging prediction model can be used as the final staging prediction model, which can then be applied to the subsequent ROP fundus image staging prediction process.

[0155] Understandably, during model training, two parallel model structures are used to complete the training (i.e., the first stage prediction sub-model and the second stage prediction sub-model), while only one model structure, the first stage prediction sub-model, is used during model inference. For example... Figure 8 As shown, in the Test phase, a Student model can be used to perform model inference to stage and predict the ROP fundus images.

[0156] The model training method provided in this application involves acquiring fundus image samples, including: a first number of ROP fundus images with staging labels and a second number of ROP fundus images without staging labels, wherein the first number is less than the second number. The first number of fundus image samples is input into a fundus image staging prediction model to obtain staging prediction outputs for the first number of fundus images, and a classification loss value is calculated. Based on the fundus image staging prediction model, the first and second number of fundus image samples are processed to obtain prediction outputs, and a prediction consistency loss value for the fundus image staging prediction model is calculated. Based on the features extracted from the fundus image samples, a semantic association consistency loss value for the fundus image staging prediction model is calculated. Based on the classification loss value, prediction consistency loss value, and semantic association consistency loss value, a target loss value for the fundus image staging prediction model is calculated. If the target loss value is within a preset range, the converged fundus image staging prediction model is used as the final staging prediction model. This application's embodiments can simultaneously utilize a small number of labeled ROP fundus images and a large number of unlabeled ROP fundus images for automatic ROP staging, helping to alleviate the annotation burden and cost for doctors. Simultaneously, by employing predictive consistency loss, useful discriminative information is fully extracted from unlabeled data, thereby significantly improving the classification performance of the deep learning model. Furthermore, by using disease semantic association consistency loss, the evolutionary relationship of disease features at different ROP stages is considered, improving the model's recognition accuracy.

[0157] Reference Figure 10 The diagram shows a schematic representation of a model training device provided in an embodiment of this application. Figure 10 As shown, the model training device 1000 may include the following modules:

[0158] The image sample acquisition module 1001 is used to acquire fundus image samples; the fundus image samples include: a first number of ROP fundus images with stage labels and a second number of ROP fundus images without stage labels, wherein the first number is less than the second number;

[0159] The classification loss calculation module 1002 is used to input a first number of fundus image samples into the fundus image staging prediction model, obtain the staging prediction output of the first number of fundus images, and calculate the classification loss value.

[0160] The prediction consistency loss calculation module 1003 is used to process the first number and the second number of fundus image samples based on the fundus image staging prediction model to obtain the prediction output, and to calculate the prediction consistency loss value between the first staging prediction model and the second staging prediction model.

[0161] The semantic consistency loss calculation module 1004 is used to calculate the semantic association consistency loss value between the first stage prediction model and the second stage prediction model based on the features extracted from the fundus image samples according to the fundus image stage prediction model.

[0162] The target loss value calculation module 1005 is used to calculate the target loss value of the fundus image staging prediction model based on the classification loss value, the prediction consistency loss value and the semantic association consistency loss value.

[0163] The staging prediction model acquisition module 1006 is used to take the converged fundus image staging prediction model as the final staging prediction model when the target loss value is within a preset range.

[0164] Optionally, the image sample acquisition module includes:

[0165] The raw image sample acquisition unit is used to acquire raw fundus image samples;

[0166] The image sample acquisition unit is used to adjust the size of the original fundus image sample to a set size to obtain the processed fundus image sample.

[0167] The fundus image sample acquisition unit is used to perform data augmentation operations on the processed fundus image sample to obtain the fundus image sample.

[0168] Optionally, the fundus image staging prediction model includes: a first staging prediction submodule and a second staging prediction submodel. The first staging prediction submodel includes: a first feature extraction layer and a first fully connected layer with N output units. The second staging prediction submodel includes: a second feature extraction layer and a second fully connected layer with N output units, where N is a positive integer.

[0169] The classification loss calculation module includes:

[0170] The first image sample acquisition unit is used to apply first random noise to the ROP fundus image without stage label to obtain a first fundus image sample.

[0171] The second image sample acquisition unit is used to apply a second random noise to the ROP fundus image without stage label to obtain a second fundus image sample.

[0172] The fundus image sample input unit is used to input the first fundus image sample into the first staging prediction sub-model and the second fundus image sample into the second staging prediction sub-model;

[0173] The first prediction label output unit is used to call the first feature extraction layer to extract high-level semantic features from the first fundus image sample, and call the first fully connected layer to perform stage prediction on the first fundus image sample based on the high-level semantic features, and output the first stage prediction label of the first fundus image sample.

[0174] The second prediction label output unit is used to call the second feature extraction layer to extract high-level semantic features from the second fundus image sample, and call the second fully connected layer to perform stage prediction on the second fundus image sample based on the high-level semantic features, and output the second stage prediction label of the second fundus image sample.

[0175] Optionally, the prediction consistency loss calculation module includes:

[0176] The model parameter acquisition unit is used to acquire the first model parameters of the first periodic prediction sub-model and the second model parameters of the second periodic prediction sub-model.

[0177] The prediction consistency loss calculation unit is used to calculate the prediction consistency loss value between the first staged prediction sub-model and the second staged prediction sub-model based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, and the second random noise.

[0178] Optionally, the semantic consistency loss calculation module includes:

[0179] A semantic similarity acquisition unit is used to acquire the similarity between the high-level semantic features of the first fundus image sample and the second fundus image sample;

[0180] The semantic consistency loss calculation unit is used to calculate the semantic association consistency loss value between the first stage prediction sub-model and the second stage prediction sub-model based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, the second random noise, and the similarity.

[0181] Optionally, the target loss value calculation module includes:

[0182] The target loss value calculation unit is used to calculate the target loss value based on the classification loss value, the prediction consistency loss value, the semantic association consistency loss value, and the loss balance coefficient.

[0183] The model training apparatus provided in this application acquires fundus image samples, which include: a first number of ROP fundus images with staging labels and a second number of ROP fundus images without staging labels, wherein the first number is less than the second number. The first number of fundus image samples is input into a first fundus image staging prediction model to obtain staging prediction outputs for the first number of fundus images, and a classification loss value is calculated. Based on the prediction outputs of the first and second staging prediction models for the fundus image samples, a prediction consistency loss value between the first and second staging prediction models is calculated. Based on the features extracted from the fundus image samples by the first and second staging prediction models, a semantic association consistency loss value between the first and second staging prediction models is calculated. Based on the classification loss value, prediction consistency loss value, and semantic association consistency loss value, a target loss value for the fundus image staging prediction model is calculated. If the target loss value is within a preset range, the converged fundus image staging prediction model is used as the final staging prediction model. Simultaneously, by using the prediction consistency loss, useful discriminative information is fully extracted from the unlabeled data, thereby significantly improving the classification performance of the deep learning model. By using the disease semantic association consistency loss, the evolutionary relationship of disease features at different stages of ROP is taken into account, which improves the model's recognition accuracy.

[0184] Additionally, this application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the above-described model training method when executed by the processor.

[0185] Figure 11 A schematic diagram of the structure of an electronic device 1100 according to an embodiment of the present invention is shown. For example... Figure 11 As shown, the electronic device 1100 includes a central processing unit (CPU) 1101, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 1102 or loaded from storage unit 1108 into random access memory (RAM) 1103. The RAM 1103 may also store various programs and data required for the operation of the electronic device 1100. The CPU 1101, ROM 1102, and RAM 1103 are interconnected via bus 1104. An input / output (I / O) interface 1105 is also connected to bus 1104.

[0186] Multiple components in electronic device 1100 are connected to I / O interface 1105, including: input unit 1106, such as keyboard, mouse, microphone, etc.; output unit 1107, such as various types of monitors, speakers, etc.; storage unit 1108, such as disk, optical disk, etc.; and communication unit 1109, such as network card, modem, wireless transceiver, etc. Communication unit 1109 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0187] The various processes and handling described above can be executed by processing unit 1101. For example, the methods of any of the above embodiments can be implemented as computer software programs tangibly contained in a computer-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by CPU 1101, one or more actions of the methods described above can be performed.

[0188] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described model training method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0189] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0191] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0192] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0193] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0194] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0195] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0197] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0198] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A model training method, characterized in that, The method includes: Acquire fundus image samples; the fundus image samples include: a first number of ROP fundus images with stage labels and a second number of ROP fundus images without stage labels, wherein the first number is less than the second number; A first number of fundus image samples are input into the fundus image staging prediction model to obtain the staging prediction output of the first number of fundus images, and the classification loss value is calculated; the fundus image staging prediction model includes: a first staging prediction sub-model and a second staging prediction sub-model. The fundus image staging prediction model is used to process the first number and the second number of fundus image samples to obtain the prediction output, and the prediction consistency loss value of the fundus image staging prediction model is calculated. Based on the fundus image staging prediction model, the semantic association consistency loss value of the fundus image staging prediction model is calculated according to the features extracted from the fundus image samples. Based on the classification loss value, prediction consistency loss value, and semantic association consistency loss value, the target loss value of the fundus image staging prediction model is calculated. If the target loss value is within a preset range, the converged fundus image staging prediction model is used as the final staging prediction model. The step-by-step prediction model based on fundus image staging calculates the semantic association consistency loss value of the fundus image staging prediction model based on features extracted from the fundus image samples, including: The similarity between high-level semantic features of a first fundus image sample and a second fundus image sample is obtained; the first fundus image sample is obtained by applying a first random noise to the ROP fundus image without stage label, and the second fundus image sample is obtained by applying a second random noise to the ROP fundus image without stage label. Based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, the second random noise, and the similarity, the semantic association consistency loss value between the first stage prediction sub-model and the second stage prediction sub-model is calculated; the first model parameters are the model parameters of the first stage prediction sub-model, and the second model parameters are the model parameters of the second stage prediction sub-model.

2. The method according to claim 1, characterized in that, The acquisition of fundus image samples includes: Obtain raw fundus image samples; The size of the original fundus image sample is adjusted to a set size to obtain the processed fundus image sample; Data augmentation is performed on the processed fundus image sample to obtain the fundus image sample.

3. The method according to claim 1, characterized in that, The first staged prediction sub-model includes a first feature extraction layer and a first fully connected layer with N output units; the second staged prediction sub-model includes a second feature extraction layer and a second fully connected layer with N output units, where N is a positive integer. The process of obtaining a prediction output by processing the first number and the second number of fundus image samples based on the fundus image staging prediction model includes: A first random noise is applied to the ROP fundus image without stage label to obtain a first fundus image sample; A second random noise is applied to the unstaging ROP fundus image to obtain a second fundus image sample; The first fundus image sample is input into the first staging prediction sub-model, and the second fundus image sample is input into the second staging prediction sub-model; The first feature extraction layer is invoked to extract high-level semantic features from the first fundus image sample, and the first fully connected layer is invoked to perform stage prediction on the first fundus image sample based on the high-level semantic features, and the first stage prediction label of the first fundus image sample is output. The second feature extraction layer is invoked to extract high-level semantic features from the second fundus image sample, and the second fully connected layer is invoked to perform stage prediction on the second fundus image sample based on the high-level semantic features, and output the second stage prediction label of the second fundus image sample.

4. The method according to claim 3, characterized in that, The calculation of the prediction consistency loss value of the fundus image staging prediction model includes: Obtain the first model parameters of the first staged prediction sub-model and the second model parameters of the second staged prediction sub-model; Based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, and the second random noise, the prediction consistency loss value between the first staged prediction sub-model and the second staged prediction sub-model is calculated.

5. The method according to claim 1, characterized in that, The target loss value of the fundus image staging prediction model is calculated based on the classification loss value, the prediction consistency loss value, and the semantic association consistency loss value, including: The target loss value is calculated based on the classification loss value, the prediction consistency loss value, the semantic association consistency loss value, and the loss balance coefficient.

6. A model training device, characterized in that, The device includes: The image sample acquisition module is used to acquire fundus image samples; the fundus image samples include: a first number of ROP fundus images with stage labels and a second number of ROP fundus images without stage labels, wherein the first number is less than the second number; The classification loss calculation module is used to input a first number of fundus image samples into the fundus image staging prediction model, obtain the staging prediction output of the first number of fundus images, and calculate the classification loss value; the fundus image staging prediction model includes: a first staging prediction sub-model and a second staging prediction sub-model. The prediction consistency loss calculation module is used to process the first number and the second number of fundus image samples based on the fundus image staging prediction model to obtain the prediction output, and to calculate the prediction consistency loss value of the fundus image staging prediction model. The semantic consistency loss calculation module is used to calculate the semantic association consistency loss value of the fundus image staging prediction model based on the features extracted from the fundus image samples. The target loss value calculation module is used to calculate the target loss value of the fundus image staging prediction model based on the classification loss value, prediction consistency loss value and semantic association consistency loss value. The staging prediction model acquisition module is used to use the converged fundus image staging prediction model as the final staging prediction model when the target loss value is within a preset range. The semantic consistency loss calculation module includes: A semantic similarity acquisition unit is used to acquire the similarity between high-level semantic features of a first fundus image sample and a second fundus image sample; the first fundus image sample is obtained by applying a first random noise to the ROP fundus image without a stage label, and the second fundus image sample is obtained by applying a second random noise to the ROP fundus image without a stage label; The semantic consistency loss calculation unit is used to calculate the semantic association consistency loss value between the first staged prediction sub-model and the second staged prediction sub-model based on the first model parameters, the second model parameters, the first quantity, the second quantity, the first random noise, the second random noise, and the similarity; the first model parameters are the model parameters of the first staged prediction sub-model, and the second model parameters are the model parameters of the second staged prediction sub-model.

7. The apparatus according to claim 6, characterized in that, The image sample acquisition module includes: The raw image sample acquisition unit is used to acquire raw fundus image samples; The image sample acquisition unit is used to adjust the size of the original fundus image sample to a set size to obtain the processed fundus image sample. The fundus image sample acquisition unit is used to perform data augmentation operations on the processed fundus image sample to obtain the fundus image sample.

8. An electronic device, characterized in that, include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the model training method as described in any one of claims 1 to 5.

9. A readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the model training method according to any one of claims 1 to 5.

Images