Neonatal fundus image classification method, imaging method and storage medium based on multi-instance classification

Abstract

The present invention discloses a method for classifying neonatal fundus images based on multi-instance classification, an imaging method and a storage medium, including obtaining a multi-instance dataset of neonatal fundus images and labeling and preprocessing to obtain a training dataset; constructing a primary model for neonatal fundus image classification and training to obtain a neonatal fundus image classification model; using the neonatal fundus image classification model to complete the actual classification of neonatal fundus images based on multi-instance classification. The present invention extracts features through multi-instance images, and classifies fundus images based on a classification model constructed based on a feature extraction mechanism and a multi-head self-attention mechanism, and implements model training based on similarity loss and classification loss of prior knowledge; therefore, the present invention can not only classify neonatal fundus images, but also has higher reliability and better accuracy.

Description

Technical Field

[0001] The invention belongs to the field of digital image processing, and in particular relates to a neonatal fundus image classification method, an imaging method and a storage medium based on multi-instance classification. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, people pay more and more attention to health. Therefore, the accuracy of medical image classification has become the focus of people's attention.

[0003] The classification results of fundus images of newborns are of great significance in both clinical and basic medical research. At present, the classification scheme for fundus images of newborns is still carried out by clinical medical imaging personnel based on their own experience and technical level. However, this manual classification scheme for fundus images of newborns is not only time-consuming and laborious, but also has poor reliability and accuracy.

[0004] At the same time, although some researchers have proposed a neural network-based fundus image classification scheme, this type of scheme has not been professionally designed for newborn fundus images. Newborn fundus images have characteristics such as light fundus color, high transparency, sparse and uneven blood vessels, small abnormal areas, and unclear features. These characteristics make the classification results of existing fundus image classification schemes still have the defects of poor reliability and accuracy. Summary of the invention

[0005] One of the purposes of the present invention is to provide a newborn fundus image classification method based on multi-instance classification with high reliability and good accuracy.

[0006] A second object of the present invention is to provide an imaging method including the neonatal fundus image classification method based on multi-instance classification.

[0007] A third object of the present invention is to provide a storage medium including the neonatal fundus image classification method based on multi-instance classification.

[0008] The method for classifying fundus images of newborns based on multi-instance classification provided by the present invention comprises the following steps:

[0009] S1. Obtain an existing multi-instance dataset of neonatal fundus images and perform corresponding annotations;

[0010] S2. The image data set obtained in step S1 is preprocessed to obtain a training data set; the preprocessing includes resizing and image normalization;

[0011] S3. Based on the feature extraction model and multi-head self-attention mechanism, a primary model for neonatal fundus image classification was constructed, which included feature extraction module, feature aggregation module, feature fusion module and classification module;

[0012] A feature extraction module is constructed based on the feature extraction model to extract offline features from the input image data; a feature aggregation module is constructed based on the multi-head self-attention mechanism to aggregate the extracted image features; a feature fusion module is constructed based on the weighted summation scheme to fuse the aggregated features; and a classification module is used to obtain the final classification result of the neonatal fundus image based on the fused features.

[0013] S4. Using the training data set obtained in step S2, based on the similarity loss and classification loss of prior knowledge, the primary model for neonatal fundus image classification constructed in step S3 is trained to obtain a neonatal fundus image classification model;

[0014] S5. Using the neonatal fundus image classification model obtained in step S4, the actual classification of the neonatal fundus image based on multi-instance classification is completed.

[0015] The step S1 specifically includes the following steps:

[0016] The existing multiple instance data of fundus images of newborns are obtained; each newborn corresponds to a number of fundus images; when annotating, only the category of the newborn is annotated, and the fundus image corresponding to the newborn is not annotated.

[0017] The step S2 specifically includes the following steps:

[0018] The image data obtained in step S1 is uniformly adjusted to a set value for the image resolution;

[0019] Then, the image data is normalized to obtain the training data set.

[0020] The step S3 specifically includes the following steps:

[0021] Build a feature extraction module based on the ResNet-50 network or RETFound network to perform offline feature extraction on the input image data;

[0022] A feature aggregation module is built based on the linear mapping layer and the multi-head self-attention mechanism to aggregate the extracted image features;

[0023] The weighted weight value is calculated based on the feature aggregation result, and a feature fusion module is constructed using a weighted summation scheme to fuse the aggregated features;

[0024] A classification module is constructed based on the linear layer to obtain the final classification result of the neonatal fundus image according to the fused features.

[0025] The feature extraction module based on the ResNet-50 network or the RETFound network specifically includes the following steps:

[0026] The pre-trained ResNet-50 network is used as the feature extraction module, and the input of the global average pooling layer in the ResNet-50 network is used as the output of the feature extraction module;

[0027] or

[0028] The pre-trained RETFound network is used as the feature extraction module, and the input of the last transformer layer in the RETFound network is used as the output of the feature extraction module;

[0029] During the classification model training or actual application, all parameters in the feature extraction module remain unchanged.

[0030] The feature aggregation module based on the linear mapping layer and the multi-head self-attention mechanism is constructed, and specifically includes the following steps:

[0031] The output of the feature extraction module is concatenated with the classification tokens and position encodings to be learned to obtain the intermediate features; it is expressed as: In the formula is an intermediate feature; is the classification token to be learned; is the output of the feature extraction module, corresponding to the output of the feature extraction module of the i-th fundus image of a newborn baby as input; Encode for position; For splicing operation;

[0032] The intermediate features are processed through three linear mapping layers to obtain Q-value features, K-value features, and V-value features; the Q-value features, K-value features, and V-value features are exchanged through the multi-head self-attention module, and the output of the multi-head self-attention module is processed through the first linear mapping layer to obtain a linear output; finally, the linear output and the intermediate features are calculated for residuals to obtain the feature aggregation result; expressed as: In the formula is the result of feature aggregation; Represents the processing of the first linear mapping layer; Represents the processing of the multi-head self-attention module; Represents the processing of three linear mapping layers.

[0033] The weighted weight value is calculated based on the feature aggregation result, and the feature fusion module is constructed by using the weighted summation scheme, which specifically includes the following steps:

[0034] Aggregate the features Processing is performed to obtain the input of the feature fusion module ; The processing specifically comprises the following steps:

[0035] Select The first classification token The corresponding output is used as the input of the feature fusion module ;

[0036] for Calculate the weight value and use the weighted sum method to calculate the feature fusion result output by the feature fusion module;

[0037] The weight value is calculated using the following formula: In the formula for The weight value of ; w is the first parameter to be learned, V is the second parameter to be learned; L is the number of input fundus images, corresponding to the total number of all fundus images of a newborn baby input;

[0038] The following formula is used for weighted summation to calculate the feature fusion result output by the feature fusion module: In the formula It is the feature fusion result output by the feature fusion module.

[0039] The classification module constructed based on the linear layer is used to obtain the final classification result of the neonatal fundus image according to the fused features, and specifically includes the following steps:

[0040] Use linear layers to build classification modules;

[0041] The feature fusion results Input into the linear layer to obtain the final classification result of the neonatal fundus image.

[0042] The similarity loss and classification loss based on prior knowledge described in step S4 are used to train the primary model for classification of neonatal fundus images constructed in step S3, which specifically includes the following steps:

[0043] For each input fundus image, select the feature fusion result corresponding to the largest weight value The feature fusion result corresponding to the minimum weight value ;

[0044] Use the following formula to calculate and The similarity s: In the formula is the instance feature corresponding to the maximum weight; is the instance feature corresponding to the minimum weight;

[0045] The following formula is used as the calculation function of similarity loss: In the formula is the similarity loss value; Y is the category label, where 0 represents the normal category;

[0046] The following formula is used as the total loss function for model training: In the formula is the total loss function value; is the classification loss value; is the set weight value;

[0047] Finally, the total loss function is used to train the primary model for neonatal fundus image classification constructed in step S3.

[0048] The present invention also provides an imaging method including the neonatal fundus image classification method based on multi-instance classification, further comprising the following steps:

[0049] S6. The classification result of the neonatal fundus image obtained in step S5 is annotated and re-imaged on the neonatal fundus image to obtain a neonatal fundus image with the classification result.

[0050] The present invention also provides a storage medium on which a computer program is stored; when the computer program is executed by a processor, the method for classifying fundus images of newborns based on multi-instance classification is implemented.

[0051] The neonatal fundus image classification method, imaging method and storage medium based on multi-instance classification provided by the present invention perform feature extraction through multi-instance images, classify fundus images based on a classification model constructed based on a feature extraction mechanism and a multi-head self-attention mechanism, and implement model training based on similarity loss and classification loss of prior knowledge; therefore, the present invention can not only realize the classification of neonatal fundus images, but also has higher reliability and better accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The figure is a schematic diagram of the method flow of the classification method of the present invention.

[0053] Figure 2 Schematic diagram of multiple instances of fundus images of newborns in the classification method of the present invention; wherein, Figure 2 (a) is the fundus image of the newborn at the first angle. Figure 2 (b) is the fundus image of the newborn from the second angle. Figure 2(c) is the fundus image of the newborn from the third angle. Figure 2 (d) is the fundus image of the newborn at the fourth angle. Figure 2 (e) is the fundus image of the newborn at the fifth angle. Figure 2 (f) is the fundus image of the newborn at the sixth angle. Figure 2 (g) is the fundus image of the newborn at the seventh angle. Figure 2 (h) is the fundus image of the newborn at the eighth angle.

[0054] Figure 3 A weighted heat diagram of an embodiment of the classification method of the present invention.

[0055] Figure 4 A heat diagram of image block similarity according to an embodiment of the classification method of the present invention.

[0056] Figure 5 Schematic diagram of weight visualization of an embodiment of the classification method of the present invention; wherein, Figure 5 (a) is a weight visualization diagram of a normal fundus image. Figure 5 (b) is a weight visualization diagram of the fundus image in the first abnormal situation. Figure 5 (c) is a schematic diagram of the weight visualization of the fundus image in the second abnormal case.

[0057] Figure 6 The figure is a schematic diagram of the method flow of the imaging method of the present invention. DETAILED DESCRIPTION

[0058] like Figure 1 The method flow chart of the classification method of the present invention is shown as follows: The method for classifying fundus images of newborns based on multi-instance classification disclosed by the present invention comprises the following steps:

[0059] S1. Obtain an existing multi-instance dataset of neonatal fundus images and perform corresponding annotations; specifically, the steps include:

[0060] Obtain existing multiple instance data of fundus images of newborns; wherein each newborn corresponds to a number of fundus images; when annotating, only the category of the newborn is annotated, and the fundus image corresponding to the newborn is not annotated; such as Figure 2 As shown: Figure 2 These are 8 fundus images of a newborn baby from different angles. When annotating, only the state of the newborn baby is marked as abnormal ( Figure 2 Corresponding to abnormal fundus hemorrhage), but whether the abnormal state is reflected in the corresponding 8 fundus images is not marked, for example Figure 2 in Figure 2 (a) ~ Figure 2 In (e), the fundus hemorrhage can be clearly seen, but Figure 2 (f) ~ Figure 2 In (h), there is no obvious indication of fundus hemorrhage, but it is marked when annotated. Figure 2 (a) ~ Figure 2 (f) All are abnormal conditions;

[0061] S2. Preprocessing the image data set obtained in step S1 to obtain a training data set; the preprocessing includes resizing and image normalization; specifically comprising the following steps:

[0062] The image data obtained in step S1 is uniformly adjusted to a set value by adjusting the image resolution to a set value, preferably ;

[0063] Then, the image data is normalized to obtain the training data set;

[0064] S3. Based on the feature extraction model and multi-head self-attention mechanism, a primary model for neonatal fundus image classification was constructed, which included feature extraction module, feature aggregation module, feature fusion module and classification module;

[0065] A feature extraction module is constructed based on the feature extraction model to extract offline features from the input image data; a feature aggregation module is constructed based on the multi-head self-attention mechanism to aggregate the extracted image features; a feature fusion module is constructed based on the weighted summation scheme to fuse the aggregated features; and a classification module is used to obtain the final classification result of the neonatal fundus image based on the fused features.

[0066] The specific implementation includes the following steps:

[0067] A feature extraction module is constructed based on the ResNet-50 network or the RETFound network to perform offline feature extraction on the input image data. The specific steps include:

[0068] The pre-trained ResNet-50 network is used as the feature extraction module, and the input of the global average pooling layer in the ResNet-50 network is used as the output of the feature extraction module;

[0069] or

[0070] The pre-trained RETFound network is used as the feature extraction module, and the input of the last transformer layer in the RETFound network is used as the output of the feature extraction module;

[0071] During the classification model training or actual application, all parameters in the feature extraction module remain unchanged;

[0072] A feature aggregation module is constructed based on the linear mapping layer and the multi-head self-attention mechanism to aggregate the extracted image features; specifically, the following steps are included:

[0073] The feature aggregation module constructed by the present invention is a process of fine-tuning image features, which is aimed at aggregating the features of a single image;

[0074] The output of the feature extraction module is concatenated with the classification tokens and position encodings to be learned to obtain the intermediate features; it is expressed as: In the formula is an intermediate feature; is the classification token to be learned; is the output of the feature extraction module, corresponding to the output of the feature extraction module of the i-th fundus image of a newborn baby as input; Encode for position; For splicing operation;

[0075] The intermediate features are processed through three linear mapping layers to obtain Q-value features, K-value features, and V-value features; the Q-value features, K-value features, and V-value features are exchanged through the multi-head self-attention module, and the output of the multi-head self-attention module is processed through the first linear mapping layer to obtain a linear output; finally, the linear output and the intermediate features are calculated for residuals to obtain the feature aggregation result; expressed as: In the formula is the result of feature aggregation; Represents the processing of the first linear mapping layer; Represents the processing of the multi-head self-attention module; Represents the processing of three linear mapping layers;

[0076] The weighted weight value is calculated based on the feature aggregation result, and a feature fusion module is constructed using a weighted summation scheme to fuse the aggregated features. Specifically, the following steps are included:

[0077] Aggregate the features Processing is performed to obtain the input of the feature fusion module ; The processing specifically comprises the following steps:

[0078] Select The first classification token The corresponding output is used as the input of the feature fusion module ;

[0079] for Calculate the weight value and use the weighted sum method to calculate the feature fusion result output by the feature fusion module;

[0080] The weight value is calculated using the following formula: In the formula for The weight value of ; w is the first parameter to be learned, V is the second parameter to be learned; L is the number of input fundus images, corresponding to the total number of all fundus images of a newborn baby input;

[0081] The following formula is used for weighted summation to calculate the feature fusion result output by the feature fusion module: In the formula It is the feature fusion result output by the feature fusion module;

[0082] The weighted summation process is the fusion process of multiple image features;

[0083] A classification module is constructed based on the linear layer to obtain the final classification result of the neonatal fundus image according to the fused features; specifically, the steps include:

[0084] Use linear layers to build classification modules;

[0085] The feature fusion results Input into the linear layer to obtain the final classification result of the neonatal fundus image;

[0086] S4. Using the training data set obtained in step S2, based on the similarity loss and classification loss of prior knowledge, the primary model for neonatal fundus image classification constructed in step S3 is trained to obtain a neonatal fundus image classification model; specifically comprising the following steps:

[0087] In order to improve the performance of the detection model, the scheme of the present invention introduces a similarity loss based on prior knowledge;

[0088] For each input fundus image, select the feature fusion result corresponding to the largest weight value The feature fusion result corresponding to the minimum weight value ;

[0089] Use the following formula to calculate and The similarity s: In the formula is the instance feature corresponding to the maximum weight; is the instance feature corresponding to the minimum weight value; if the input fundus of a newborn is in a normal state, then each corresponding instance (i.e., the corresponding fundus image) is in a normal state; if the input fundus of a newborn is in an abnormal state, then each corresponding instance (i.e., the corresponding fundus image) should include at least one abnormal state image of the fundus image; in this way, and Represented as the most severe and least severe instances, the similarity between the two should be high for negative samples and low for positive samples;

[0090] The following formula is used as the calculation function of similarity loss: In the formula is the similarity loss value; Y is the category label, where 0 represents the normal category;

[0091] The following formula is used as the total loss function for model training: In the formula is the total loss function value; is the classification loss value; is the set weight value, preferably 0.05;

[0092] Finally, the total loss function is used to train the primary model for neonatal fundus image classification constructed in step S3;

[0093] S5. Using the neonatal fundus image classification model obtained in step S4, the actual classification of the neonatal fundus image based on multi-instance classification is completed.

[0094] The classification method of the present invention is further described below in conjunction with an embodiment:

[0095] Figure 3 is a thermal diagram of weight values ​​in an embodiment of the classification method of the present invention, Figure 4 is the similarity heat map of cls_token and patch embedding after weight calculation; Figure 3 and Figure 4 It can be seen that the scheme of the present invention can perceive the differences in each position of the fundus image, so that more effective information can be obtained from the image and used for subsequent image classification;

[0096] Figure 5 This is a weight visualization diagram. This embodiment explores the weight of each instance after softmax, where one result is selected from each of three different categories (normal, abnormal retinopathy of prematurity, and abnormal fundus hemorrhage) for study; Figure 5 (a) is the normal category. Since they are all normal, the weight distribution is relatively even; Figure 5 (b) and (c) will assign weights to areas with obvious lesions, where Figure 5 In (b), the gray-white dividing line of the peripheral retina on the temporal side of images 2, 4, 5, and 6 is ridge-like, with avascular areas around it and popcorn-like changes in front of the ridges. In particular, the abnormal position of the fifth image is the most obvious and has received the highest attention weight. Figure 5(c) is a picture of abnormal fundus hemorrhage. There are obvious abnormal fundus hemorrhage in the first five fundus images, so the weight is mainly focused on the first five images;

[0097] pass Figure 5 It can be seen that the weight calculation process in the solution of the present invention is relatively reasonable for the prediction of normal and abnormal levels at the instance level, which also proves the effectiveness and reliability of the method of the present invention.

[0098] The effect of the method of the present invention is described below in conjunction with a comparative example:

[0099] The experiments were conducted on a dataset of 8886 samples, totaling 115621 fundus images, collected from a provincial maternal and child health care hospital. The specific details of the dataset are shown in Table 1.

[0100] Table 1 Schematic diagram of comparative experimental data sets

[0101]

[0102] The comparison indicators include four performance indicators: sensitivity (Sen), specificity (Spec), F1-score and AUC. The experiment is mainly divided into two parts, which are experiments on features extracted by different feature extractors. Each module has two types of MIL methods, which are divided into MIL without fine-tuning features and MIL with learnable parameters to fine-tune features w / o FT (without Fine-tuning) and FT (Fine-tuning). The models are trained on the training set and validation set, and then tested on the test set. The compared multiple instance learning methods are:

[0103] MaxMIL solution: proposed by Junjie Hu and other researchers in the paper "Automated analysis for retinopathy of prematurity by deep neural network" in 2018;

[0104] ABMIL: A solution proposed by Milse et al. in the paper “Attention-based deep multiple instance learning” in 2018.

[0105] DSMIL: A solution proposed by Bin Li et al. in their 2021 paper Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning.

[0106] TransMIL: a solution proposed by Zhucheng Shao et al. in the paper "Transformer based correlated multiple instance learning for whole slide image classification" in 2021;

[0107] GAttn: A solution proposed by Lu Zhao et al. in their 2023 paper “Generalized attention-based deep multi-instance learning”.

[0108] RGMIL: A solution proposed by Zhaolong Du et al. in the 2023 paper “RGMIL: Guide Your Multiple-Instance Learning Model with Regressor”.

[0109] R2T-MIL: A solution proposed by Wenhao Tang et al. in their paper “Feature Re-Embedding: Towards foundation model-level performance in computational pathology” in 2024.

[0110] The performance data obtained from the comparative experiment is shown in Table 2:

[0111] Table 2 Schematic diagram of comparative experimental performance indicators

[0112]

[0113] As can be seen from Table 2, the comparison test uses MaxMIL as the baseline and reports performance indicators with various MILs on the collected data sets. Regardless of which feature extractor is used or compared with other multiple instance learning methods, the method of the present invention can achieve the best performance in various performance indicators.

[0114] like Figure 6 The method flow diagram of the imaging method of the present invention is shown as follows: the imaging method disclosed in the present invention, which includes the neonatal fundus image classification method based on multi-instance classification, comprises the following steps:

[0115] S1. Obtain an existing multi-instance dataset of neonatal fundus images and perform corresponding annotations;

[0116] S2. The image data set obtained in step S1 is preprocessed to obtain a training data set; the preprocessing includes resizing and image normalization;

[0117] S3. Based on the feature extraction model and multi-head self-attention mechanism, a primary model for neonatal fundus image classification was constructed, which included feature extraction module, feature aggregation module, feature fusion module and classification module;

[0118] A feature extraction module is constructed based on the feature extraction model to extract offline features from the input image data; a feature aggregation module is constructed based on the multi-head self-attention mechanism to aggregate the extracted image features; a feature fusion module is constructed based on the weighted summation scheme to fuse the aggregated features; and a classification module is used to obtain the final classification result of the neonatal fundus image based on the fused features.

[0119] S4. Using the training data set obtained in step S2, based on the similarity loss and classification loss of prior knowledge, the primary model for neonatal fundus image classification constructed in step S3 is trained to obtain a neonatal fundus image classification model;

[0120] S5. The neonatal fundus image classification model obtained in step S4 is used to complete the actual classification of the neonatal fundus image based on multi-instance classification;

[0121] S6. The classification result of the neonatal fundus image obtained in step S5 is annotated and re-imaged on the neonatal fundus image to obtain a neonatal fundus image with the classification result.

[0122] The imaging method provided by the present invention can be directly applied to an existing neonatal fundus image device (such as a neonatal fundus image imaging system), or directly applied to a terminal (such as a computer); in specific application, the existing scheme is used to obtain the actual neonatal fundus image, and then the obtained data is input into the corresponding machine device or terminal. At this time, the machine device or terminal can obtain the actual neonatal fundus image classification result according to the imaging method disclosed in the present invention, and display the classification result on the original image through different types of representation (such as color), and then perform secondary imaging and output; at this time, the output image is an image with the fundus image classification result, which can reflect the actual neonatal fundus image and the classification result, thereby greatly facilitating the subsequent work of clinical medical staff and laboratory experimenters.

Claims

1. A method for classifying neonatal fundus images based on multi-instance classification, characterized in that The steps include: S1. Obtain an existing multi-instance dataset of neonatal fundus images and perform corresponding annotations; S2. The image data set obtained in step S1 is preprocessed to obtain a training data set; the preprocessing includes resizing and image normalization; S3. Based on the feature extraction model and multi-head self-attention mechanism, a primary model for neonatal fundus image classification was constructed, which included feature extraction module, feature aggregation module, feature fusion module and classification module; A feature extraction module is constructed based on the feature extraction model to perform offline feature extraction on the input image data; a feature aggregation module is constructed based on the multi-head self-attention mechanism to aggregate the extracted image features; A feature fusion module is constructed based on a weighted summation scheme to fuse the aggregated features; The classification module is used to obtain the final classification result of the neonatal fundus image based on the fused features; The specific steps include: Build a feature extraction module based on the ResNet-50 network or RETFound network to perform offline feature extraction on the input image data; A feature aggregation module is built based on the linear mapping layer and the multi-head self-attention mechanism to aggregate the extracted image features; The specific steps include: The output of the feature extraction module is concatenated with the classification tokens and position encodings to be learned to obtain the intermediate features; it is expressed as In the formula is an intermediate feature; is the classification token to be learned; is the output of the feature extraction module, corresponding to the output of the feature extraction module of the i-th fundus image of a newborn baby as input; Encode for position; For splicing operation; The intermediate features are processed through three linear mapping layers to obtain Q-value features, K-value features, and V-value features; the Q-value features, K-value features, and V-value features are exchanged through a multi-head self-attention module, and the output of the multi-head self-attention module is processed through the first linear mapping layer to obtain a linear output; finally, the linear output and the intermediate features are subjected to residual calculation to obtain the feature aggregation result; Expressed as In the formula is the result of feature aggregation; Represents the processing of the first linear mapping layer; Represents the processing of the multi-head self-attention module; Represents the processing of three linear mapping layers; The weighted weight value is calculated based on the feature aggregation result, and a feature fusion module is constructed using a weighted summation scheme to fuse the aggregated features; A classification module is constructed based on the linear layer to obtain the final classification result of the neonatal fundus image according to the fused features; S4. Using the training data set obtained in step S2, based on the similarity loss and classification loss of prior knowledge, the primary model for neonatal fundus image classification constructed in step S3 is trained to obtain a neonatal fundus image classification model; S5. Using the neonatal fundus image classification model obtained in step S4, the actual classification of the neonatal fundus image based on multi-instance classification is completed.

2. The method for classifying fundus images of newborns based on multi-instance classification according to claim 1, characterized in that The step S1 specifically includes the following steps: Acquire existing multiple instance data of fundus images of newborns; wherein each newborn corresponds to a number of fundus images; when annotating, only the category of the newborn is annotated, and the fundus image corresponding to the newborn is not annotated; The step S2 specifically includes the following steps: The image data obtained in step S1 is uniformly adjusted to a set value for the image resolution; Then, the image data is normalized to obtain the training data set.

3. The method for classifying neonatal fundus images based on multi-instance classification according to claim 2 is characterized in that The feature extraction module based on the ResNet-50 network or the RETFound network specifically includes the following steps: The pre-trained ResNet-50 network is used as the feature extraction module, and the input of the global average pooling layer in the ResNet-50 network is used as the output of the feature extraction module; or The pre-trained RETFound network is used as the feature extraction module, and the input of the last transformer layer in the RETFound network is used as the output of the feature extraction module; During the classification model training or actual application, all parameters in the feature extraction module remain unchanged.

4. The method for classifying fundus images of newborns based on multi-instance classification according to claim 3 is characterized in that The weighted weight value is calculated based on the feature aggregation result, and the feature fusion module is constructed by using the weighted summation scheme, which specifically includes the following steps: Aggregate the features Processing is performed to obtain the input of the feature fusion module ; The processing specifically comprises the following steps: Select The first classification token The corresponding output is used as the input of the feature fusion module ; for Calculate the weight value and use the weighted sum method to calculate the feature fusion result output by the feature fusion module; The weight value is calculated using the following formula: In the formula for The weight value of ; w is the first parameter to be learned, V is the second parameter to be learned; L is the number of input fundus images, corresponding to the total number of all fundus images of a newborn baby input; The following formula is used for weighted summation to calculate the feature fusion result output by the feature fusion module: In the formula It is the feature fusion result output by the feature fusion module.

5. The method for classifying neonatal fundus images based on multi-instance classification according to claim 4 is characterized in that The classification module constructed based on the linear layer is used to obtain the final classification result of the neonatal fundus image according to the fused features, and specifically includes the following steps: Use linear layers to build classification modules; The feature fusion results Input into the linear layer to obtain the final classification result of the neonatal fundus image.

6. The method for classifying neonatal fundus images based on multi-instance classification according to claim 5 is characterized in that The similarity loss and classification loss based on prior knowledge described in step S4 are used to train the primary model for classification of neonatal fundus images constructed in step S3, which specifically includes the following steps: For each input fundus image, select the feature fusion result corresponding to the largest weight value The feature fusion result corresponding to the minimum weight value ; Use the following formula to calculate and The similarity s: In the formula is the instance feature corresponding to the maximum weight; is the instance feature corresponding to the minimum weight; The following formula is used as the calculation function of similarity loss: In the formula is the similarity loss value; Y is the category label, where 0 represents the normal category; The following formula is used as the total loss function for model training: In the formula is the total loss function value; is the classification loss value; is the set weight value; Finally, the total loss function is used to train the primary model for neonatal fundus image classification constructed in step S3.

7. An imaging method comprising the method for classifying neonatal fundus images based on multi-instance classification according to any one of claims 1 to 6, characterized in that The following steps are also included: S6. The classification result of the neonatal fundus image obtained in step S5 is annotated and re-imaged on the neonatal fundus image to obtain a neonatal fundus image with the classification result.

8. A storage medium having a computer program stored thereon; when the computer program is executed by a processor, the method for classifying fundus images of newborns based on multi-instance classification according to any one of claims 1 to 6 is implemented.

Images