Neonatal ROP identification and labeling method, system, storage medium and electronic device

Abstract

The present invention provides a method, system, storage medium and electronic device for identifying and labeling ROP in neonates, and relates to the field of image processing technology. The present invention improves the accuracy of ROP identification and labeling through a unified algorithm in the image domain and a method using a dual network structure, thereby solving the technical problem of low accuracy of existing ROP identification tools, and eliminating the problem of model performance deviation caused by the different domains of training data generated by different devices, thereby improving the accuracy of ROP identification and labeling. At the same time, the present invention adopts a dual identification and labeling network to respectively identify and label different features in the ROP image, avoiding the problem of different data imbalance of different features causing the network to pay different attention to different features, thereby accurately identifying two different features and achieving a better labeling effect for ROP lesions.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method, system, storage medium and electronic device for identifying and labeling ROP in newborns. Background Art

[0002] Retinal retinal detachment (ROP) is a retinal disease that primarily affects premature, low-birth-weight infants. It is characterized by rapid progression, a short treatment window, and significant visual impairment. If not promptly detected and treated, some ROP cases can develop into retinal detachment, causing permanent vision damage. Therefore, timely detection and diagnosis of ROP is crucial for protecting normal vision development in infants and young children.

[0003] Studies have shown that the occurrence of ROP is usually accompanied by the appearance of two important characteristics: one characteristic is called the stage of ROP, which can usually be regarded as the formation of vascular and non-vascular areas, resulting in the formation of a white band-like dividing line between the vascular and non-vascular areas, which is also called the ridge; the other characteristic is the abnormal curvature of blood vessels caused by abnormal proliferation of blood vessels.

[0004] However, in actual clinical examinations, manual judgment of these two features often leads to inaccurate results due to subjective factors. ROP identification tools are typically used to assist examiners in this judgment. However, current ROP identification tools fail to account for the domain differences caused by different imaging devices during training, resulting in low accuracy after training. Summary of the Invention

[0005] (1) Technical problems solved

[0006] In response to the shortcomings of the existing technology, the present invention provides a method, system, storage medium and electronic device for identifying and labeling ROP in newborns, which solves the technical problem of low accuracy of existing ROP identification tools.

[0007] (2) Technical solution

[0008] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0009] In a first aspect, the present invention provides a method for identifying and labeling ROP in newborns, comprising:

[0010] S1. Acquire several fundus images, unify the domain of the fundus images, and obtain a data set;

[0011] S2. Annotate the data set and divide it into a first data set and a second data set, wherein the first data set includes two categories: fundus images with ridges and normal fundus images, and the second data set includes two categories: fundus images with tortuous blood vessels and normal fundus images;

[0012] S3. Train the preset stage recognition and annotation network through the first data set, and train the preset plus recognition and annotation network through the second data set to obtain a recognition and annotation model. The recognition and annotation model includes the trained stage recognition and annotation network and the plus recognition and annotation network. The recognition and annotation model is used to uniformly identify and annotate the fundus images to be tested after domaining.

[0013] Preferably, the domain of the unified fundus image includes:

[0014] Use the circle detection method to detect the center point of the retinal circle in the fundus image, identify the horizontal diameter of the circle, and perform the following processing based on the horizontal diameter:

[0015] a. If the horizontal diameter part is not within the image range, then this type of fundus image is cleaned out;

[0016] b. The horizontal diameter is entirely within the image range, and the minimum width of the invalid area on both sides of the retinal area is greater than n% of the retinal area diameter; from the two intersection points of the diameter and the circle, pixels of n% of the diameter are taken outward along the diameter direction, and the excess outer boundaries are intercepted. The longitudinal boundaries are also intercepted at the corresponding proportion according to the distance intercepted by the horizontal boundaries, where n is an arbitrary number from 1 to 5;

[0017] c. The horizontal diameter is entirely within the image range, but the minimum width of the invalid area on both sides of the retinal area is less than n% of the retinal area diameter. If the minimum width is less than n% of the retinal area diameter, pixel filling is performed until the width is equal to n% of the retinal area diameter. The longitudinal boundary is also proportionally operated according to the distance intercepted or filled by the horizontal boundary.

[0018] Preferably, before executing step S2, the method for identifying and labeling ROP in newborns further includes:

[0019] Perform data augmentation on the dataset.

[0020] Preferably, the structure of the phase identification and annotation network and the plus identification and annotation network are the same, both including a feature extraction network and an identification and annotation network;

[0021] Among them, the feature extraction network is Resnet18;

[0022] The recognition and annotation network is a parallel overall classifier and RPN network.

[0023] Preferably, the method for identifying and labeling ROP in newborns further comprises:

[0024] S4. Identify and annotate the test samples through the recognition and annotation model, output the classification results and annotation results of all samples, and record the IOU data of positive samples respectively.

[0025] Preferably, when the recognition and annotation model identifies and annotates the fundus image to be tested after the unified domain, the IOU score is used as a confidence evaluation indicator for evaluating the annotation results of the ROP feature annotation network. When the IOU score of the fundus image to be tested is greater than or equal to 0.5 of the average IOU in the test phase, the annotation result is credible; when the IOU score is less than 0.5 of the average IOU in the test phase, the annotation result is unreliable.

[0026] In a second aspect, the present invention provides a newborn ROP identification and labeling system, comprising:

[0027] A unified image domain module is used to obtain a number of fundus images, unify the domains of the fundus images, and obtain a data set;

[0028] a data labeling and classification module, which labels the data set and divides it into a first data set and a second data set, wherein the first data set includes two categories: fundus images with ridges and normal fundus images, and the second data set includes two categories: fundus images with curved blood vessels and normal fundus images;

[0029] The model training module is used to train the preset stage recognition and annotation network through the first data set, and to train the preset plus recognition and annotation network through the second data set to obtain a recognition and annotation model. The recognition and annotation model includes the trained stage recognition and annotation network and the plus recognition and annotation network. The recognition and annotation model is used to uniformly identify and annotate the fundus images to be tested after domain recognition.

[0030] Preferably, the domain of the unified fundus image includes:

[0031] Use the circle detection method to detect the center point of the retinal circle in the fundus image, identify the horizontal diameter of the circle, and perform the following processing based on the horizontal diameter:

[0032] a. If the horizontal diameter part is not within the image range, then this type of fundus image is cleaned out;

[0033] b. The horizontal diameter is entirely within the image range, and the minimum width of the invalid area on both sides of the retinal area is greater than n% of the retinal area diameter; from the two intersection points of the diameter and the circle, pixels of n% of the diameter are taken outward along the diameter direction, and the excess outer boundaries are intercepted. The longitudinal boundaries are also intercepted at the corresponding proportion according to the distance intercepted by the horizontal boundaries, where n is an arbitrary number from 1 to 5;

[0034] c. The horizontal diameter is entirely within the image range, but the minimum width of the invalid area on both sides of the retinal area is less than n% of the retinal area diameter. If the minimum width is less than n% of the retinal area diameter, pixel filling is performed until the width is equal to n% of the retinal area diameter, and the longitudinal boundary is also proportionally filled according to the horizontal boundary filling distance.

[0035] In a third aspect, the present invention provides a computer-readable storage medium storing a computer program for identifying and labeling ROP in newborns, wherein the computer program enables a computer to execute the method for identifying and labeling ROP in newborns as described above.

[0036] In a fourth aspect, the present invention provides an electronic device, comprising:

[0037] One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs include methods for executing the neonatal ROP identification and labeling method as described above.

[0038] (3) Beneficial effects

[0039] The present invention provides a method, system, storage medium, and electronic device for identifying and labeling ROP in newborns. Compared with the existing technology, it has the following advantages:

[0040] The present invention improves the accuracy of ROP identification and annotation by using a unified algorithm in the image domain and a dual-network structure. This solves the technical problem of low accuracy of existing ROP identification tools and eliminates the problem of model performance deviation caused by the different domains of training data generated by different devices, thereby improving the accuracy of ROP identification and annotation. At the same time, the present invention uses a dual recognition and annotation network to separately identify and annotate different features in ROP images, avoiding the problem of data imbalance between different features that causes the network to pay different attention to different features. This allows for accurate identification of two different features, achieving better annotation of ROP lesions. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1 4 is a block diagram of a method for identifying and labeling ROP in newborns according to an embodiment of the present invention;

[0043] Figure 2 A feature extraction network with a spatial attention module added in an embodiment of the present invention;

[0044] Figure 3 This is the identification and annotation network in the embodiment of the present invention. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0046] The embodiments of the present application solve the technical problem of low accuracy of existing ROP recognition tools by providing a method, system, storage medium and electronic device for identifying and labeling ROP in newborns, and eliminate the problem of model performance deviation caused by the different domains of fundus images in training data generated by different devices, thereby improving the accuracy of ROP recognition and labeling.

[0047] The technical solution in the embodiments of the present application is to solve the above technical problems, and the overall idea is as follows:

[0048] The AI ​​labeling method for ROP provided in the embodiment of the present invention mainly preprocesses and labels ROP lesions in fundus images of premature infants without retinal detachment. Compared with previous ROP identification tools, the tool provided in the embodiment of the present invention can simultaneously output ROP classification and labeling results, and improve the accuracy of ROP identification and labeling through a unified algorithm in the image domain and a dual-network structure method. Finally, combined with credibility checks, the interpretability of the algorithm and the usability of auxiliary diagnosis are enhanced.

[0049] In order to better understand the above technical solution, the above technical solution will be described in detail below with reference to the accompanying drawings and specific implementation methods.

[0050] An embodiment of the present invention provides a method for identifying and labeling ROP in newborns, the method comprising:

[0051] S1. Acquire several fundus images and unify the domains of the fundus images to obtain a dataset;

[0052] S2. Annotate the data set and divide it into a first data set and a second data set, wherein the first data set includes two categories: fundus images with ridges and normal fundus images, and the second data set includes two categories: fundus images with tortuous blood vessels and normal fundus images;

[0053] S3. Train the preset stage recognition and annotation network through the first data set, and train the preset plus recognition and annotation network through the second data set to obtain a recognition and annotation model. The recognition and annotation model includes the trained stage recognition and annotation network and the plus recognition and annotation network. The recognition and annotation model is used to uniformly identify and annotate the fundus images to be tested after domaining.

[0054] The embodiments of the present invention improve the accuracy of ROP identification and annotation by using a unified algorithm in the image domain and a dual-network structure. This solves the technical problem of low accuracy of existing ROP identification tools and eliminates the problem of model performance deviation caused by the different domains of training data generated by different devices, thereby improving the accuracy of ROP identification and annotation. At the same time, the embodiments of the present invention use a dual recognition and annotation network to separately identify and annotate different features in the ROP image, avoiding the problem of data imbalance of different features causing the network to pay different attention to different features. This allows the accurate identification of two different features, achieving better annotation effect for ROP lesions.

[0055] The following is a detailed description of each step:

[0056] In step S1, several fundus images are acquired and their domains are unified to obtain a dataset. The specific implementation process is as follows:

[0057] Fundus images taken by different devices may have different domains. Specifically, the devices process fundus images differently, resulting in different sizes of invalid areas outside the visible retinal area in the fundus image, or the devices crop the retinal area to a certain extent, causing the shape of the visible area to change. The differences in the visible areas of the images mentioned above may be identified as classification features during the training process, resulting in a shift in the training results. The embodiments of the present invention pre-process fundus images with a unified algorithm for the fundus image domain to solve the above problems.

[0058] Domain-specific methods for unifying fundus images include:

[0059] Use the circle detection method to detect the center point of the retinal circle in the fundus image, identify the horizontal diameter of the circle, and perform the following processing based on the horizontal diameter:

[0060] 1. The horizontal diameter part is not within the image range: This indicates that the areas on both sides of the retina are cropped and there is information missing. Such images are removed from the dataset.

[0061] 2. If the entire horizontal diameter is within the image range, and the minimum width of the ineffective areas on both sides of the retinal area is greater than n% of the retinal area diameter, pixels n% of the diameter are taken from the two intersection points of the diameter and the circle, and the excess outer boundary is cut off. The vertical boundary is also cut off at a corresponding ratio based on the distance cut off for the horizontal boundary. n is any number from 1 to 5. In this embodiment of the present invention, n is 2.

[0062] 3. If the entire horizontal diameter is within the image range, but the minimum width of the invalid area on both sides of the retinal area is less than n% of the retinal area diameter: If the minimum width is less than n% of the retinal area diameter, fill the pixels until the width equals n% of the retinal area diameter. When performing the above operation, the vertical boundary is also proportionally filled according to the horizontal boundary filling distance.

[0063] Finally, the fundus images are resized to obtain a dataset. In the embodiment of the present invention, the resized images are resized to 640 pixels × 480 pixels.

[0064] In step S2, the data set is annotated and divided into a first data set and a second data set. The first data set includes two categories: fundus images with ridges and normal fundus images, and the second data set includes two categories: fundus images with curved blood vessels and normal fundus images. The specific implementation process is as follows:

[0065] It should be noted that, before executing step S2, the method further includes: performing data augmentation on the data set, and the data augmentation method includes random rotation, random flipping and other methods.

[0066] All data were first classified, with each sample assigned two classification labels: the presence of staging features (ridges) and the presence of a plus feature (abnormal vessel curvature). Samples with ridges were then labeled with rectangular boxes, while samples with curved vessels were labeled with rectangular boxes for areas of abnormal vessel curvature. Two datasets were then constructed using the same data: the first dataset was divided into two categories: ridges and normal, and the second dataset was divided into two categories: curved vessels and normal.

[0067] In step S3, a preset stage recognition and annotation network is trained using the first data set, and a preset plus recognition and annotation network is trained using the second data set to obtain a recognition and annotation model. The recognition and annotation model includes the trained stage recognition and annotation network and the plus recognition and annotation network. The recognition and annotation model is used to uniformly recognize and annotate the fundus images to be tested after domaining. The specific implementation process is as follows:

[0068] This embodiment of the present invention uses a dual recognition and annotation network for feature recognition and annotation. The two networks have the same structure. During feature recognition and annotation, data is input into both networks simultaneously, and the final output is the ROP staging and plus disease annotation results. The functions of the two networks are described below:

[0069] The preset staging recognition and annotation network is used to identify the presence of ridges in fundus images and to detect the staging features of ROP, and the plus recognition and annotation network is used to identify the abnormal vascular curvature features of plus. The reason for using a dual recognition and annotation network: In common ROP examination data sets, due to the different shooting angles of fundus images and the different development levels of the disease, there is a difference in the number of images with staging features and plus features in the data set, resulting in an imbalance in the amount of data for the two features. The advantage of using a dual recognition and annotation network is that different networks separately identify and annotate different features in the ROP image, avoiding the problem of data imbalance of different features causing the network to pay different attention to different features, thereby accurately identifying two different features and achieving better annotation effects for ROP lesions. The following is a detailed description of the two recognition and annotation networks:

[0070] The staging recognition and annotation network and the plus recognition and annotation network are structurally identical, both consisting of a feature extraction network and a recognition and annotation network. The first half of the feature extraction network uses ResNet-18 for feature extraction, while the second half of the recognition and annotation network uses a parallel holistic classifier and a Region Proposal Network (RPN) for ROP classification and lesion annotation.

[0071] The feature extraction network Resnet18 in the first half is often used to extract high-dimensional features from images and use the extracted features for subsequent classification tasks. The Resnet18 network used here is as follows Figure 2 As shown, the network consists of a 7×7 convolutional layer and eight residual blocks. Each residual block consists of two 3×3 convolutional layers. Residual connections are set up in the residual blocks to prevent model degradation. Batch Normalization layers are set after the 3×3 convolutional layers to control gradient explosion, prevent gradient vanishing, and accelerate model training. Considering the convolutional and fully connected layers, the Resnet18 network has a total of 18 weight layers. The residual connections of the residual blocks are expressed as:

[0072] H(x)=F(x)+x

[0073] Where x is the input of the residual block, F includes two 3×3 convolution operations, two Batch Normalization operations, and a ReLU activation function, H(x) represents the output of the residual block, and the + sign represents the direct element-wise addition of the matrix.

[0074] In addition, to better extract ridge and vascular region features for classification, a spatial attention mechanism was incorporated into the Resnet18 network. The attention calculation module is placed between residual blocks. The multidimensional features extracted by each residual block are input into the spatial attention module after an activation function to calculate the spatial attention matrix. This matrix is ​​then multiplied with the pre-calculated multidimensional matrix elements. This strengthens features in the feature map that have a significant impact on classification and suppresses irrelevant features. The spatial attention module consists of a maximum pooling layer, an average pooling layer, and a 7×7×2 convolutional layer. The formula is as follows:

[0075] M(G)=σ(f 7×7 ([AvgPool(G);(MaxPool(G)]))

[0076] G represents the input feature matrix. After average pooling (AvgPool) and maximum pooling (MaxPool), two one-dimensional matrices are obtained. These two matrices are concatenated and input to the convolutional layer. Finally, the activation function is applied to generate the attention map M(G). The corresponding elements of M(G) and G are multiplied to obtain the attention-adjusted features.

[0077] The parallel overall classifier + RPN network in the second half is as follows Figure 3 As shown, the features extracted from different segments of the first half of the Resnet18 network are received respectively. The overall classifier adopts the structure of AdaptiveAvgPooling layer + Softmax layer, and the input is the feature matrix output by the 8 residual blocks of Resnet18, which mainly performs ROP disease diagnosis based on the full-image features extracted by Resnet. The reason for adding the classifier here is that the previous feature annotation scheme adds a classifier after the feature annotation network to perform feature classification, which is based on the features after RPN network screening, that is, partial features of the image for classification, and the accuracy of the classification result depends on the accuracy of the feature screening of the RPN network. The advantage of adding the overall classifier after the Resnet network in the embodiment of the present invention is that the full-image features extracted by the Resnet network are directly used for classification, and the classification result is independent of the feature screening ability of the RPN network, providing independent constraints for network optimization.

[0078] The RPN network uses a common 3×3×256 convolutional layer + branching structure + proposal layer + POI pooling layer + FCN layer. The input features are the features output by the sixth residual block of Resnet18. This is because too low-dimensional features will increase the total number of anchors and increase the computational complexity, while too high-dimensional features will result in too few anchors and fail to accurately cover the feature points in the original image. The output of the RPN network is then input into the bounding box regression layer and the local classifier layer.

[0079] The anchors used to detect the target are set with three aspect ratios, namely (1:1, 1:2, 2:1). The initial sizes of the three anchors are designed to be (400:400, 300:600, 440:220), and the unit is pixels. The above three anchors are the largest group. In addition, two groups of anchors are set, with sizes of 50% and 25% of the initial anchors respectively. The first branch of the branch structure in the RPN network structure consists of a 1×1×18 convolution layer + a Softmax layer, which is used to determine whether all anchors contain the features to be detected; the second branch is a single 1×1×36 convolution layer, which is used to provide anchor offset features to better enable the pre-set anchors to cover the groundtruth area.

[0080] The Proposal layer combines the classification and offset features of the anchors to select the anchors that best cover the groundtruth area. The ROI pooling layer converts the features output by the Proposal layer into features of appropriate size for calculating the subsequent bounding box regression and classification results. The final loss function consists of three parts: the classification loss from the overall classifier, the classification loss from the local classifier, and the regression loss from the RPN network. The loss function is as follows:

[0081]

[0082] Where, L total represents the cross entropy loss of the overall classifier, p represents the predicted probability, and p * Indicates the true probability (Groundtruth); L cls represents the cross entropy loss of the local classifier, p i Indicates the probability of the i-th Anchors prediction, Represents the true probability of the i-th Anchors. Here, when the IOU of the i-th Anchors and the global Groundtruth is greater than 0.7, Considered as 1; N cls is the size of mini-batch, which is 256; N reg is the number of Anchors positions, which is equal to the number of input feature pixels, here 40×30=1200; λ is the balance N cls and N reg The coefficient is 4 here; t i is the parameterized coordinate vector of Anchors, related to x, y, w, h; t i * It is Groundtruth, which is calculated in advance based on the preset Anchors; L reg is smoothL1loss, and the calculation formula is as follows:

[0083]

[0084]

[0085] Where x represents the numerical difference between the predicted box and the true box. After the above steps, the overall classification result of the ROP image and the feature annotation box are obtained, completing the identification and annotation of the features in the ROP image.

[0086] During the model training process of the embodiment of the present invention, the network adopts the small batch gradient descent algorithm as the optimization algorithm; the training epoch is set to 250, the initial learning rate is set to 0.01, and the learning rate decay strategy is adopted during training to prevent the loss of the minimum value. The specific decay strategy is to adjust the learning rate with equal step size, the step interval is set to 10 epochs, and the decay degree is set to decay to 0.9 times the original learning rate.

[0087] In the specific implementation process, the method further includes: S4, identifying and labeling the test samples through the recognition and labeling model, outputting the classification results and labeling results of all samples, and recording the IOU data of the positive samples respectively. Specifically:

[0088] The test samples are fundus images of the same domain. The fundus images are input into the trained stage recognition and annotation network to obtain the stage classification results and stage annotation results.

[0089] The preprocessed fundus image is input into the trained plus recognition and annotation network to obtain the plus classification result and plus annotation result.

[0090] When any classification result of a sample is positive, the sample is considered to have ROP, and the staging feature and the plus feature are marked in the original image.

[0091] Output the classification results and labeling results of all samples, and record the IOU data of positive samples respectively.

[0092] The calculation process of IOU data includes:

[0093] Grad-CAM (Gradient-weighted Class Activation Mapping) is used for boundary annotation. It uses the following features from the last output of Resnet18 and the classification results of the overall classifier to visualize the areas in the fundus image that have the greatest impact on classification through gradient backpropagation. A rectangular boundary is then drawn around the area with the greatest impact using threshold filtering. The principle of Grad-CAM is as follows:

[0094]

[0095] Where L represents the thermal matrix obtained from gradient calculation; c represents the category; y represents the predicted probability of the category; Z is the size of the feature matrix; and A represents the data points in the feature layer. i, j, and k are the coordinate parameters of the feature points in the feature matrix. Since the main features of ROP are the presence of ridges and vascular tortuosity, areas that have a significant impact on the classification results are generally considered to be lesion areas, that is, areas with ridges or abnormal vascular tortuosity.

[0096] The IOU (intersection over union) is calculated between the boundary annotations obtained by Grad-CAM and the boundary annotations obtained by the RPN network. The IOU calculation formula is as follows:

[0097]

[0098] In the subsequent use of the recognition and annotation model, the IOU score is used as a confidence evaluation indicator for evaluating the annotation results of the ROP feature annotation network: when the IOU score of the sample to be tested is greater than or equal to 0.5 of the average IOU in the test phase, the annotation result is considered credible; when the IOU score is less than 0.5 of the average IOU in the test phase, it indicates that there is a difference between the characteristics of the network classification and the characteristics of the network annotation, and the annotation result is unreliable and needs to be further checked by the inspector.

[0099] An embodiment of the present invention further provides a neonatal ROP identification and labeling system, the system comprising:

[0100] A unified image domain module is used to obtain a number of fundus images and unify the domains of the fundus images to obtain a data set;

[0101] a data labeling and classification module, which labels the data set and divides it into a first data set and a second data set, wherein the first data set includes two categories: fundus images with ridges and normal fundus images, and the second data set includes two categories: fundus images with curved blood vessels and normal fundus images;

[0102] The model training module is used to train the preset stage recognition and annotation network through the first data set, and to train the preset plus recognition and annotation network through the second data set to obtain a recognition and annotation model. The recognition and annotation model includes the trained stage recognition and annotation network and the plus recognition and annotation network. The recognition and annotation model is used to uniformly identify and annotate the fundus images to be tested after domain recognition.

[0103] It is understandable that the neonatal ROP identification and labeling system provided by the embodiment of the present invention corresponds to the above-mentioned neonatal ROP identification and labeling method. The explanation, examples, beneficial effects, etc. of its relevant contents can refer to the corresponding contents in the neonatal ROP identification and labeling method, and will not be repeated here.

[0104] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program for identifying and labeling ROP in newborns, wherein the computer program enables a computer to execute the method for identifying and labeling ROP in newborns as described above.

[0105] An embodiment of the present invention further provides an electronic device, including:

[0106] one or more processors;

[0107] Memory; and

[0108] One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the programs including instructions for executing the neonatal ROP identification and labeling method as described above.

[0109] In summary, compared with the existing technology, the present invention has the following beneficial effects:

[0110] 1. The embodiments of the present invention improve the accuracy of ROP recognition and labeling through a unified algorithm in the image domain and a dual-network structure, thereby solving the technical problem of low accuracy of existing ROP recognition tools and eliminating the problem of model performance deviation caused by the different domains of training data generated by different devices, thereby improving the accuracy of ROP recognition and labeling.

[0111] 2. The embodiment of the present invention adopts a dual recognition and annotation network to separately identify and annotate different features in the ROP image, avoiding the problem of data imbalance of different features causing the network to pay different attention to different features, thereby accurately identifying two different features and achieving better annotation effect of ROP lesions.

[0112] 3. The embodiment of the present invention further enhances the reliability of the recognition and annotation model by calculating the collaborative credibility of the annotation results of the RPN network and the Grad-CAM module.

[0113] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0114] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for identifying and labeling ROP in newborns, characterized in that: include: S1. Acquire several fundus images, unify the domain of the fundus images, and obtain a data set; S2. Annotate the data set and divide it into a first data set and a second data set, wherein the first data set includes two categories: fundus images with ridges and normal fundus images, and the second data set includes two categories: fundus images with tortuous blood vessels and normal fundus images; S3. Train a preset stage recognition and annotation network using the first data set, and train a preset plus recognition and annotation network using the second data set to obtain a recognition and annotation model, wherein the recognition and annotation model includes the trained stage recognition and annotation network and the plus recognition and annotation network, and the recognition and annotation model is used to uniformly perform recognition and annotation on the fundus images to be tested after domaining; The domain of the unified fundus image includes: Use the circle detection method to detect the center point of the retinal circle in the fundus image, identify the horizontal diameter of the circle, and perform the following processing based on the horizontal diameter: a. If the horizontal diameter part is not within the image range, then this type of fundus image is cleaned; b. The entire horizontal diameter is within the image range, and the minimum width of the invalid area on both sides of the retinal area is greater than n% of the retinal area diameter; n% of the diameter's pixels are taken from the two intersection points of the diameter and the circle along the diameter direction outward, and the excess outer boundaries are intercepted. The longitudinal boundaries are also intercepted at the corresponding proportion according to the distance intercepted by the horizontal boundaries, where n is an arbitrary number from 1 to 5; c. The horizontal diameter is entirely within the image range, but the minimum width of the invalid area on both sides of the retinal area is less than n% of the retinal area diameter. If the minimum width is less than n% of the retinal area diameter, pixel filling is performed until the width is equal to n% of the retinal area diameter. The longitudinal boundary is also proportionally operated according to the distance intercepted or filled by the horizontal boundary.

2. The method for identifying and labeling ROP in newborns according to claim 1, wherein: Before executing step S2, the method for identifying and labeling ROP in newborns further includes: Perform data augmentation on the dataset.

3. The method for identifying and labeling ROP in newborns according to claim 1, wherein: The structure of the phase identification and annotation network and the plus identification and annotation network are the same, both including a feature extraction network and an identification and annotation network; Among them, the feature extraction network is Resnet18; The recognition and annotation network is a parallel overall classifier and RPN network.

4. The method for identifying and labeling ROP in newborns according to claim 1, wherein: The neonatal ROP identification and labeling method further includes: S4. Identify and annotate the test samples through the recognition and annotation model, output the classification results and annotation results of all samples, and record the IOU data of positive samples respectively.

5. The method for identifying and labeling ROP in newborns according to claim 4, wherein: When the recognition and annotation model identifies and annotates the fundus image to be tested after unification of the domain, the IOU score is used as the confidence evaluation indicator for evaluating the annotation results of the ROP feature annotation network. When the IOU score of the fundus image to be tested is greater than or equal to 0.5 of the average IOU in the test phase, the annotation result is credible; When the IOU score is less than 0.5 of the average IOU in the test phase, the annotation result is unreliable.

6. A newborn ROP identification and labeling system, characterized in that: include: A unified image domain module is used to obtain a number of fundus images, unify the domains of the fundus images, and obtain a data set; a data labeling and classification module, which labels the data set and divides it into a first data set and a second data set, wherein the first data set includes two categories: fundus images with ridges and normal fundus images, and the second data set includes two categories: fundus images with curved blood vessels and normal fundus images; A model training module is used to train a preset stage recognition and annotation network using the first data set, and to train a preset plus recognition and annotation network using the second data set, to obtain a recognition and annotation model, wherein the recognition and annotation model includes the trained stage recognition and annotation network and the plus recognition and annotation network, and the recognition and annotation model is used to uniformly perform recognition and annotation on the fundus images to be tested after domaining; The domain of the unified fundus image includes: Use the circle detection method to detect the center point of the retinal circle in the fundus image, identify the horizontal diameter of the circle, and perform the following processing based on the horizontal diameter: a. If the horizontal diameter part is not within the image range, then this type of fundus image is cleaned; b. The entire horizontal diameter is within the image range, and the minimum width of the invalid area on both sides of the retinal area is greater than n% of the retinal area diameter; n% of the diameter's pixels are taken from the two intersection points of the diameter and the circle along the diameter direction outward, and the excess outer boundaries are intercepted. The longitudinal boundaries are also intercepted at the corresponding proportion according to the distance intercepted by the horizontal boundaries, where n is an arbitrary number from 1 to 5; c. The horizontal diameter is entirely within the image range, but the minimum width of the invalid area on both sides of the retinal area is less than n% of the retinal area diameter. If the minimum width is less than n% of the retinal area diameter, pixel filling is performed until the width is equal to n% of the retinal area diameter. The longitudinal boundary is also proportionally filled according to the distance of the horizontal boundary filling.

7. A computer-readable storage medium, characterized in that It stores a computer program for identifying and labeling ROP in newborns, wherein the computer program enables a computer to execute the method for identifying and labeling ROP in newborns as described in any one of claims 1 to 5.

8. An electronic device, characterized in that: include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and the programs include a method for executing the neonatal ROP identification and labeling method as described in any one of claims 1 to 5.

Images