A method for identifying HEp-2 antinuclear antibody fluorescence images based on deep learning
Through a neural network based on deep learning-based manual feature guidance and multi-scale feature fusion, the problem of incomplete information extraction in high-resolution HEp-2 anti-nuclear antibody fluorescent map is solved, achieving higher recognition accuracy and better generalization performance.
Patent Information
- Application Number
- CN202311170348.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-12
- Publication Date
- 2025-07-18
- Estimated Expiration
- 2043-09-12
AI Technical Summary
The prior art is difficult to extract local texture and global shape information simultaneously in high-resolution HEp-2 anti-nuclear antibody fluorescent images, resulting in the model's low accuracy when identifying the fluorescent mode of HEp-2 anti-nuclear antibody, and ignores the characteristics of weak discriminant regions, affecting generalization performance.
Using a deep learning-based approach, neural networks are designed through manual feature guidance and multi-scale feature fusion, including CNN feature extractors, LBP and HOG manual feature extractors, combined with global average pooling and transformation functions, consistency losses are calculated, and neural networks are trained to identify karyotype categories of HEp-2 anti-nuclear antibody fluorescence maps.
The accuracy of the model in identifying the fluorescence map of HEp-2 anti-nuclear antibody is improved, and local texture and global shape information can be effectively extracted, the generalization ability of the model is enhanced, the loss of details and global information is avoided, and the recognition effect is improved.
Smart Images
Figure CN117197806B_ABST
Abstract
Description
Technical Field
[0001] A method for identifying HEp-2 anti-nuclear antibody fluorescence images based on deep learning, which is used to identify the karyotype categories of HEp-2 anti-nuclear antibody fluorescence images, belongs to the fields of deep learning and medical image recognition. Background Art
[0002] The HEp-2 anti-nuclear antibody fluorescence pattern image is an important reference for doctors to diagnose autoimmune diseases. At present, the recognition of anti-nuclear antibody fluorescence pattern images is mainly completed by the naked eye, and requires the recognition personnel to undergo long-term training and have certain medical literacy. Since the method of manual naked eye recognition is time-consuming and laborious, and the recognition results are affected by subjective factors such as the personal experience of the recognition personnel, the academic community has been exploring an automatic recognition method for HEp-2 anti-nuclear antibody fluorescence patterns. The 21st International Conference on Pattern Recognition announced a set of preprocessed anti-nuclear antibody fluorescence images, where each image is a fluorescence image of a single cell. Classical deep models trained on this dataset can often effectively identify the fluorescence patterns of single cells. However, in a clinical environment, fluorescence pattern images are often taken at the sample level, with hundreds of cells in each image, and there may be multiple fluorescence patterns among these hundreds of cells. Doctors need to make a comprehensive diagnosis based on the number and types of fluorescence patterns present in the sample. Therefore, it is of great significance to develop a sample-level fluorescence pattern recognition system that can be used in a clinical environment. However, when classical deep classification models are directly applied to clinically captured samples, the following problems exist:
[0003] 1. Due to the high resolution of sample-level fluorescence images, it is difficult to directly input them into a deep model for training. If the fluorescence images are directly scaled and then input into the model, it will inevitably lead to the loss of detailed information such as texture. If the fluorescence images are cut and then input into the model, it will cause the model to be difficult to learn global information.
[0004] 2. The model often only focuses on the features of the most discriminative regions and ignores the features of other less discriminative regions, resulting in poor generalization performance of the model. Summary of the Invention
[0005] Aiming at the problems in the above research, the purpose of the present invention is to provide a method for identifying HEp-2 anti-nuclear antibody fluorescence images based on deep learning, so as to solve the problem of poor recognition accuracy of HEp-2 anti-nuclear antibody fluorescence pattern recognition in the prior art.
[0006] To achieve the above purpose, the present invention adopts the following technical solutions:
[0007] A method for identifying HEp-2 anti-nuclear antibody fluorescence images based on deep learning, comprising:
[0008] Step 1: Collect HEp-2 anti-nuclear antibody fluorescence images and corresponding labels.
[0009] Step 2: Scale and cut the fluorescence images so that each sub-image has the same scale.
[0010] Step 3: Design a neural network based on manual feature guidance and multi-scale feature fusion.
[0011] Step 4: Train the neural network.
[0012] Step 5: Input the HEp-2 anti-nuclear antibody fluorescence image with the label to be predicted to obtain its label.
[0013] Furthermore, the specific steps of Step 1 are as follows: Collect HEp-2 anti-nuclear antibody fluorescence images from clinical channels, manually identify their fluorescence patterns, and assign real labels y. The real label y includes eight labels, namely homogeneous type, Golgi type, nucleolar type, centromere type, nuclear membrane type, cytoplasmic type, speckled type, and nuclear dot type.
[0014] Furthermore, the specific steps of Step 2 are as follows:
[0015] Scale the collected original fluorescence images into pictures with a resolution of 2400*2400, and then cut them into 12*12 sub-images, each sub-image with a resolution of 200*200, to obtain a sub-image set S1. Scale the original fluorescence images into pictures with a resolution of 800*800, and then cut them into 4*4 sub-images, each sub-image with a resolution of 200*200, to obtain a sub-image set S2.
[0016] Furthermore, the specific steps of Step 3 are as follows:
[0017] The neural network consists of the following modules:
[0018] CNN feature extractor F1. Input the sub-image set S1 into F1, and the output of the hidden layer is The feature output is
[0019] CNN feature extractor F2. Input the sub-image set S2 into F2, and the output of the hidden layer is The feature output is
[0020] LBP manual feature and Consistency loss calculation module. Input the sub-image set S1 into the LBP manual feature extractor to extract the texture information of the fluorescence image to obtain the manual feature A, and then calculate the consistency loss L1 between the obtained hidden layer output After global average pooling and the transformation function F ψ And the manual feature A;
[0021] HOG manual feature and Consistency loss calculation module. The sub-image set S2 is input into the HOG manual feature extractor to extract the edge shape information of the fluorescence image, resulting in the manual feature B. Then, the obtained hidden layer output is passed through global average pooling and a transformation function and then calculates the consistency loss L2 with the manual feature B;
[0022] Feature fusion and classification module. The feature output of each sub-image in the sub-image set S1 is concatenated with the feature output of the corresponding sub-image in the corresponding sub-image set S2 to form the feature H ω , and then H ω is input into the classifier C, and the classifier outputs a label vector Label vector takes the maximum value among instances and calculates the cross-entropy loss L with the true label y ce ;
[0023] Calculate the total loss L total , which is the weighted sum of L ce , L1, and L2.
[0024] L total = L ce + λ(L1 + L2).
[0025] Furthermore, the specific steps of step 4 are as follows:
[0026] The data set is divided into a training set, a validation set, and a test set in a ratio of 8:1:1. The adjustable hyperparameter λ takes values of 0.1, 0.2, 0.5, and 1. During training, L total is used as the model loss, and the neural network is trained using the gradient descent strategy and the Adam optimizer for multiple iterations. When the prediction accuracy no longer improves for 5 consecutive epochs on the validation set, the model is saved, and the models saved under different hyperparameters are compared, and the one with the highest prediction accuracy is used as the final model.
[0027] Furthermore, the specific steps of step 5 are as follows:
[0028] The HEp-2 anti-nuclear antibody fluorescence image of the fluorescence pattern to be recognized is input into the model to obtain its label.
[0029] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0030] Using manual features to guide the model to learn the most important features for fluorescence pattern determination, by concatenating the features of sub-images of different sizes, the model can extract both small-scale local texture information and large-scale shape information, and synthesize the above information to improve the model's ability to recognize fluorescence patterns, specifically:
[0031] 1. Scale the original fluorescence image at different scaling rates, extract features and then splice them, so that the extracted features contain both relatively detailed information such as texture information and relatively large-scale information such as shape information. Extract multiple features of the fluorescence image from different scales, enabling the model to have better generalization ability.
[0032] 2. Use the HOG handcrafted feature extractor to extract the edge shape information of the fluorescence image, and use the LBP handcrafted feature extractor to extract the texture information of the fluorescence image, and respectively guide the CNN feature extractors of the sub-images with different scaling rates, so that the CNN feature extractors of the sub-images with different scaling rates respectively focus on capturing texture information and edge shape information. Significantly alleviates the problem that a single model only extracts features of the most discriminative regions while ignoring the features of other less discriminative regions.
[0033] 3. In the present invention, the features extracted from the sub-images of two scaling rates are spliced. Global information is extracted from the smaller-scaled image; detailed information is extracted from the unscaled and cut image; by combining these two, the loss of detailed information and global information is avoided. Description of the Drawings
[0034] Figure 1 It is a simplified diagram of the neural network module. Detailed Embodiment
[0035] The following will give a detailed description of the embodiments of the present invention. Although the present invention will be described and illustrated in conjunction with some specific embodiments, it should be noted that the present invention is not limited only to these embodiments. On the contrary, any modifications or equivalent replacements made to the present invention shall be covered within the scope of the claims of the present invention.
[0036] In addition, for better illustration of the present invention, numerous specific details are given in the following detailed embodiments. Those skilled in the art will understand that the present invention can also be implemented without these specific details.
[0037] The present invention will be further described below in conjunction with the drawings and specific embodiments.
[0038] The present invention proposes a method for extracting the fluorescence pattern of anti-nuclear antibody fluorescence images based on a feature pyramid extractor guided by handcrafted features, which solves the problem of poor recognition accuracy of the fluorescence pattern of anti-nuclear antibody fluorescence images in the prior art.
[0039] Step 1: Collect HEp-2 anti-nuclear antibody fluorescence images and corresponding labels, where each fluorescence image corresponds to one or more labels.
[0040] Step 2: Scale and cut the fluorescence images so that the scales of each instance image are the same.
[0041] Step 3: Design the neural network module.
[0042] Step 4: Train the neural network.
[0043] Step 5: Input the HEp-2 antinuclear antibody fluorescence image whose label needs to be predicted, and obtain its label.
[0044] The main process of the present invention includes: 1) Collect HEp-2 antinuclear antibody fluorescence images and corresponding labels; 2) Scale and cut the fluorescence images; 3) Design the neural network module; 4) Train the neural network; 5) Input the HEp-2 antinuclear antibody fluorescence image whose label needs to be predicted, and obtain its label. The specific implementation steps are as follows.
[0045] I. Collect HEp-2 antinuclear antibody fluorescence images and corresponding labels
[0046] Collect HEp-2 antinuclear antibody fluorescence images from clinical channels, manually identify their fluorescence patterns, and assign them eight labels, namely homogeneous type, Golgi body type, nucleolar type, centromere type, nuclear membrane type, cytoplasmic type, speckled type, and nuclear dot type.
[0047] II. Scale and cut the fluorescence images
[0048] Scale the collected original fluorescence images into images with a resolution of 2400*2400, and then cut them into 12*12 sub-images S1, each sub-image with a resolution of 200*200. Scale the original fluorescence images into images with a resolution of 800*800, and then cut them into 4*4 sub-images S2, each sub-image with a resolution of 200*200.
[0049] III. Design the neural network module, as Figure 1 shown:
[0050] The neural network consists of the following modules:
[0051] CNN feature extractor F1: Input the sub-image set S1 into F1, and the output of the hidden layer is The feature output is
[0052] CNN feature extractor F2: Input the sub-image set S2 into F2, and the output of the hidden layer is The feature output is
[0053]
[0054]
[0055] LBP handcrafted features and Consistency loss calculation module. Input the sub-image set S1 into the LBP handcrafted feature extractor to extract the texture information of the fluorescence image, obtaining the handcrafted feature A. Then, for the obtained hidden layer output perform global average pooling and pass it through the transformation function F ψ and calculate the consistency loss L1 with the handcrafted feature A;
[0056]
[0057] HOG handcrafted feature and Consistency loss calculation module. Input the sub-image set S2 into the HOG handcrafted feature extractor to extract the edge shape information of the fluorescence image, obtaining the handcrafted feature B. Then, for the obtained hidden layer output perform global average pooling and pass it through the transformation function and calculate the consistency loss L2 with the handcrafted feature B;
[0058]
[0059] Feature fusion and classification module. Concatenate the feature output of each sub-image in the sub-image set S1 with the corresponding feature output of the corresponding sub-image in the sub-image set S2 to form the feature H ω , then input H ω into the classifier C, and the classifier outputs the label vector The label vector takes the maximum value among instances and calculates the cross-entropy loss L with the true label y ce ;
[0060]
[0061] Calculate the total loss, which is the weighted sum of L ce , L1, and L2;
[0062] L total = L ce + λ(L1 + L2)
[0063] where λ is an adjustable hyperparameter.
[0064] IV. Training the neural network
[0065] Divide the dataset into training set, validation set, and test set with a ratio of 8:1:1. The values of λ are 0.1, 0.2, 0.5, and 1. During training, use L total as the model loss, train the neural network using the gradient descent strategy, and use the Adam optimizer for multiple iterations. When the prediction accuracy no longer improves for 5 consecutive epochs on the validation set, save the model, compare the models saved under different hyperparameters, and select the model with the highest prediction accuracy as the final model.
[0066] 5. Input the HEp-2 antinuclear antibody fluorescence image for which the label needs to be predicted to obtain its label.
Claims
1. A method for identifying HEp-2 antinuclear antibody fluorescence images based on deep learning, characterized in that, It includes the following steps: Step 1: Collect HEp-2 antinuclear antibody fluorescence images and assign corresponding true labels y; Step 2: Scale and cut the fluorescence images so that each sub-image has the same scale, obtaining a sub-image set and the sub-image set ; Step 3: Design a neural network based on the guidance of manual features and multi-scale feature fusion; Step 4: Train the neural network; Step 5: Input the HEp-2 antinuclear antibody fluorescence image whose label needs to be predicted to obtain its label; The specific steps of step 2 are as follows: Scale the collected original fluorescence images to pictures with a resolution of 2400*2400, and then cut them into 12*12 sub-images, each with a resolution of 200*200, to obtain a sub-image set , scale the original fluorescence images to pictures with a resolution of 800*800, and then cut them into 4*4 sub-images, each with a resolution of 200*200, to obtain a sub-image set ; The specific steps of step 3 are as follows: The neural network consists of the following modules: CNN Feature Extractor , the subset of images is input , and the output of the hidden layer is , and the feature output is ; CNN Feature Extractor , the subset of images is input , the output of the hidden layer is , and the feature output is ; LBP handcrafted feature and The consistency loss calculation module, takes the sub-image set as input to the LBP handcrafted feature extractor to extract the texture information of the fluorescence image into handcrafted feature A, and then takes the obtained hidden layer output through global average pooling and transformation function to calculate the consistency loss with handcrafted feature A ; HOG handcrafted feature and consistency loss calculation module, input the sub-image set into the HOG handcrafted feature extractor to extract the edge shape information of the fluorescence image to the handcrafted feature B, and then calculate the consistency loss between the obtained hidden layer output after global average pooling and the transformation function and the handcrafted feature B ; The feature fusion and classification module combines the feature outputs of each sub - graph in the sub - graph set with the feature outputs of the corresponding sub - graphs in its corresponding sub - graph set to form a feature . Then, it inputs the into a classifier . The classifier outputs a label vector . The maximum value of the label vector is taken among instances and cross - entropy loss is calculated with the true label ; Calculate the total loss , , are hyperparameters.
2. The method for identifying HEp-2 antinuclear antibody fluorescence images based on deep learning according to claim 1, wherein The specific steps of step 1 are as follows: Collect HEp-2 antinuclear antibody fluorescence images from clinical channels, manually identify their fluorescence patterns, and assign true labels y. The true labels y include eight labels, namely homogeneous type, Golgi body type, nucleolar type, centromere type, nuclear membrane type, cytoplasmic type, speckled type, and nuclear dot type.
3. A method for identifying HEp-2 antinuclear antibody fluorescence images based on deep learning according to claim 1, characterized in that, The specific steps of step 4 are as follows: Divide the dataset into a training set, a validation set, and a test set in a ratio of 8:1:1, which is an adjustable hyperparameter The values are 0.1, 0.2, 0.5, and 1. During training, use as the model loss, use the gradient descent strategy to train the neural network, and use the Adam optimizer for multiple iterations. When the prediction accuracy no longer improves for 5 consecutive epochs on the validation set, save the model, compare the models saved under different hyperparameters, and use the one with the highest prediction accuracy as the final model.