Method and device for ANA typing in immunofluorescence images

Abstract

The present invention relates to immunofluorescence image processing technology, and discloses a method and device for ANA typing of immunofluorescence images, which constructs an ANA atlas library by processing immunofluorescence images and ANA diagnostic information; labels the data of the ANA atlas library through label filling and One-Hot encoding; extracts features by manual features and deep pre-trained features of a large language model; obtains an ANA karyotype multi-label classification model through a multi-label classification algorithm; and is used to feedback and update the multi-label classification model. The present invention combines manual feature extraction, deep pre-trained features and CNN deep features to construct a multi-label classification model. There is no need to manually mark the region of interest, and all ANA karyotype situations in clinical practice can be addressed end-to-end, with good scalability; a virtuous cycle of data and model is formed with the help of a feedback mechanism, which is better applied to clinical practice.

Description

Technical Field

[0001] The present invention relates to immunofluorescence image processing technology, and in particular to a method and device for ANA typing of immunofluorescence images. Background Art

[0002] Autoimmune diseases (AIDs) are a group of diseases caused by the body's loss of tolerance to self-antigens, which in turn triggers the immune system to attack its own tissues, ultimately leading to damage to various systemic tissues and organs. Currently, the diagnosis of AIDs relies primarily on laboratory tests, among which the indirect immunofluorescence assay (IIFA) on HEP-2 cells to identify antinuclear antibodies (ANA) is a standard procedure supporting AID diagnosis. The accuracy of determining the ANA's positive or negative status and specific karyotype directly impacts the diagnosis of AID and the determination of treatment options.

[0003] In actual clinical practice, ANA-positive karyotypes are complex, including atypical, rare, and complex karyotypes. When interpreting IIFA images, physicians, limited by their experience, may sometimes overlook, fail to recognize, or fail to distinguish certain ANA karyotypes. In diagnostic reports, coarse-grained ANA diagnoses, such as granularity (AC-4, 5), may be presented without precise, fine-grained diagnoses, such as fine granularity (AC-4) or coarse granularity (AC-5), while sometimes precise, fine-grained diagnoses are presented. Furthermore, physicians have a low ability to distinguish complex ANA karyotypes.

[0004] Current research on ANA classification is mainly divided into ANA classification for specimen images and ANA classification for single cells. The former's research mostly focuses on the classification of single-karyotype ANA, and the ANA karyotypes involved are only some common karyotypes, which cannot cope with complex real-world clinical scenarios. Single-cell-based ANA classification methods must first accurately segment individual cells, and cell segmentation itself is very challenging; more importantly, some karyotypes cannot be determined by a single cell and require the combination of multiple cells for diagnosis. In addition, how to obtain a sample-level diagnosis based on the classification results of multiple cells is also a difficult problem; connecting the entire process in series increases the uncertainty of correctly predicting ANA karyotypes. Many ANA classification methods based on specimen images are based on target detection algorithms, such as the R-CNN series, SSD series, and YOLO series algorithms. These methods rely on doctors to mark out regions of interest (ROIs). The labeling process is time-consuming and labor-intensive, difficult to implement on a large scale, and has low scalability. Summary of the Invention

[0005] The present invention addresses the problems in the prior art that the labeling process is time-consuming and labor-intensive, difficult to implement on a large scale, and has low scalability, and provides a method and device for ANA typing of immunofluorescence images.

[0006] In order to solve the above technical problems, the present invention is solved by the following technical solutions:

[0007] A method for ANA typing using immunofluorescence images, comprising:

[0008] S1: Constructing an ANA atlas library by processing immunofluorescence images and ANA diagnostic information;

[0009] S2: Data labeling: labeling the ANA atlas through label filling and one-hot encoding;

[0010] S3: Feature extraction, which extracts features from hand-crafted features and deep pre-trained features of a large language model;

[0011] S4: ANA karyotype multi-label classification model, obtained by multi-label classification algorithm;

[0012] S5: Data-driven multi-label classification model feedback and update, used to provide feedback and update to the multi-label classification model.

[0013] Preferably, the multi-label classification algorithm includes:

[0014] S4.1, ANA karyotype classification model multi-feature fusion, for S3 feature extraction of manual features and deep pre-trained features and classification model Backbone CNN feature map F c×h×w Fusion is performed to obtain the second feature map F″ m×h×w ; Where c, h, w are the number of channels, height and width of the feature map respectively, and m = c + 2;

[0015] S4.2, Generation of prediction results, for the second feature map F″ of the fused multi-feature ANA karyotype classification model m×h×w Producing prediction results;

[0016] S4.3, ANA karyotype classification model retraining, generate a new training set by refilling the label to retrain the ANA karyotype classification model.

[0017] Preferably, the multi-feature fusion of the ANA karyotype classification model includes:

[0018] The feature conversion obtains the converted manual feature H′ and the converted deep pre-training feature D′. The manual feature H is converted through the fully connected layer FC1 to obtain the converted manual feature H′; the deep pre-training feature D is converted through the fully connected layer FC2 to obtain the converted deep pre-training feature D′; the number of output nodes of the fully connected layer FC1 and the fully connected layer FC2 are both h×w;

[0019] The concatenation of manual features and deep pre-training feature shapes, reshape the converted manual features H′ and the converted deep pre-training features D′ into shapes that can be combined with F c×h×w The shapes spliced ​​in the channel dimension are respectively obtained as the manual feature splicing shape H″ 1×h×w and deep pre-trained feature concatenation shape D″ 1×h×w ;

[0020] The first feature map F ′ m×h×w To obtain the above F c×h×w , Hand-made feature splicing shape H″ 1×h×w and deep pre-trained feature concatenation shape D″ 1×h×w The three types of feature maps are spliced ​​according to the channel dimension to obtain the first feature map F′ m×h×w ; where m = c + 2;

[0021] The second characteristic F″ n×h×w Acquisition of the first feature map F ′ After the attention module, the second feature map F″ is obtained m×h×w .

[0022] As an example, the prediction result is generated, the second feature map F″ m×h×w Through the CSRA module, the prediction output S of each sample in different ANA categories is obtained n×t , where n is the batch size during model training and t is the number of ANA categories.

[0023] Preferably, the ANA karyotype classification model retraining includes: training a converged multi-label classification model M1;

[0024] The multi-label classification model M1 is retrained to obtain the training model M. The retraining includes label refilling: for coarse-grained labels that are missing fine-grained labels, fine-grained labels are filled; thereby obtaining a new training set. Using the new training set, the multi-label classification model M1 is fine-tuned as a pre-training model to obtain the training model M.

[0025] As a preference, the design of manual features in feature extraction includes:

[0026] Grayscale images are obtained by using the original color images of immunofluorescence images;

[0027] The calculation of the co-occurrence matrix GLCM is done by calculating the gray-level co-occurrence matrix GLCM of the gray-level image;

[0028] Obtaining manual texture features H, which are calculated based on the gray-level co-occurrence matrix GLCM; texture features H include contrast, difference, angular second moment ASM, homogeneity, energy and autocorrelation;

[0029] The design of deep pre-training features in feature extraction includes:

[0030] To obtain the image embedding feature D of the immunofluorescence image, the CLIP model is used to encode the fluorescence image to obtain the image embedding feature D corresponding to the immunofluorescence image. The image embedding feature D is a deep pre-trained feature D.

[0031] Preferably, multi-label classification model feedback and updates include:

[0032] Construction of an independent test set: immunofluorescence images and ANA karyotype information were collected according to the method of constructing an ANA atlas library to construct an independent test set;

[0033] Update of the ANA atlas library by desensitizing and structuring the immunofluorescence images and the corresponding ANA typing diagnostic conclusions;

[0034] Fine-tuning training of the new model: for the updated ANA atlas, fine-tuning training of the new model is performed in stages; some samples of the ANA atlas are included in the independent test set. After excluding these samples, the remaining samples are used as the development data set for fine-tuning training of the new model, denoted as Dev; the model currently applied online is regarded as the old model M old ; With the continuous expansion of the ANA atlas library, when Dev adds N samples, the old model M old Use the pre-trained model to fine-tune the new model until convergence to obtain the new model M new ;

[0035] Comparison of the new model with the old model using McNemar's test new Compared with the old model M old When there is a significant difference between the two performances, the significance level is 0.05, and M new The overall accuracy is higher than that of M old , the new model is used to replace the old model currently applied online; otherwise, the old model is used.

[0036] In order to solve the above technical problems, the present invention also provides a device for ANA typing of immunofluorescence images, which is implemented by a method for ANA typing of immunofluorescence images.

[0037] The present invention has significant technical effects due to the adoption of the above technical solutions:

[0038] Based on the immunofluorescence images of HEP-2 cells, the present invention combines manual feature extraction, deep pre-training features and CNN deep features, fills coarse and fine-grained labels according to the HEp-2 cell karyotype classification tree, and constructs a multi-label classification model.

[0039] This method does not require manual annotation of regions of interest and can handle all ANA karyotype situations in clinical practice end-to-end, with good scalability;

[0040] This paper uses CNN activation map technology to visualize the decision basis of the ANA classification model;

[0041] The feedback mechanism implemented by the present invention gradually realizes the iterative upgrade of the model. The upgraded AI model will provide more accurate prediction results, thus forming a virtuous cycle of data and model, and better applied to clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a flow chart of the present invention;

[0043] Figure 2 It is the ANA digital coding scheme of the present invention;

[0044] Figure 3 Schematic diagram of the multi-feature fusion module of the present invention;

[0045] Figure 4 It is a flow chart of the model application of the present invention;

[0046] Figure 5 is a model feedback flow chart of the present invention;

[0047] Figure 6 is an example of an immunofluorescence image of the present invention;

[0048] Figure 7 This is a visualization effect diagram of the smoothed Grad CAM of the present invention. DETAILED DESCRIPTION

[0049] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0050] Example 1

[0051] Methods for ANA typing using immunofluorescence images, such as Figure 1 The methods include:

[0052] S1: Constructing an ANA atlas library by processing immunofluorescence images and ANA diagnostic information;

[0053] S2: Data labeling: labeling the ANA atlas through label filling and one-hot encoding;

[0054] S3: Feature extraction, which extracts features from hand-crafted features and deep pre-trained features of a large language model;

[0055] S4: ANA karyotype multi-label classification model, obtained by multi-label classification algorithm;

[0056] S5: Data-driven multi-label classification model feedback and update, used to provide feedback and update to the multi-label classification model.

[0057] Data labeling includes:

[0058] Label filling: fine-grained labels are filled by adding their corresponding coarse-grained labels. Since there are both coarse-grained and fine-grained labels in the diagnostic conclusion, the HEp-2 cell karyotype classification tree in the International Consensus on Antinuclear Antibody Fluorescence Model (ICAP) is used. For fine-grained labels, their corresponding coarse-grained labels are added, such as fine granular type (AC-4) is expanded to granular type (AC-4, 5) and fine granular type (AC-4).

[0059] One-hot encoding converts the ANA atlas codes into digital codes. Based on the HEp-2 cell karyotype classification tree, ICAP codes are converted into digital codes. There are 37 categories in total, coded as 0, 1, 2, ..., 36. Figure 2 The diagram shows the correspondence between the numeric code and the ICAP code, as well as the corresponding karyotype name, major category, and whether it is a leaf node in the HEp-2 cell karyotype classification tree. Because ANA typing is a multi-label classification task, a one-hot encoding of length 37, denoted as T, is used as the class label for each sample for training the multi-label classification model.

[0060] Processing of immunofluorescence images and ANA diagnostic information includes:

[0061] S1.1: Data desensitization: Desensitize sensitive data information such as immunofluorescence images and ANA diagnosis. Sensitive information includes name and contact information. Desensitization helps protect patient privacy.

[0062] S1.2: Data collection: Collect desensitized immunofluorescence images and ANA diagnostic information. Data collection is performed through the LIS system and indirect immunofluorescence experimental processing platform. The collected data includes the patient's HEp-2 cell karyotype diagnostic information, including negative, nuclear karyotype, cytoplasmic karyotype, and cell mitotic phase karyotype.

[0063] S1.3: Data standardization, standardize the collected data.

[0064] The standardized method converts the text information of HEp-2 cell ANA karyotype diagnostic information into standardized ICAP code information based on the international consensus on antinuclear antibody fluorescence model (ICAP).

[0065] The design of manual features in feature extraction includes: obtaining grayscale images, obtaining grayscale images through the original color images of immunofluorescence images; calculating the grayscale co-occurrence matrix GLCM, by calculating the grayscale co-occurrence matrix GLCM of the grayscale image; obtaining manual texture features H, calculating the manual texture features H based on the grayscale co-occurrence matrix GLCM; texture features H include contrast, difference, angular second moment (ASM), homogeneity, energy and autocorrelation.

[0066] The design of deep pre-training features in feature extraction includes: obtaining the image embedding feature D of the immunofluorescence image, encoding the fluorescence image using the CLIP model to obtain the image embedding feature D corresponding to the immunofluorescence image, and the image embedding feature D is the deep pre-training feature D. Specifically, the CLIP (Contrastive Language-Image Pre-Training) model is used to encode the fluorescence image into a feature vector of length 512;

[0067] In view of the existence of composite karyotypes, that is, the presence of multiple ANA karyotype characteristics in the same sample, a multi-label classification algorithm is used to cope with complex clinical scenarios. Specifically, the original fluorescence image I is scaled to 448×448, and the scaled fluorescence image is denoted as I ′ , the backbone of the model uses ResNet101, fluorescence image I ′ The feature maps are obtained by forward propagation through Backbone, denoted as F c×h×w , where c, h, and w are the number of channels, height, and width of the feature map, respectively. In the embodiment, the sizes of c, h, and w are 2048, 14, and 14, respectively. The sizes of the tensors involving c, h, and w in the embodiment are calculated based on this.

[0068] Multi-label classification algorithms include:

[0069] S4.1, ANA karyotype classification model multi-feature fusion, for S3 feature extraction of manual features and deep pre-trained features and classification model Backbone CNN feature map F c×h×w Fusion is performed to obtain the second feature map F″ m×h×w ;

[0070] S4.2, Generation of prediction results, for the second feature map F″ of the fused multi-feature ANA karyotype classification model m×h×w Producing prediction results;

[0071] S4.3, ANA karyotype classification model retraining, generate a new training set by refilling the label to retrain the ANA karyotype classification model.

[0072] Multi-feature fusion of ANA karyotype classification model Figure 3 Including: feature conversion to obtain the converted manual feature H ′ And the converted deep pre-training feature D ′ , the manual feature H is converted through the fully connected layer FC1 to obtain the converted manual feature H ′ ; The deep pre-training feature D is converted through the fully connected layer FC2 to obtain the converted deep pre-training feature D ′ ; The number of output nodes of the fully connected layer FC1 and the fully connected layer FC2 are both h×w;

[0073] The manual features and deep pre-trained feature shapes are spliced ​​together. The converted manual features H′ and the converted deep pre-trained features D′ are reshaped into shapes that can be spliced ​​with F in the channel dimension, and the manual feature splicing shapes H″ are obtained respectively. 1×h×w and deep pre-trained feature concatenation shape D″ 1×h×w ;

[0074] The first feature map F ′ m×h×w To obtain the above F c×h×w , Hand-made feature splicing shape H″ 1×h×w and deep pre-trained feature concatenation shape D″ 1×h×w The three types of feature maps are spliced ​​according to the channel dimension to obtain the first feature map F′ m×h×w ; where m = c + 2;

[0075] The second characteristic F″ m×h×w Acquisition of the first feature map F ′ After the attention module, the second feature map F″ is obtained m×h×wSpecifically, the attention module adopts the classic SE (Squeeze-and-Excitation) module to adaptively learn the feature map F ′ The contribution of different channel features to the final ANA classification enables the model to focus more on the information that is more critical to the ANA classification task from numerous feature information from different sources, while reducing or filtering out other irrelevant information.

[0076] Generation of prediction results, second feature map F″ m×h×w Through the CSRA module, the prediction output S of each sample in different ANA categories is obtained n×t , where n is the batch size during model training and t is the number of ANA categories.

[0077] The ANA karyotype classification model retraining includes: training a converged multi-label classification model M1;

[0078] The multi-label classification model M1 is retrained to obtain the training model M. The retraining includes label refilling: for coarse-grained labels that are missing fine-grained labels, fine-grained labels are filled; thereby obtaining a new training set. Using the new training set, the multi-label classification model M1 is fine-tuned as a pre-training model to obtain the training model M.

[0079] The head of the ANA karyotype multi-label classification model uses a multi-label classification head CSRA (Class-Specific Residual Attention) module with an attention mechanism. The number of categories is 37, which can cover all possible cases of ANA karyotype. The above feature map F″ m×h×w Through the CSRA module, the prediction output S of each sample in different ANA categories can be obtained. n×t , where n is the batch size during model training and t is the number of ANA categories. In the embodiment, n is 64 and t is 37.

[0080] During the model training phase, the cross-entropy loss is calculated based on the model's predicted output S and the true category label T of the fluorescence image. This is implemented using PyTorch's CrossEntropyLoss. Model training is performed in two phases: the first phase trains a converged multi-label classification model, denoted as M1; the second phase is model retraining, which includes label re-padding and fine-tuning. Label re-padding involves filling in the most likely fine-grained label for coarse-grained labels that lack fine-grained labels. Although label padding is used to fill in the fine-grained labels with their corresponding coarse-grained labels during data labeling, there are still samples that only have coarse-grained labels but lack specific fine-grained labels.

[0081] This situation arises from a lack of confidence due to limited experience or unclear image features, leading doctors to only assign coarse labels, even though fine-grained karyotype features are still present in the image. However, for samples with mixed coarse and fine-grained labels, simply ignoring the missing fine-grained labels can interfere with model training. Therefore, it is necessary to selectively relabel samples that have coarse labels but lack the corresponding fine-grained labels. Based on the model M1 trained in the first stage, forward inference is performed on the samples in the training set, retaining the samples with correct predictions.

[0082] If the original labels contain coarse-grained labels but lack the corresponding fine-grained labels, when the model correctly predicts the coarse labels, it also predicts the corresponding fine labels. In this case, the model prediction is also considered correct. In this case, the fine label with the highest prediction probability for each coarse label is used as the filler label, thereby obtaining the fine labels of the coarse-grained labels. In this way, a new training set is obtained.

[0083] Fine-tuning training uses a new training set and the above-trained model M1 as the pre-training model for fine-tuning training to obtain a new model M, which is used as the final model for practical application.

[0084] Multi-label classification model feedback and updates include:

[0085] Construction of an independent test set: immunofluorescence images and ANA karyotype information were collected according to the method of constructing an ANA atlas library to construct an independent test set;

[0086] Update of the ANA atlas library by desensitizing and structuring the immunofluorescence images and the corresponding ANA typing diagnostic conclusions;

[0087] Fine-tuning training of the new model: for the updated ANA atlas, fine-tuning training of the new model is performed in stages; some samples of the ANA atlas are included in the independent test set. After excluding these samples, the remaining samples are used as the development data set for fine-tuning training of the new model, denoted as Dev; the model currently applied online is regarded as the old model M old ; With the continuous expansion of the ANA atlas library, when Dev adds N samples, the old model M old Use the pre-trained model to fine-tune the new model until convergence to obtain the new model M new ;

[0088] Comparison of the new model with the old model, using McNemar's test to compare the new model M new Compared with the old model M old When there is a significant difference between the two performances, the significance level is 0.05, and M new The overall accuracy is higher than that of M old, the new model is used to replace the old model of the current online application; otherwise, the old model M is used. old .

[0089] Example 2

[0090] Based on Example 1, in the model application stage of this embodiment, the process of the ANA karyotype multi-label classification model outputting specific karyotype prediction results is as follows: Figure 4 As shown: Based on the trained model M, the original fluorescence image I undergoes the same process as above to obtain the predicted output S, and the predicted output S of the model M is obtained. ′ ; Using the sigmoid function, the prediction confidence p of different ANA karyotypes of the sample is obtained; the karyotype category with a prediction confidence greater than 0.5 is taken as the potential karyotype T ′ If the number of potential karyotypes is 0, meaning the prediction confidence for all karyotypes is less than 0.5, the karyotype category with the highest confidence is selected as the prediction result. If the number of potential karyotypes exceeds a specific value k, the category with the top k highest confidence levels is selected as the prediction result. In this specific embodiment, k is set to 3 because in actual clinical practice, doctors usually report a maximum of three karyotypes when issuing diagnostic reports, which meets clinical needs.

[0091] When the potential karyotype T meets the above conditions ′ In cases where both positive and negative karyotypes exist, the ANA karyotype multi-label classification model may produce contradictory judgments on the positive and negative nature of the sample. In this case, to reduce missed positive diagnoses, the model only outputs positive karyotypes. For example, the model predicts potential karyotypes of negative (AC-0), homogeneous (AC-1), and finely granular (AC-4), with corresponding prediction confidence levels of 0.72, 0.68, and 0.65, respectively. After addressing this special case, the final model-predicted karyotypes are homogeneous (AC-1) and finely granular (AC-4).

[0092] Figure 6 The original fluorescence image. In the diagnostic report, the ANA karyotype of this sample is: homogeneous type (AC-1) and granular type (AC-4, 5). In other words, this example is a case of composite karyotype. The ANA karyotype predicted by the model is also: granular type (AC-4, 5) and homogeneous type (AC-1). The corresponding prediction confidence levels are 0.8826 and 0.7901, respectively.

[0093] In order to better illustrate how the model makes ANA typing decisions, the activation map can be directly obtained using the Grad-CAM algorithm based on the scaled fluorescence image and the trained model M.

[0094] With the help of Class Activation Map and CAM technology, the key areas based on which the model is classified are intuitively displayed, providing model interpretability and enhancing doctors' trust in the model's prediction results.

[0095] The Grad-CAM activation map is used, and the specific implementation is based on the MMPreTrain framework. The target layer for visualization is the last normalized layer of ResNet101, specifically backbone.layer4.2.bn3. In order to reduce the noise in CAM and highlight the key decision areas, the original heat map is smoothed by both Aug-Smooth and Eigen-Smooth at the test time to obtain the smoothed Grad CAM visualization effect. Figure 7 .

[0096] Example 3

[0097] Based on the above embodiment, this embodiment covers all situations of ANA classification in real clinical practice, but is limited to some karyotypes, such as rare karyotypes, complex karyotypes, and atypical karyotypes; the amount of data is small or even missing, resulting in a low recognition rate for these karyotypes.

[0098] In clinical practice, data with diagnostic conclusions are constantly accumulated. After data desensitization and structuring, these data are updated to the ANA atlas library to enrich and enrich the categories and numbers of ANA karyotypes. The updated ANA atlas library can be used for further iteration and updating of the AI ​​model; once the performance indicators are significantly improved, the model will be updated and applied to ANA auxiliary reading to provide more accurate prediction results. This data-driven model feedback mechanism can gradually improve the accuracy and consistency of physicians' ANA karyotype interpretations. The detailed process is as follows: Figure 5 ;

[0099] Constructing a Prospective Independent Test Set: Immunofluorescence images and ANA karyotype information were prospectively collected according to the methodology for constructing the ANA atlas. Three experts then reviewed the ANA karyotype information and reached consensus, using this as the gold standard. This independent test set was then constructed to evaluate model performance. Because some karyotypes are rare, few in number, or even missing, samples will be selectively selected to enrich the independent test set during the ongoing update of the ANA atlas, ensuring greater representativeness, diversity, and categorical balance.

[0100] The ANA atlas library is updated. During IIFA interpretation clinical practice, immunofluorescence images and corresponding ANA typing diagnostic conclusions are continuously accumulated, desensitized, and structured before being updated to the ANA atlas library. Because AI models are trained on a high-quality ANA atlas library, they potentially learn from expert knowledge and build consensus. Therefore, in clinical applications, this library can be used to identify potentially difficult samples, which have greater potential to improve model performance.

[0101] Potentially difficult samples include the following situations: (i) when the prediction results of the AI-assisted recognition system are inconsistent with the physician's interpretation results; (ii) when the ANA karyotype multi-label classification model has contradictory judgments on the positive and negative nature of the sample, that is, when the probability of predicting a negative is greater than 0.5, the probability of at least one positive karyotype is also greater than 0.5; (iii) when there are disagreements between multiple doctors or multiple models on the ANA typing conclusions of the same sample.

[0102] Fine-tune the new model in stages: Since some samples of the ANA atlas are included in the prospective independent test set, after excluding these samples, the remaining samples are used as the development data set for fine-tuning the new model, denoted as Dev. The model currently applied online is regarded as the old model, denoted as M old As the ANA atlas library continues to expand, when Dev adds N samples, the old model M old Use the pre-trained model to fine-tune the new model until convergence, and obtain the new model M new In the embodiment, N is 500.

[0103] Performance comparison of the new and old models: Based on the above prospective independent test set, the new model M is compared using the McNemar test. new Compared with the old model M old When there is a significant difference between the two performances (the significance level is 0.05), and M new The overall accuracy is higher than that of M old , the new model is used to replace the old model currently applied online; otherwise, the old model is used.

[0104] Example 4

[0105] Based on the above embodiments, this embodiment is a device for ANA typing of immunofluorescence images, which is implemented by a method for ANA typing of immunofluorescence images.

Claims

1. A method for ANA typing based on immunofluorescence images, comprising: S1: Constructing an ANA atlas library by processing immunofluorescence images and ANA diagnostic information; S2: Data labeling: labeling the ANA atlas through label filling and one-hot encoding; S3: Feature extraction, which extracts features from hand-crafted features and deep pre-trained features of a large language model; S4: ANA karyotype multi-label classification model, obtained by multi-label classification algorithm; S5: Data-driven multi-label classification model feedback and update, used to provide feedback and update to the multi-label classification model; Multi-label classification algorithms include: S4.1, ANA karyotype classification model multi-feature fusion, for S3 feature extraction of manual features and deep pre-trained features and classification model Backbone CNN feature map Fusion is performed to obtain the second feature map ; where c, h, and w are the number of channels, height, and width of the feature map, respectively, and ; S4.2, Generation of prediction results, for the second feature map of the fused multi-feature ANA karyotype classification model Producing prediction results; S4.3, ANA karyotype classification model retraining, by refilling the labels to generate a new training set for ANA karyotype classification model retraining; ANA karyotype classification model multi-feature fusion includes: The conversion of features obtains the converted manual features And the converted deep pre-trained features , the manual feature H is converted through the fully connected layer FC1 to obtain the converted manual feature ; The deep pre-training feature D is converted through the fully connected layer FC2 to obtain the converted deep pre-training feature ; The number of output nodes of the fully connected layer FC1 and the fully connected layer FC2 are both ; The concatenation of manual features and deep pre-trained feature shapes converts the converted manual features into And the converted deep pre-trained features Reshape them into The shapes spliced ​​in the channel dimension are respectively obtained as the manual feature splicing shapes and deep pre-trained feature splicing shape ; First feature map To obtain the above , manual feature stitching shape and deep pre-trained feature splicing shape The three types of feature maps are spliced ​​according to the channel dimension to obtain the first feature map ;in ; Second feature map Acquisition of the first feature map After the attention module, the second feature map is obtained .

2. The method for ANA typing based on immunofluorescence images according to claim 1, wherein: Generation of prediction results, second feature map Through the CSRA module, the prediction output of each sample in different ANA categories is obtained , where n is the batch size during model training and t is the number of ANA categories.

3. The method for ANA typing based on immunofluorescence images according to claim 1, wherein: The ANA karyotype classification model retraining includes: training a converged multi-label classification model M1; The multi-label classification model M1 is retrained to obtain the training model M. The retraining includes label refilling: for coarse-grained labels that are missing fine-grained labels, fine-grained labels are filled; thereby obtaining a new training set. Using the new training set, the multi-label classification model M1 is fine-tuned as a pre-training model to obtain the training model M.

4. The method for ANA typing based on immunofluorescence images according to claim 1, wherein: The design of manual features in feature extraction includes: Grayscale images are obtained by using the original color images of immunofluorescence images; The calculation of the co-occurrence matrix GLCM is done by calculating the gray-level co-occurrence matrix GLCM of the gray-level image; The acquisition of manual texture features is based on the gray-level co-occurrence matrix GLCM. The manual texture features include contrast, difference, angular second moment ASM, homogeneity, energy and autocorrelation. The manual texture feature is the manual feature H. The design of deep pre-training features in feature extraction includes: To obtain the image embedding features of the immunofluorescence image, the CLIP model is used to encode the immunofluorescence image to obtain the image embedding features of the corresponding immunofluorescence image. The image embedding features are deep pre-trained features D.

5. The method for ANA typing based on immunofluorescence images according to claim 1, wherein: Multi-label classification model feedback and updates include: Construction of an independent test set: immunofluorescence images and ANA karyotype information were collected according to the method of constructing an ANA atlas library to construct an independent test set; Update of the ANA atlas library by desensitizing and structuring the immunofluorescence images and the corresponding ANA typing diagnostic conclusions; Fine-tuning training of the new model: for the updated ANA atlas, fine-tuning training of the new model is carried out in stages; some samples of the ANA atlas are included in the independent test set. After excluding these samples, the remaining samples are used as the development data set for fine-tuning training of the new model, recorded as Dev; the model currently applied online is regarded as the old model ; With the continuous expansion of the ANA atlas library, when Dev adds N samples, the old model Use the pre-trained model to fine-tune the new model until convergence to obtain a new model ; Comparison of the new model with the old model using McNemar's test With the old model When there is a significant difference between the two performances, the significance level is 0.05, and The overall accuracy is higher than , the new model is used to replace the old model currently applied online; otherwise, the old model is used.

6. A device for ANA typing based on immunofluorescence images, characterized in that: A device implemented by the method for ANA typing using immunofluorescence images as described in any one of claims 1 to 4.

Images