An ANA cell image multi-mode analysis prediction method, system, device and medium
By combining dynamic sampling and a multi-label classification network, the problems of background redundancy and feature interference in ANA cell detection are solved, improving the detection accuracy and clinical applicability of ANA cell images, especially the ability to identify small sample categories.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 笑纳科技(苏州)有限公司
- Filing Date
- 2026-05-11
- Publication Date
- 2026-07-14
Smart Images
Figure CN122176705B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of medical image processing technology, and in particular to a method, system, device and medium for multimodal analysis and prediction of ANA cell images. Background Technology
[0002] Antinuclear antibody (ANA) testing is an experimental test used to detect the presence of antinuclear antibodies in the blood, primarily for screening for autoimmune diseases. When acquiring cell images using indirect immunofluorescence (IIF), clinical practice requires the interpretation of 30 standardized fluorescent karyotypes, including homogeneous, granular, centromere, fine granular, coarse granular, nucleolar, nuclear membrane, and multi-nuclear karyotypes.
[0003] Traditional manual interpretation suffers from strong subjectivity and low consistency among doctors; high image resolution and dense cell density make manual identification labor-intensive, and the misjudgment rate is high when multiple cell types coexist.
[0004] Existing deep learning-based ANA detection methods have the following problems:
[0005] Directly inputting the acquired raw ANA image into the deep learning network results in a low proportion of cell regions and high background redundancy due to the large size of the raw ANA image, which affects the efficiency of feature learning and the accuracy of detection.
[0006] In image analysis of interphase cells, different components have obvious visual feature differences. The cell nucleus is a detectable target and is identified by detecting its boundary contour features. However, the cytoplasm and the internal details of the cell nucleus are texture distribution features. When the same network is used for feature learning and training, they are prone to mutual interference, making it difficult to take into account both texture contour modeling and internal texture details. It is difficult to achieve high-precision kernel detection and multi-label texture classification in the same feature space at the same time.
[0007] To address the problems in existing technologies, this invention provides a method, system, device, and medium for multimodal analysis and prediction of ANA cell images. Summary of the Invention
[0008] The purpose of this invention is to provide a method, system, device, and medium for multimodal analysis and prediction of ANA cell images, in order to solve the technical problem of low accuracy in existing deep learning-based ANA cell detection methods for chromosome abnormality detection, which use the same network to detect and identify multiple cell karyotypes.
[0009] The technical solution of this invention is: a multi-modal analysis and prediction method for ANA cell images, comprising:
[0010] The original ANA image is input into the target detection network to detect and locate target cells with a preset morphology in real time. When the first target cell is detected, a dynamic sampling mechanism is triggered to sample the current frame image containing the target cell and multiple subsequent frame images.
[0011] A sliding window is used to crop the region containing the target cell in the sampled frame image to obtain the image patch to be classified;
[0012] A multi-label classification network is constructed, with the cell image patch as the input and a K-dimensional probability vector output. Each dimension of the probability vector represents the probability that the image patch belongs to the corresponding cell karyotype category.
[0013] Based on the classification and detection results of the test image using the trained multi-label classification network and the detection results of interphase cells, the output image is labeled with cell location and multi-modal probability.
[0014] Preferably, the multi-label classification network includes a ResNet152 backbone network, a CBAM attention module, and a category decoupling attention module connected in sequence, and the multi-label classification network has a fully connected layer with K nodes plus a Sigmoid activation function, with the total number of cell karyotype categories preset to be the value of K.
[0015] Preferably, the sliding window cropping is performed on the region containing the detection box and its neighborhood, and frames that do not detect interphase cells are directly discarded without cropping, and the center of the cropping window is aligned with the center of the detection box;
[0016] When using a sliding window for cropping, the cropping size is X×X, with a step size of... ,for ,in, Indicates the overlap ratio of the sliding window. X represents the side length of the sliding window.
[0017] Preferably, the implementation of the category decoupling attention module includes:
[0018] Constructing prototype vectors: for each cell karyotype category Constructing prototype vectors ;
[0019] Similarity calculation and attention weight generation: Feature maps output by the CBAM module Each spatial location and prototype vector The data is stitched together, and the similarity between each spatial location and the category prototype is calculated. A 1×1 convolutional layer is then used to compress the number of channels in the stitched data to 1. Activation function generates class attention weights ;
[0020] Based on prototype guidance and feature enhancement: Transforming prototype vectors... With feature map Each spatial location is multiplied channel by channel to enhance the original features with the prototype at each spatial location, and then category attention weights are applied. Weight the enhancement results;
[0021] For each of the K cell karyotype categories, similarity calculation and attention weight generation are performed, followed by prototype-guided feature enhancement operations, and the results are then combined with the original feature map. Summing them together yields the output features. .
[0022] Preferably, the multi-label loss function is a composite loss function. : ;
[0023] Indicates the balance coefficient; This represents the multi-label binary cross-entropy loss function, used to measure the prediction probability. With real labels The differences between them Indicates the cell karyotype category index value; This represents the category-related regularized loss function.
[0024] Preferably, the CBAM attention module includes two parts: channel attention and spatial attention, as shown in the following formula:
[0025] Channel attention is represented as: ;
[0026] Spatial attention is represented as: ;
[0027] Feature enhancement is represented as: ;
[0028] in, This represents the input feature map, with dimension 1. C represents the number of feature channels, and H and W represent the height and width of the feature map, respectively.
[0029] This represents the global average pooling operation, which calculates the mean of the input feature map over a spatial dimension H×W, and outputs a feature map with a dimension of H×W. ;
[0030] This represents a multilayer perceptron network, consisting of two fully connected layers, used to learn nonlinear mapping relationships between channels;
[0031] This represents the channel attention weight graph, with dimensions of [missing information]. , used to characterize the importance weight of each channel;
[0032] Represents a spatial attention weight graph, with dimensions of . , used to characterize the importance weights of different spatial locations;
[0033] Represents the sigmoid function;
[0034] This indicates element-wise multiplication, used for weighted feature addition;
[0035] This represents the output enhanced feature map.
[0036] An ANA cell image multimodal analysis and prediction system, used to implement the aforementioned ANA cell image multimodal analysis and prediction method, includes:
[0037] The image acquisition module is used to acquire raw cell images;
[0038] The dynamic sampling module has a built-in target detection network for detecting and locating images, and for cropping and sampling target cells;
[0039] A multi-label classification network, comprising a ResNet backbone network, a CBAM attention module, and a category decoupling attention module connected in sequence, takes a dynamically sampled set of image patches as input, and outputs the probability that each test image belongs to one or more preset categories after pre-training a convolutional neural network.
[0040] The training and optimization module is used to optimize the training of the multi-label classification network during the training process;
[0041] The evaluation module is used to evaluate the performance of multi-label classification networks;
[0042] The classification output module jointly outputs the detection results of the multi-label classification network and the target cell localization detection results.
[0043] An electronic device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the aforementioned ANA cell image multimodal analysis and prediction method.
[0044] A computer-readable storage medium storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by a processor to implement the aforementioned ANA cell image multimodal analysis and prediction method.
[0045] Compared with the prior art, the advantages of the present invention are:
[0046] This invention is based on a dynamic sampling mechanism triggered by interphase cell detection. By performing sliding window cropping sampling on the effective frame image, the proportion of cell regions in the image is increased, computational redundancy is reduced, and background interference is decreased.
[0047] This invention embeds a CBAM attention module and a category decoupling attention module into the backbone of a constructed multi-label classification network. This allows for dynamic interaction between learnable category prototype vectors and image features, focusing on the core discriminative features of each category. In particular, for rare karyotypes with few samples, the prototype vectors provide category priors, guiding the network to focus on neglected feature points and enhancing the ability to identify categories in small samples. By constructing a composite loss function to optimize and train the multi-label classification network, the accuracy and clinical applicability of multimodal analysis of ANA cell images are improved.
[0048] This invention evaluates the performance of multi-label classification networks using Hamming check codes, F1 scores, and the Clinical Consistency Index (CCS), making the network's optimization objectives and evaluation criteria more aligned with the working habits of clinicians and improving the model's practicality in assisted diagnosis. Attached Figure Description
[0049] The present invention will be further described below with reference to the accompanying drawings and embodiments:
[0050] Figure 1 This is an overall flowchart of the ANA cell image multimodal analysis and prediction method described in this invention;
[0051] Figure 2 This is a diagram of the multi-label classification network structure described in this invention;
[0052] Figure 3 This is a schematic diagram of the dynamic sampling and sliding window clipping described in this invention;
[0053] Figure 4 This is a structural diagram of the category decoupling attention module described in this invention;
[0054] Figure 5 This is a schematic diagram comparing the classification rate of small sample categories with the embodiments of the present invention and the prior art. Detailed Implementation
[0055] The present invention will be further described in detail below with reference to specific embodiments:
[0056] like Figure 1 As shown, a multimodal analysis and prediction method for ANA cell images includes the following steps:
[0057] Step 1: Detection and identification of target cells.
[0058] Real-time detection and localization of target cells with a preset specific morphology are performed. In this embodiment of the invention, the target cells with the specific morphology are interphase cells.
[0059] The original ANA image is input into the pre-trained YOLOv8 target detection network to locate and identify interphase cells. The detection output is as follows:
[0060] ;
[0061] in, This represents the position information of the a-th detection box (including center coordinates, width, and height). Indicates the predicted category label; Indicates the confidence score;
[0062] When YOLOv8 successfully detects interphase cells in the current frame T0 (the detection confidence exceeds a preset threshold, such as 0.8), the system records the frame image as a valid frame image and continuously acquires the subsequent N valid frame images. Frame images in which no interphase cells are detected are directly discarded and not analyzed further.
[0063] Step 2: Sliding window cropping of the original ANA image.
[0064] When using a sliding window to crop the original ANA image, for N consecutively acquired images, the system only performs sliding window cropping on the region containing the detection box, and the center of the cropping window is aligned with the center of the detection box.
[0065] The cropping size (unit: pixels) is set as follows: ;
[0066] Set the original image size to The step size is: .
[0067] in: Indicates the overlap ratio of the sliding window. , X represents the side length of the sliding window (in pixels).
[0068] For example, in a specific implementation, X is set to 512 and the overlap rate r is set to 20%.
[0069] The set of image patches after being cropped by the sliding window is represented as follows: : .
[0070] By setting overlapping sliding windows for sequential cropping, the cell region is ensured to be completely covered while preserving some contextual information. See the appendix for the dynamic sampling process. Figure 3 As shown, limiting the window overlap rate to no more than 30% focuses computational resources on effective cell regions, reducing system computational redundancy.
[0071] Step 3: Construction of multi-label classification network and model training.
[0072] A multi-label classification network is constructed by combining the acquired image patches with the P input. See the attached diagram for a schematic of the multi-label classification network structure. Figure 2 As shown, it includes a backbone network, a CBAM attention module, a category decoupling attention module, and a multi-label classification head.
[0073] In this embodiment, the backbone network uses ResNet152 to extract initial feature maps from the input image patch set P. .
[0074] A CBAM attention module, which includes channel attention and spatial attention, is introduced after the ResNet152 residual block.
[0075] Channel attention is represented as: ;
[0076] Spatial attention is represented as: ;
[0077] Feature enhancement is represented as: .
[0078] in:
[0079] This represents the input feature map, with dimension 1. Where C represents the number of feature channels, and H and W represent the height and width of the feature map, respectively.
[0080] This represents the global average pooling operation, which calculates the mean of the input feature map over a spatial dimension H×W, and outputs a feature map with a dimension of H×W. .
[0081] This represents a multilayer perceptron network, consisting of two fully connected layers, used to learn nonlinear mapping relationships between channels.
[0082] This represents the channel attention weight graph, with dimensions of [missing information]. , used to characterize the importance weight of each channel.
[0083] Represents a spatial attention weight graph, with dimensions of . , used to characterize the importance weights of different spatial locations.
[0084] This represents element-wise multiplication (Hadamard product), used for weighting features.
[0085] The output enhanced feature map is used for subsequent multi-label classification prediction.
[0086] Channel attention computation performs global average pooling and global max pooling on the input feature map in the spatial dimension to obtain two channel descriptors. Then, the two channel descriptors are non-linearly transformed through a shared fully connected layer to generate channel attention weights. Finally, the generated channel attention weights are multiplied with the original feature map to obtain a feature map with channel attention applied.
[0087] Spatial attention computation performs global average pooling and global max pooling on the feature map with channel attention applied, resulting in two spatial descriptors. The two spatial descriptors are then concatenated and nonlinearly transformed through a convolutional layer to generate spatial attention weights. Finally, the generated spatial attention weights are multiplied by the feature map with channel attention applied to obtain the final adaptively refined feature map.
[0088] A significant co-occurrence relationship exists among AC0-AC29, and ordinary classification networks are easily dominated by high-frequency categories, resulting in low category recognition rates for small samples. To address the problems of category co-occurrence and difficulty in small sample recognition in multi-label classification, this invention introduces a category decoupling attention module after the CBAM attention module. The structure is shown in the appendix. Figure 4 As shown, the category decoupling attention module enhances feature decoupling by introducing category prototype vectors.
[0089] First, construct prototype vectors for each cell karyotype category. This is obtained by average pooling the features of samples of this category in the training set;
[0090] Feature maps output by the CBAM attention module Calculate the similarity between each spatial location and the category prototype to generate category attention weights. ;
[0091] in To perform a 1×1 convolution, reduce the number of channels to 1.
[0092] The final feature is represented as ;
[0093] in This represents channel-wise multiplication, used to transform the prototype vector... Broadcast to the spatial dimension, K represents the total number of cell karyotype categories.
[0094] By decoupling the attention module by category, explicit class prior guidance is introduced to enhance the feature representation of few-sample categories, while alleviating the category coupling problem. This enables the constructed multi-label classification network to strengthen features of few-sample categories, suppress background redundancy, and improve multi-label discrimination ability.
[0095] The feature map output by the category decoupling attention module The input is a multi-class classification head, which includes a global average pooling layer and a fully connected layer with K nodes, and uses the sigmoid function as the activation function. The output is a K-dimensional probability vector. Each dimension Indicates that the image belongs to the first... The probability of a cell karyotype, K, is equal to the number of cell karyotype categories. The output is represented as:
[0096] ;
[0097] in: Indicates the first The probability of cell-like karyotype; This represents the sigmoid function.
[0098] Step 4: Model optimization training.
[0099] In the training process of the model, this embodiment uses a composite loss function. : By introducing category co-occurrence priors, the consistency and robustness of multi-label classification are improved.
[0100] This represents the multi-label binary cross-entropy loss function, used to measure the difference between the predicted probability and the true label. This represents the balance coefficient.
[0101] The multi-label binary cross-entropy loss function (Multi-label BCE) is expressed as follows:
[0102]
[0103] Where K represents the number of cell karyotype types, This represents the true label. In the loss function, This represents the cell karyotype category index value.
[0104] This represents the category relevance regularization loss function, which constructs a co-occurrence matrix based on the labels of the training set. Define category relevance penalty: ; Indicates cell karyotype category and The frequency of simultaneous occurrence.
[0105] For frequently co-occurring categories, output similar probabilities, while for infrequently co-occurring categories, allow or even encourage probability differences, reducing prediction conflicts and further promoting feature decoupling and balancing coefficients. In this embodiment, it is set to 0.1.
[0106] Step 5: Construct multi-label model evaluation metrics and evaluate the model.
[0107] The performance of multi-label classification networks is evaluated using multi-label evaluation metrics, including Hamming check code and F1 score.
[0108] Hamming Accuracy is used to evaluate the model's average predictive ability on each label; F1 score is used to measure the balance between precision and recall in multi-label detection.
[0109] In addition to Hamming checksum and F1 score, this embodiment also sets up a clinical conformity index (CCS) to evaluate the model.
[0110] For each test image, if the top Z categories with the highest predicted probabilities contain the doctor's actual diagnosis category, then it is considered consistent. Z represents the number of predicted categories, and Z is a positive integer.
[0111] The Clinical Conformity Index (CCS) represents the proportion of samples that reach consensus among all tested samples.
[0112] A smaller Z-score indicates more accurate predictions, and a higher Clinical Consensus (CCS) value also indicates more accurate predictions. This aligns with real-world clinical applications and effectively reflects the model's value in assisting diagnosis.
[0113] See attached document Figure 5 As shown, the method of this embodiment is compared with existing technologies (e.g., Chinese Patent Application No. CN202311139120.5, entitled "Method and Apparatus for ANA Typing of Immunofluorescence Images," which discloses an end-to-end ANA typing model that can adapt to complex real clinical karyotypes by constructing an end-to-end ANA typing model through multi-source feature fusion and multi-label learning strategies, and continuously optimizing the model through a data-driven feedback mechanism) in terms of recognition accuracy for small sample categories (rare karyotypes). The results show that, thanks to the category decoupling attention module, the present invention significantly improves the recognition accuracy for small sample karyotypes.
[0114] Step 6: Multi-model collaborative fusion output.
[0115] The final classification result is obtained by fusing the following two parts:
[0116] The YOLOv8 object detection network outputs interphase cell detection results, including image, location, cell karyotype category, and multi-label classification probability output by the multi-label classification network.
[0117] The system marks the location of interphase cells on the image and displays the multi-label classification probability of the region for doctors' reference.
[0118] This invention addresses the issues of background redundancy and interference through a dynamic sampling mechanism. The constructed multi-label classification network embeds a CBAM attention module and a category decoupling attention module into the ResNet152 backbone network, enabling dynamic interaction between learnable category prototype vectors and image features, thereby enhancing the ability to identify categories in small samples. By constructing a composite loss function to optimize and train the multi-label classification network, the accuracy and clinical applicability of multi-modal analysis of ANA cell images are improved.
[0119] This invention also provides a multimodal analysis and prediction system for ANA cell images, used to implement the above-mentioned multimodal analysis and prediction method for ANA cell images. The system includes:
[0120] The image acquisition module is used to acquire raw cell images;
[0121] The dynamic sampling module has a built-in target detection network for detecting and locating images, and for cropping and sampling target cells;
[0122] A multi-label classification network takes a dynamically sampled set of image patches as input and outputs the probability that each test image belongs to one or more preset categories after being pre-trained by a convolutional neural network.
[0123] The training and optimization module is used to optimize the training of the multi-label classification network during the training process;
[0124] The evaluation module is used to evaluate the performance of multi-label classification networks;
[0125] The classification output module jointly outputs the detection results of the multi-label classification network and the target cell localization detection results.
[0126] This invention provides a method and system for multimodal analysis and prediction of ANA cell images. It reduces background interference through dynamic sampling, enhances small sample class recognition ability through class decoupling attention, and optimizes multi-label output through class correlation penalty, thereby improving the accuracy and clinical applicability of multimodal analysis of ANA cell images.
[0127] Based on the above method and system, the present invention provides a specific embodiment to illustrate the implementation process of the method.
[0128] In this embodiment, the original ANA image resolution is 2048×2048. A YOLOv8 model (pre-trained weights) is used for interphase cell detection, with a detection threshold of 0.5. When the first interphase cell is detected, the current frame is recorded as T0, and the next four frames (T1-T4) are continuously acquired. For each valid frame, a 512×512 image patch is extracted centered on the detection box, with a stride of 358 pixels (corresponding to a 30% overlap). If the image patch exceeds the original image boundary, it is filled with zeros. Approximately 200 image patches are obtained in total.
[0129] In multi-label classification networks, the backbone network uses ResNet152, and the output feature map has C=30 channels and a spatial size of 7×7. Prototype vectors are constructed for each category. The feature map is obtained by averaging the features of all samples of this category in the training set across the spatial dimension, which is 30. The feature map output by the CBAM attention module is then used. and After concatenation, 1×1 convolution is used to generate category attention weights. Then, the final features are obtained by weighted summation through channel-wise multiplication. This module is placed after the last convolutional layer of ResNet152 and before the classification head.
[0130] The training set contains 5000 labeled images, each with 0-5 labels. The label co-occurrence matrix M is obtained from the training set statistics and normalized to [0,1]. The balance coefficients in the composite loss function used for model training are... The value is set to 0.1, the Adam optimizer is used, the initial learning rate is 0.001, and the training is conducted for 50 epochs.
[0131] The Hamming checksum of the validation set reached 0.92, the F1 score reached 0.88, and the clinical conformity index (CCS) reached 0.95.
[0132] During the testing phase, for each image, the model outputs 30 probability values. The top three categories are used as the unit index for judging accuracy. If the doctor's diagnosis is included in these three categories, it is considered consistent. After statistical analysis of 1000 test images, the CCS was 0.96, indicating that the model has high reliability in clinical auxiliary diagnosis.
[0133] This invention also provides an electronic device, which includes a processor and a memory; the memory stores one or more instructions, which are adapted for the processor to load and execute, to implement the ANA cell image multimodal analysis and prediction method as described in the above method embodiments.
[0134] Memory is used to store software programs and modules. The processor executes these stored software programs and modules to perform various functional applications and data processing. Memory mainly includes a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functions, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory may include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory may also include a memory controller to provide the processor with access to the memory.
[0135] The internal structure of the electronic device provided in the embodiments of the present invention may include, but is not limited to, a processor, a memory, and a communication interface. The processor, memory, and communication interface in the electronic device may be connected by a bus or other means. In the embodiments of this specification, a connection via a bus is taken as an example.
[0136] The processor (or CPU, Central Processing Unit) is the computing and control core of the electronic device. A communication interface is used for communication between the memory and the processor. The memory stores programs and data. It is understood that the memory here can be a high-speed RAM storage device, or a non-volatile memory device, such as at least one disk storage device; optionally, it can also be at least one storage device located remotely from the aforementioned processor. The memory provides storage space, which stores the operating system of the electronic device, and may include, but is not limited to, Windows (an operating system), Linux (an operating system), etc. This invention does not limit this; and the storage space also stores computer programs (including program code) suitable for being loaded and executed by the processor. In the embodiments of this specification, the processor loads and executes the computer program stored in the memory to implement the ANA cell image multi-modal analysis and prediction method provided in the above method embodiments.
[0137] This invention also provides a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction, at least one program, code set, or instruction set related to implementing the ANA cell image multimodal analysis and prediction method in the method embodiments. The at least one instruction, at least one program, code set, or instruction set can be loaded and executed by the processor of the electronic device to implement the ANA cell image multimodal analysis and prediction method provided in the above method embodiments.
[0138] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0139] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments, while other embodiments fall within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than those shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0140] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.
[0141] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.
[0142] The above description is merely a preferred embodiment of the present invention and should not be construed as limiting the scope of the invention. Therefore, any equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.
Claims
1. A method for multimodal analysis and prediction of ANA cell images, characterized in that, include: The original ANA image is input into the target detection network to perform real-time detection and localization of target cells with a preset morphology. When the first target cell is detected, a dynamic sampling mechanism is triggered to sample the current frame image containing the target cell and multiple subsequent frame images. A sliding window is used to crop the region containing the target cell in the sampled frame image to obtain the image patch to be classified; The sliding window cropping is performed on the region containing the detection box and its neighborhood. Frames in which no interphase cells are detected are directly discarded without cropping. The center of the cropping window is aligned with the center of the detection box. When using a sliding window for cropping, the cropping size is X×X, with a step size of... ,for ,in, Indicates the overlap ratio of the sliding window. X represents the side length of the sliding window; A multi-label classification network is constructed, with the cell image patch as the input and a K-dimensional probability vector output. Each dimension of the probability vector represents the probability that the image patch belongs to the corresponding cell karyotype category. The multi-label classification network includes a ResNet152 backbone network, a CBAM attention module, and a category decoupling attention module connected in sequence. The multi-label classification network has a fully connected layer with K nodes and a Sigmoid activation function. The total number of cell karyotype categories is preset to be equal to the value of K. Based on the classification and detection results of the test image using the trained multi-label classification network and the detection results of interphase cells, the output image is labeled with cell location and multi-modal probability.
2. The method for multimodal analysis and prediction of ANA cell images according to claim 1, characterized in that, The implementation of the category decoupling attention module includes: Constructing prototype vectors: for each cell karyotype category Constructing prototype vectors ; Similarity calculation and attention weight generation: Feature maps output by the CBAM module Each spatial location and prototype vector The data is stitched together, and the similarity between each spatial location and the category prototype is calculated. A 1×1 convolutional layer is then used to compress the number of channels in the stitched data to 1. Activation function generates class attention weights ; Based on prototype guidance and feature enhancement: Transforming prototype vectors... With feature map Each spatial location is multiplied channel by channel to enhance the original features with the prototype at each spatial location, and then category attention weights are applied. Weight the enhancement results; For each of the K cell karyotype categories, similarity calculation and attention weight generation are performed, followed by prototype-guided feature enhancement operations, and the results are then combined with the original feature map. Summing them together yields the output features. .
3. The method for multimodal analysis and prediction of ANA cell images according to claim 1, characterized in that, A multi-label classification network is trained and optimized using a multi-label loss function, which employs a composite loss function. : ; Indicates the balance coefficient; This represents the multi-label binary cross-entropy loss function, used to measure the prediction probability. With real labels The differences between them Indicates the cell karyotype category index value; This represents the category-related regularized loss function.
4. The method for multimodal analysis and prediction of ANA cell images according to claim 1, characterized in that, The CBAM attention module comprises two parts: channel attention and spatial attention. The specific formula is as follows: Channel attention is represented as: ; Spatial attention is represented as: ; Feature enhancement is represented as: ; in, This represents the input feature map, with dimension 1. C represents the number of feature channels, and H and W represent the height and width of the feature map, respectively. This represents the global average pooling operation, which calculates the mean of the input feature map over a spatial dimension H×W, and outputs a feature map with a dimension of H×W. ; This represents a multilayer perceptron network, consisting of two fully connected layers, used to learn nonlinear mapping relationships between channels; This represents the channel attention weight graph, with dimensions of [missing information]. , used to characterize the importance weight of each channel; Represents a spatial attention weight graph, with dimensions of . , used to characterize the importance weights of different spatial locations; Represents the sigmoid function; This indicates element-wise multiplication, used for weighted feature addition; This represents the output enhanced feature map.
5. An ANA cell image multimodal analysis and prediction system, used to implement the ANA cell image multimodal analysis and prediction method according to any one of claims 1-4, characterized in that, include: The image acquisition module is used to acquire raw cell images; The dynamic sampling module has a built-in target detection network for detecting and locating images, and for cropping and sampling target cells; A multi-label classification network, comprising a ResNet backbone network, a CBAM attention module, and a category decoupling attention module connected in sequence, takes a dynamically sampled set of image patches as input, and outputs the probability that each test image belongs to one or more preset categories after pre-training a convolutional neural network. The training and optimization module is used to optimize the training of the multi-label classification network during the training process; The evaluation module is used to evaluate the performance of multi-label classification networks; The classification output module jointly outputs the detection results of the multi-label classification network and the target cell localization detection results.
6. An electronic device, characterized in that, The device includes a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or at least one program being loaded and executed by the processor to implement the ANA cell image multimodal analysis and prediction method as described in any one of claims 1-4.
7. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the ANA cell image multimodal analysis and prediction method as described in any one of claims 1-4.