Cell detection map generation method based on deep learning, electronic equipment and program product

Abstract

The invention provides a cell detection map generation method based on deep learning, electronic equipment and a program product. The method comprises the following steps: acquiring a cell image, wherein the cell image comprises an image area corresponding to at least one cell; the cell image is input into a pre-established multi-modal heterogeneous network model based on deep learning, fusion features output by the multi-modal heterogeneous network model are obtained, and the multi-modal heterogeneous network model comprises a first branch unit, a second branch unit and a feature cross attention unit; updating the fusion features based on a pre-established dynamic transfer learning module to obtain updated fusion features; based on a pre-established cell feature decoupling module, decoupling the cell image to obtain a decoupling feature; and generating a cell map based on the updated fusion features and decoupling features. Therefore, the marking dependence is reduced, the artifact interference in the cell map is reduced, and the accuracy and interpretability of the cell map are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cell image processing, and in particular to a method for generating cell detection maps based on deep learning, an electronic device, and a program product. Background Art

[0002] Cell atlases provide important support for biological research in many aspects. For example, through cell atlases, we can understand the functions and roles of different cell types in physiological processes. In the existing cell atlas generation process, single-modality convolutional networks (such as ResNet and DenseNet) are usually used to process cell images, which affects the accuracy of the generated cell atlas. For example, single-modality convolutional networks cannot fuse morphological and staining features. Without the fusion of morphological and staining features, it is easy to cause feature confusion due to cell overlap and artifact interference, resulting in inaccurate classification of the cell atlas. Summary of the Invention

[0003] In view of this, the purpose of the embodiments of the present application is to provide a cell detection atlas generation method, electronic device and program product based on deep learning, which can improve the accuracy of generated cell atlases and improve the problem of inaccurate content of generated cell atlases due to the inability to integrate morphological and staining features.

[0004] To achieve the above technical objectives, the technical solutions adopted in this application are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for generating a cell detection atlas based on deep learning, the method comprising:

[0006] Acquiring a cell image, wherein the cell image includes at least one image region corresponding to a cell;

[0007] Inputting the cell image into a pre-established multimodal heterogeneous network model based on deep learning, obtaining a fusion feature output by the multimodal heterogeneous network model, wherein the multimodal heterogeneous network model includes a first branch unit, a second branch unit, and a feature cross-attention unit, wherein the first branch unit is a ResNet-Transformer network unit, configured to extract morphological features from the cell image; the second branch unit is a dynamic convolutional pyramid network unit, configured to extract staining features from the cell image; and the feature cross-attention unit is configured to fuse the morphological features and the staining features to obtain the fusion feature;

[0008] Based on a pre-established dynamic transfer learning module, the fusion feature is updated to obtain an updated fusion feature;

[0009] Decoupling the cell image based on a pre-established cell feature decoupling module to obtain decoupling features;

[0010] A cell atlas is generated based on the updated fusion features and the decoupled features.

[0011] In conjunction with the first aspect, in some optional embodiments, the cell image is input into a pre-established multimodal heterogeneous network model based on deep learning, and the fusion features output by the multimodal heterogeneous network model are obtained, including:

[0012] Inputting the cell image into the first branch unit and the second branch unit in the multimodal heterogeneous network model;

[0013] Performing feature extraction on the cell image using the ResNet subunit in the first branch unit to obtain primary features;

[0014] The Transformer subunit in the first branch unit is used to reorganize and integrate the primary features using multiple attention heads to obtain the morphological features;

[0015] The dynamic convolutional pyramid network unit convolves the cell image through the second branch unit to obtain the staining feature;

[0016] The morphological feature and the staining feature are fused through the feature cross attention unit to obtain the fused feature.

[0017] In combination with the first aspect, in some optional implementations, multiple attention heads are used to perform feature recombination and integration on the primary features to obtain the morphological features, including:

[0018] Using each attention head, based on the corresponding attention weight, the primary features are reorganized to obtain the reorganized primary features. The attention weight is expressed as:

[0019]

[0020] Attention i (Q i , K i , V i ) represents the attention weight of the i-th attention head; Softmax() represents the activation function; Q i , K i and V i denote the query vector, key vector, and value vector of the i-th attention head respectively; d k1 represents the key vector K i The dimension of ; T represents the transpose of the vector; represents Q in the i-th attention head i , Ki and V i The projection matrix; F low representing said primary feature;

[0021] The reorganized primary features are integrated to obtain the morphological features representing the global semantic expression.

[0022] In combination with the first aspect, in some optional embodiments, the morphological feature and the staining feature are fused by the feature cross attention unit to obtain the fused feature, including:

[0023] The morphological features and the staining features are fused through the feature cross attention unit based on the layer normalization function to obtain the fused features. The calculation formula of the fusion is expressed as:

[0024] F fusion =LayerNorm(F morph +FFN(Attention(Q,K,V)))

[0025]

[0026] Q=W q1 F morph

[0027] K=W k1 F stain

[0028] V=W v1 F stain

[0029] Where, F fusion Represents the fusion feature; LayerNorm() represents the layer normalization function; Attention(Q,K,V) represents the attention weight of feature intersection; Q, K, V represent the query vector, key vector and value vector of feature intersection respectively; W q1 、W k1 、W v1 Represents the projection matrices of Q, K, and V respectively; F morph Represents the morphological features; FFN() represents the feedforward neural network function; F stain Indicates the staining characteristics; d k2 Represents the dimension of the key vector K.

[0030] In conjunction with the first aspect, in some optional implementations, the fusion feature is updated based on a pre-established dynamic transfer learning module to obtain an updated fusion feature, including:

[0031] Calculating, by the dynamic transfer learning module, a distribution difference between a source domain feature distribution and a target domain feature distribution, wherein the source domain feature distribution is obtained based on pre-labeled cell images and the target domain feature distribution is obtained based on unlabeled cell images;

[0032] Based on the distribution difference, the Attention(Q, K, V) in the fusion calculation formula, the bias term in the FFN() function, and the scaling parameter in LayerNorm() are adjusted to obtain updated fusion features.

[0033] In conjunction with the first aspect, in some optional embodiments, decoupling the cell image based on a pre-established cell feature decoupling module to obtain decoupling features includes:

[0034] The cell feature decoupling module decomposes the cell image according to the adversarial feature separation algorithm to obtain the decoupled feature, which is expressed as:

[0035] I=α·C nucleus +β·C cytoplasm +γ·C artifact

[0036] Where, I represents the cell image, α, β, and γ are the decomposition coefficients corresponding to the reconstruction of the cell nucleus, cytoplasm, and artifacts, respectively; C nucleus represents the cell nucleus feature after decoupling of the cell image; C cytoplasm represents the cytoplasmic features after decoupling of the cell image; C artifact Represents the artifact features after decoupling of the cell image; the decoupling features include the cell nucleus feature C nucleus 、Cytoplasmic characteristics C cytoplasm and artifact features C artifact .

[0037] In conjunction with the first aspect, in some optional embodiments, generating a cell atlas based on the updated fusion feature and the decoupling feature includes:

[0038] Performing feature splicing on the updated fusion feature, the cell nucleus feature, and the cytoplasm feature to obtain a spliced ​​feature as an intermediate feature;

[0039] Based on a pre-established U-Net module, the intermediate features are mapped into a cell atlas of a specified resolution;

[0040] The artifact feature is used as an attention gating signal to suppress the noise area in the cell atlas, thereby obtaining a cell atlas representing the suppressed artifact.

[0041] In conjunction with the first aspect, in some optional implementations, the method further includes:

[0042] Inputting the intermediate features into a pre-established classification module to obtain a cell classification result;

[0043] Generate a visual heat map based on the gradient weighted class activation mapping algorithm and the updated fusion features;

[0044] The cell image, the cell atlas after suppressing artifacts, the classification results, and the visualized heat map are displayed in a multi-view fusion manner.

[0045] In a second aspect, an embodiment of the present application further provides an electronic device, comprising a processor and a memory coupled to each other, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the electronic device executes the above-mentioned method.

[0046] In a third aspect, an embodiment of the present application further provides a program product, comprising a computer program, which implements the above method when executed by a processor.

[0047] The invention adopting the above technical solution has the following advantages:

[0048] In the technical solution provided in this application, the first and second branch units in the multimodal heterogeneous network model are used to extract features from cell images to obtain morphological features and staining features, and the morphological features and staining features are fused through the feature cross-attention unit to obtain fused features; the fused features are updated using the dynamic transfer learning module; the cell feature decoupling module is used to decouple the cell image to obtain decoupled features; and the updated fused features and decoupled features are used to generate a cell atlas. In this way, the multimodal heterogeneous network model can realize cross-modal feature interaction, so that the morphological features of the cell (such as size and shape) and the staining features (such as hematoxylin-eosin-DAB intensity) can be learned in an associative manner, avoiding the traditional series / additive fusion that leads to information loss. The dynamic transfer learning module is conducive to adapting to the staining differences between different laboratories, ensuring the domain invariance of feature expression, reducing annotation dependence, and improving cross-domain robustness; the cell feature decoupling module is conducive to providing interpretable substructure (such as cell nucleus, cytoplasm, artifact) features for atlas generation, which is conducive to reducing artifact interference in cell atlases and improving the accuracy and interpretability of cell atlases; the synergistic effect of fusion features and decoupling features is conducive to the realization of refined reconstruction and classification of cell atlases. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The present application may be further illustrated by the non-limiting embodiments provided in the accompanying drawings. It should be understood that the following drawings illustrate only certain embodiments of the present application and are therefore not to be construed as limiting the scope of the present application. It is understood that a person skilled in the art can derive other relevant drawings from these drawings without inventive effort.

[0050] Figure 1 A schematic flow chart of the deep learning-based cell detection atlas generation method provided in an embodiment of the present application.

[0051] Figure 2 The structure and data flow of the graph generation system provided in the embodiment of this application.

[0052] Icons: 200-atlas generation system; 210-multimodal heterogeneous network model; 211-first branch unit; 212-second branch unit; 213-feature cross-attention unit; 220-dynamic transfer learning module; 230-cell feature decoupling module; 240-atlas generation module. DETAILED DESCRIPTION

[0053] The present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that similar or identical parts in the drawings or descriptions are numbered the same. Implementations not shown or described in the drawings are known to those of ordinary skill in the art. In the description of this application, the terms "first," "second," etc. are used solely to distinguish descriptions and are not to be construed as indicating or implying relative importance.

[0054] The present application provides an electronic device that may include a processor and memory. The memory stores a computer program that, when executed by the processor, enables the electronic device to perform the corresponding steps of the following cell detection profile generation method. The electronic device may be, but is not limited to, a personal computer, a server, or other device.

[0055] Please refer to Figure 1 and Figure 2 , the present application also provides a cell detection atlas generation method based on deep learning, which can be applied to the above-mentioned electronic device, and each step of the method is executed or implemented by the electronic device. Among them, the electronic device is deployed with a map generation system 200, and the map generation system 200 includes at least one software function module that can be stored in the electronic device in the form of software or firmware (Firmware) or solidified in the operating system (Operating System, OS). The processor is used to execute the executable modules stored in the memory, such as the software function modules and computer programs included in the map generation system 200.

[0056] In this embodiment, the atlas generation system 200 may include a multimodal heterogeneous network model 210, a dynamic transfer learning module 220, a cell feature decoupling module 230, and an atlas generation module 240. The cell detection atlas generation method may include the following steps:

[0057] Step 110, obtaining a cell image, wherein the cell image includes at least one image region corresponding to a cell;

[0058] Step 120: Input the cell image into a pre-established multimodal heterogeneous network model 210 based on deep learning to obtain a fusion feature output by the multimodal heterogeneous network model 210, wherein the multimodal heterogeneous network model 210 includes a first branch unit 211, a second branch unit 212, and a feature cross-attention unit 213. The first branch unit 211 is a ResNet-Transformer network unit for extracting morphological features from the cell image; the second branch unit 212 is a dynamic convolutional pyramid network unit for extracting staining features from the cell image; and the feature cross-attention unit 213 is used to fuse the morphological features and the staining features to obtain the fusion feature.

[0059] Step 130: updating the fusion feature based on the pre-established dynamic transfer learning module 220 to obtain an updated fusion feature;

[0060] Step 140 , decoupling the cell image based on the pre-established cell feature decoupling module 230 to obtain decoupling features;

[0061] Step 150: Generate a cell atlas based on the updated fusion features and the decoupled features.

[0062] In the above-mentioned embodiment, the first branch unit 211 and the second branch unit 212 in the multimodal heterogeneous network model 210 are used to extract features from the cell image to obtain morphological features and staining features, and the morphological features and staining features are fused through the feature cross-attention unit 213 to obtain fused features. This can achieve cross-modal feature interaction, so that the morphological features (such as size and shape) of the cell and the staining features (such as Hematoxylin-Eosin-DAB intensity) can be associated with each other. Learning, avoiding traditional series / additive fusion, which leads to information loss. The dynamic transfer learning module 220 is used to update the fused features, which is conducive to adapting to the staining differences of different laboratories, ensuring the domain invariance of feature expression, and improving cross-domain robustness. The cell feature decoupling module 230 is used to decouple the cell image to obtain decoupled features, which is conducive to providing interpretable substructure (such as cell nucleus, cytoplasm, artifact) features for atlas generation, which is conducive to reducing artifact interference in the cell atlas and improving the accuracy and interpretability of the cell atlas. The updated fused features and decoupled features are used to generate a cell atlas. In this way, the synergistic effect of fusion features and decoupling features is conducive to the refined reconstruction and classification of cell atlases and improves the accuracy of cell atlases.

[0063] The following is a detailed description of the steps in the cell detection map generation method:

[0064] In step 110, the cell image is the image to be detected for cell mapping. It can be a pre-prepared image or an image acquired in real time, and the method for acquiring the cell image is not specifically limited here. When generating a cell map, the electronic device can obtain the cell image from a server storing cell images, a microscope acquiring electronic images in real time, or other equipment, and the method for acquiring the cell image is not specifically limited here.

[0065] In this embodiment, the cell image is an RGB color image, which is usually an image obtained by photographing a cell sample under a microscope. In one cell image, multiple cells can usually be seen, that is, the cell image usually includes multiple cell regions.

[0066] The input multimodal heterogeneous network model 210 is a preprocessed image, and the preprocessing method can be flexibly set according to the actual situation. For example, the size of the cell image is cropped to a set size, which can be flexibly set according to the actual situation and is not specifically limited here.

[0067] In step 120, the multimodal heterogeneous network model 210 may include a first branch unit 211, a second branch unit 212 and a feature cross-attention unit 213, wherein the first branch unit 211 is a ResNet-Transformer network unit, which is used to extract morphological features from the cell image; the second branch unit 212 is a dynamic convolutional pyramid network unit, which is used to extract staining features from the cell image; the feature cross-attention unit 213 is used to fuse the morphological features and the staining features to obtain the fused features.

[0068] In step 120, the cell image is input into a pre-established multimodal heterogeneous network model 210 based on deep learning, and the fusion features output by the multimodal heterogeneous network model 210 are obtained, which may include:

[0069] Step 121 , inputting the cell image into the first branch unit 211 and the second branch unit 212 in the multimodal heterogeneous network model 210 ;

[0070] Step 122, performing feature extraction on the cell image through the ResNet subunit in the first branch unit 211 to obtain primary features;

[0071] Step 123: Using the Transformer subunit in the first branch unit 211, multiple attention heads are used to reorganize and integrate the primary features to obtain the morphological features.

[0072] Step 124: Using the second branch unit 212, the dynamic convolutional pyramid network unit convolves the cell image to obtain the staining feature;

[0073] Step 125: The morphological feature and the staining feature are fused through the feature cross attention unit 213 to obtain the fused feature.

[0074] In the first branch unit 211, the ResNet-Transformer network unit includes a ResNet subunit and a Transformer subunit. The ResNet subunit can be a ResNet-50 backbone network, which can be used to extract multi-scale local morphological features of cell images (such as local detail information such as cell nuclear morphology and edge curvature) layer by layer through residual blocks. By stacking residual blocks, cell morphological features are gradually abstracted to avoid gradient disappearance.

[0075] The final feature map extracted by the ResNet subunit serves as the low-level morphological features, or primary features. The method used by the ResNet subunit to extract primary features is conventional and will not be described here.

[0076] The Transformer subunit can be a Transformer encoder, which can combine the multi-head attention mechanism with the primary features, model global context dependencies (such as the spatial distribution relationship between the nucleus and cytoplasm), and output global morphological features.

[0077] In step 123, the Transformer subunit in the first branch unit 211 uses multiple attention heads to reorganize and integrate the primary features to obtain the morphological features, which may include

[0078] Using each attention head, based on the corresponding attention weight, the primary features are reorganized to obtain the reorganized primary features. The attention weight is expressed as:

[0079]

[0080] Attention i (Q i , K i , V i ) represents the attention weight of the i-th attention head; Softmax() represents the activation function; Q i , K i and V i denote the query vector, key vector, and value vector of the i-th attention head respectively; d k1 represents the key vector K i The dimension of ; T represents the transpose of the vector; represents Q in the i-th attention head i , K i and V i The projection matrix; F low representing said primary feature;

[0081] The reorganized primary features are integrated to obtain the morphological features representing the global semantic expression.

[0082] Understandably, the Transformer subunit can flatten the primary features into a sequence of spatial positions, represented as x i , which is the feature of the i-th 32×32 pixel region in the cell image; then, for each x i Adding a learnable position code p can enhance the perception of spatial information and obtain the encoded features, which are expressed as: embed =x i +p.

[0083] For each attention head, the encoded feature x embed , projected as query Q i (Query), key K i(Key), value V i (Value), thus obtaining the single-head attention weight Attention i (Q i , K i , V i ); Based on the attention weight, the value vector V i Perform weighted summation to obtain the reorganized primary features, which are the spatial-semantic reconstruction results of the feature map (i.e., primary features) under single-head attention.

[0084] The process of integrating the reorganized primary features is to concatenate the "reorganized primary features" output by all attention heads and perform dimensionality reduction through a linear layer to obtain the morphological features that represent the global semantic expression. The concatenation and dimensionality reduction process is conventional and will not be detailed here.

[0085] In step 124, the dynamic convolutional pyramid network performs multi-scale dynamic convolution on the cell image using a dynamic convolution kernel. This dynamic convolution kernel adaptively adjusts convolution parameters based on the input image content, effectively addressing the color gamut differences between H&E and DAB staining and enhancing robustness to staining differences. H&E and DAB are two different existing staining methods and will not be further described here.

[0086] The pyramid structure can perceive the staining intensity distribution (such as cytoplasmic uniformity and artifact areas) from multiple scales.

[0087] As an example, the coloring feature is calculated as follows:

[0088]

[0089] K dy =MLP(GAP(I)) (6)

[0090] The coloring features obtained for each layer After concatenation and dimensionality reduction, the coloring feature F output by the second branch unit 212 is obtained. stain Among them, the method of feature splicing and dimensionality reduction is conventional and will not be repeated here. Refers to the staining features obtained by extracting cell images from the s-th pyramid layer, where s represents the pyramid level number, s∈{1, 2, 3}, corresponding to different resolution processing. s (I;K dy ) represents the dynamic convolution pyramid network (DCP) that uses the dynamically generated convolution kernel K on the input image I at the sth level. dy , output the coloring feature map of the corresponding scale.

[0091] Kdy Represents a dynamic convolution kernel. GAP(I) performs global average pooling on the spatial dimensions of the input image I, outputting channel-level statistical features. This function compresses image spatial information while preserving channel-level staining intensity distribution characteristics (such as overall H&E staining depth). The MLP() function maps the output channel statistics GAP(I) to dynamic convolution kernel parameters. MLP stands for Multilayer Perceptron.

[0092] Step 125, fusing the morphological feature and the staining feature through the feature cross attention unit 213 to obtain the fused feature, may include:

[0093] The feature cross attention unit 213 is used to fuse the morphological features and the staining features based on the layer normalization function to obtain the fused features. The calculation formula for the fusion is expressed as:

[0094] F fusion =LayerNorm(F morph +FFN(Attention(Q,K,V))) (7)

[0095]

[0096] Q=W q1 F morph (9)

[0097] K=W k1 F stain (10)

[0098] V=W v1 F stain (11)

[0099] Where, F fusion Represents the fusion feature; LayerNorm() represents the layer normalization function; Attention(Q,K,V) represents the attention weight of feature intersection; Q, K, V represent the query vector, key vector and value vector of feature intersection respectively; W q1 、W k1 、W v1 Represents the projection matrices of Q, K, and V respectively; F morph Represents the morphological features; FFN() represents the feedforward neural network function; F stain Indicates the staining characteristics; d k2 Represents the dimension of the key vector K.

[0100] In this embodiment, the feature cross-attention unit 213 can suppress unimodal noise (such as staining artifacts that interfere with morphological analysis) and dynamically select complementary information between morphological and staining features (such as the nuclear membrane region with high staining contrast) through attention weighting. In the nuclear edge region, morphological features provide structural information, and staining features enhance boundary contrast, which helps enhance modal complementarity. Attention weighting facilitates the visualization of the model's focus area (such as the chromosome arrangement in a mitotic image).

[0101] The multimodal heterogeneous network model 210 adopts a two-branch deep neural network (the first branch unit 211 and the second branch unit 212), which can realize the extraction of cell morphological features and staining features (or called staining texture features); using the feature cross-attention unit 213 to fuse the two features can significantly improve the feature expression ability, which is conducive to improving the accuracy of subsequent map generation.

[0102] In step 130, based on the pre-established dynamic transfer learning module 220, the fusion feature is updated to obtain an updated fusion feature, including:

[0103] Step 131: Calculate, by the dynamic transfer learning module 220, a distribution difference between a source domain feature distribution and a target domain feature distribution, wherein the source domain feature distribution is obtained based on pre-labeled cell images, and the target domain feature distribution is obtained based on unlabeled cell images;

[0104] Step 132: Based on the distribution difference, the fusion feature is updated to obtain an updated fusion feature.

[0105] It is understandable that the dynamic transfer learning module 220 is configured as a module that can execute step 131 and step 132.

[0106] In step 131 , the method for calculating the distribution difference between the source domain feature distribution and the target domain feature distribution is conventional and will not be described in detail here.

[0107] In step 132, the fusion feature may be updated as follows:

[0108]

[0109] Where F′ fusion Refers to the updated fusion feature; μ refers to the feature update learning rate, which can be flexibly set according to the actual situation, such as a value of 0.01; τ refers to the KL divergence loss weight, which can be flexibly set according to the actual situation, such as a value of 0.1; Refers to partial differential operation; MMD refers to maximum mean difference; KL refers to divergence; MMD and KL can both be obtained based on distribution differences, and the calculation method is conventional, which will not be repeated here.

[0110] Understandably, the dynamic transfer learning module 220 can realize online adaptation of the feature space, which is beneficial to reducing the training time of the target domain data (i.e., images that need to be detected by cell atlas) without retraining the entire model; by aligning the fusion feature distribution of the source domain (labeled cell images) and the target domain (unlabeled cell images) by the maximum mean difference MMD, the inter-domain difference can be reduced. The KL divergence constraint can ensure the consistency of the source domain and the target domain in the classifier output space and prevent the loss of semantic information due to excessive alignment. In addition, in this embodiment, unsupervised cross-domain migration can be achieved by updating the fusion features online (rather than retraining the entire model).

[0111] When training the atlas generation system 200, first, prepare the source domain data (labeled cell images, the labeling content includes: cell nucleus / cytoplasm segmentation mask, cell classification label) and the target domain data (unlabeled cell images, i.e., no segmentation and classification labeling). Both the source domain data and the target domain data are pre-processed data, and the pre-processing method is conventional, such as staining normalization, data enhancement, etc. Then, the prepared source domain data and target domain data are input into the atlas generation system 200. Using the dynamic transfer learning module 220, self-supervision-dynamic weight joint training can be achieved. The task of reconstructing cell substructures (such as cell nucleus / cytoplasm masks) can be introduced to reduce labeling dependence. Combined with the confidence weighting mechanism, the training sample weights are dynamically optimized. The confidence weighting mechanism is as follows: for the target domain data, the pseudo-label confidence is calculated, the weight of high confidence (low entropy) samples is close to 1, and the weight of low confidence samples is close to 0; high confidence samples focus on the main task, and low confidence samples focus on self-supervision tasks and domain alignment. Main task (source domain data): cell classification and atlas generation. Self-supervised task (target domain data): cell substructure reconstruction.

[0112] The cell feature decoupling module 230 is configured as a module that can execute the corresponding content of step 140 .

[0113] In step 140, based on the pre-established cell feature decoupling module 230, the cell image is decoupled to obtain decoupled features, including:

[0114] The cell feature decoupling module 230 decomposes the cell image according to the adversarial feature separation algorithm to obtain the decoupled features, which are expressed as:

[0115] I=α·C nucleus +β·C cytoplasm +γ·C artifact (13)

[0116] Where, I represents the cell image, α, β, and γ are the decomposition coefficients corresponding to the reconstruction of the cell nucleus, cytoplasm, and artifacts, respectively; Cnucleus represents the cell nucleus feature after decoupling of the cell image; C cytoplasm represents the cytoplasmic features after decoupling of the cell image; C artifact Represents the artifact features after decoupling of the cell image; the decoupling features include the cell nucleus feature C nucleus 、Cytoplasmic characteristics C cytoplasm and artifact features C artifact .

[0117] In this embodiment, the adversarial disentanglement loss algorithm can be used to force morphological features (such as cell nucleus shape) and functional features (such as staining intensity) to be orthogonalized, which is beneficial to improving the interpretability of the model; the decoupled features support independent optimization strategies, which is beneficial to supporting fine-grained pathological analysis.

[0118] Step 150 may be implemented by the pre-configured graph generation module 240 .

[0119] In step 150, a cell atlas is generated based on the updated fusion features and the decoupled features, including:

[0120] Step 151, performing feature splicing on the updated fusion feature, the cell nucleus feature, and the cytoplasm feature to obtain a spliced ​​feature as an intermediate feature;

[0121] Step 152: Mapping the intermediate features into a cell atlas of a specified resolution based on a pre-established U-Net module;

[0122] Step 153: Use the artifact feature as an attention gating signal to suppress the noise region in the cell atlas, and obtain a cell atlas representing the suppressed artifact.

[0123] It is understandable that in step 151, the process of feature splicing is conventional and will not be described in detail here.

[0124] The U-Net module is configured as a neural network module that maps intermediate features to a cell atlas of a specified resolution. The intermediate features can be used to generate an initial cell atlas, and the specified resolution can be flexibly set according to actual conditions.

[0125] Artifact feature C artifact The attention weight map is generated through the convolutional layer, and the noise area is suppressed through attention gating, so as to obtain a more accurate cell map and reduce artifact interference.

[0126] In this embodiment, the method may further include:

[0127] Inputting the intermediate features into a pre-established classification module to obtain a cell classification result;

[0128] Generate a visual heat map based on the Gradient Weighted Class Activation Mapping (Grad-CAM) algorithm and the updated fusion features;

[0129] The cell image, the cell atlas after suppressing artifacts, the classification results, and the visualized heat map are displayed in a multi-view fusion manner.

[0130] In this embodiment, the classification module can be a pre-trained classifier, such as a lightweight MLP (Multilayer Perceptron) with three fully connected layers, which can output cell category probabilities based on intermediate features, thereby realizing the classification of different cells. The types of cells that support classification can be trained according to needs.

[0131] While cell classification and heatmap generation are conventional, in this example, cross-domain interpretability can be enhanced by linking classification and heatmap results. For example, by linking heatmaps with classification results, decision-making can be intuitively demonstrated, potentially enabling training for pathology residents.

[0132] During the multi-view fusion display process, a fusion display interface is provided. The fusion display interface includes a main view and an auxiliary view. The main view can superimpose a cell atlas on the original cell image (such as the original H&E image, semi-transparent). In the display interface, the function of "clicking on the heat map to locally enlarge and display the nuclear-cytoplasm ratio" can be configured. By integrating the complementary information of different imaging modalities (such as cell nuclear morphology, staining distribution, classification probability, and thermal activation area), cross-modal cross-validation is achieved through spatial alignment and superimposed display, thereby improving the reliability of abnormal area identification.

[0133] For example, bone marrow blood cells can be classified into erythroid cells, granulocyte cells, lymphocyte cells, etc. When a cell map is required for bone marrow blood cells, the method provided in this embodiment can be used to quickly generate a cell map to assist doctors in bone marrow blood cell analysis. In addition, standardized cell map examples can be provided for medical education.

[0134] In this embodiment, the processor may be an integrated circuit chip having signal processing capabilities. The above-mentioned processor may be a general-purpose processor. For example, the processor may be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, which may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application.

[0135] The memory may be, but is not limited to, a random access memory, a read-only memory, a programmable read-only memory, an erasable programmable read-only memory, an electrically erasable programmable read-only memory, etc. In this embodiment, the memory may be used to store cell images, cell atlases, etc. Of course, the memory may also be used to store programs, which the processor executes upon receiving an execution instruction.

[0136] It should be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the electronic device described above can refer to the corresponding processes of each step in the aforementioned method, and will not be elaborated here.

[0137] An embodiment of the present application also provides a program product, including a computer program, which implements the above-mentioned cell detection map generation method when executed by a processor.

[0138] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented through hardware or by means of software plus a necessary general hardware platform. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including a number of instructions for enabling a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0139] In the embodiments provided in the present application, it should be understood that the disclosed equipment and methods can also be implemented in other ways. The equipment and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the equipment, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of code, and a part of the module, program segment or code includes one or more executable instructions for implementing the specified logical function. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. In addition, the functional modules in the various embodiments of the present application can be integrated together to form an independent part, or each module can exist alone, or two or more modules can be integrated to form an independent part.

[0140] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for generating a cell detection map based on deep learning, characterized in that: The method comprises: Acquiring a cell image, wherein the cell image includes at least one image region corresponding to a cell; Inputting the cell image into a pre-established multimodal heterogeneous network model based on deep learning, obtaining a fusion feature output by the multimodal heterogeneous network model, wherein the multimodal heterogeneous network model includes a first branch unit, a second branch unit, and a feature cross-attention unit, wherein the first branch unit is a ResNet-Transformer network unit, configured to extract morphological features from the cell image; the second branch unit is a dynamic convolutional pyramid network unit, configured to extract staining features from the cell image; and the feature cross-attention unit is configured to fuse the morphological features and the staining features to obtain the fusion feature; Based on a pre-established dynamic transfer learning module, the fusion feature is updated to obtain an updated fusion feature; Decoupling the cell image based on a pre-established cell feature decoupling module to obtain decoupling features; A cell atlas is generated based on the updated fusion features and the decoupled features.

2. The method according to claim 1, characterized in that Inputting the cell image into a pre-established multimodal heterogeneous network model based on deep learning, and obtaining fusion features output by the multimodal heterogeneous network model, including: Inputting the cell image into the first branch unit and the second branch unit in the multimodal heterogeneous network model; Performing feature extraction on the cell image using the ResNet subunit in the first branch unit to obtain primary features; The Transformer subunit in the first branch unit is used to reorganize and integrate the primary features using multiple attention heads to obtain the morphological features; The dynamic convolutional pyramid network unit convolves the cell image through the second branch unit to obtain the staining feature; The morphological feature and the staining feature are fused through the feature cross attention unit to obtain the fused feature.

3. The method according to claim 2, characterized in that Multiple attention heads are used to reorganize and integrate the primary features to obtain the morphological features, including: Using each attention head, based on the corresponding attention weight, the primary features are reorganized to obtain the reorganized primary features. The attention weight is expressed as: Attention i (Q i , K i , V i ) represents the attention weight of the i-th attention head; Softmax() represents the activation function; Q i , K i and V i denote the query vector, key vector, and value vector of the i-th attention head respectively; d k1 represents the key vector K i The dimension of ; T represents the transpose of the vector; represents Q in the i-th attention head i , K i and V i The projection matrix; F low representing said primary feature; The reorganized primary features are integrated to obtain the morphological features representing the global semantic expression.

4. The method according to claim 2, characterized in that The morphological feature and the staining feature are fused through the feature cross attention unit to obtain the fused feature, including: The morphological features and the staining features are fused through the feature cross attention unit based on the layer normalization function to obtain the fused features. The calculation formula of the fusion is expressed as: F fusion =LayerNorm(F morph +FFN(Attention(Q,K,V))) Q=W q1 F morph K=W k1 F stain V=W v1 F stain Where, F fusion Represents the fusion feature; LayerNorm() represents the layer normalization function; Attention(Q,K,V) represents the attention weight of feature intersection; Q, K, V represent the query vector, key vector and value vector of feature intersection respectively; W q1 、W k1 、W v1 Represents the projection matrices of Q, K, and V respectively; F morph Represents the morphological features; FFN() represents the feedforward neural network function; F stain Indicates the staining characteristics; d k2 Represents the dimension of the key vector K.

5. The method according to claim 1, wherein Based on a pre-established dynamic transfer learning module, the fusion features are updated to obtain updated fusion features, including: Calculating, by the dynamic transfer learning module, a distribution difference between a source domain feature distribution and a target domain feature distribution, wherein the source domain feature distribution is obtained based on pre-labeled cell images and the target domain feature distribution is obtained based on unlabeled cell images; Based on the distribution difference, the fusion feature is updated to obtain an updated fusion feature.

6. The method according to claim 1, characterized in that Based on a pre-established cell feature decoupling module, the cell image is decoupled to obtain decoupled features, including: The cell feature decoupling module decomposes the cell image according to the adversarial feature separation algorithm to obtain the decoupled feature, which is expressed as: I=α·C nucleus +β·C cytoplasm +γ·C artifact Where, I represents the cell image, α, β, and γ are the decomposition coefficients corresponding to the reconstruction of the cell nucleus, cytoplasm, and artifacts, respectively; C nucleus represents the cell nucleus feature after decoupling of the cell image; C cytoplasm represents the cytoplasmic features after decoupling of the cell image; C artifact Represents the artifact features after decoupling of the cell image; the decoupling features include the cell nucleus feature C nucleus 、Cytoplasmic characteristics C cytoplasm and artifact features C artifact .

7. The method according to claim 6, characterized in that Generating a cell atlas based on the updated fusion feature and the decoupled feature, including: Performing feature splicing on the updated fusion feature, the cell nucleus feature, and the cytoplasm feature to obtain a spliced ​​feature as an intermediate feature; Based on a pre-established U-Net module, the intermediate features are mapped into a cell atlas of a specified resolution; The artifact feature is used as an attention gating signal to suppress the noise area in the cell atlas, thereby obtaining a cell atlas representing the suppressed artifact.

8. The method according to claim 7, characterized in that The method further comprises: Inputting the intermediate features into a pre-established classification module to obtain a cell classification result; Generate a visual heat map based on the gradient weighted class activation mapping algorithm and the updated fusion features; The cell image, the cell atlas after suppressing artifacts, the classification results, and the visualized heat map are displayed in a multi-view fusion manner.

9. An electronic device, characterized in that: The electronic device includes a processor and a memory coupled to each other, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the electronic device executes the method according to any one of claims 1 to 8.

10. A program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 8 when the computer program is executed by a processor.

Images