Cell morphology recognition method and device based on multimodal fusion

Abstract

The present invention discloses a cell morphology recognition method and device based on multimodal fusion. The present invention obtains a cell image of a cell to be identified, inputs the cell image into a pre-trained semantic segmentation network, predicts cell region information, combines the cell region information with the cell image, and inputs the information into a pre-trained convolutional neural network. Features of the cell type and state are extracted, and the extracted features are input into a pre-trained text decoder for decoding. The morphological description of the cell to be identified is output, and similarity matching is performed on the morphological description using a pre-built cell morphology description library to obtain the category of the cell to be identified. The present invention can efficiently and accurately identify different types of pathological cell images, has strong interpretability, and can increase the number of recognition categories by adding descriptions of new targets, reducing the number of times the model needs to be retrained.

Description

Technical Field

[0001] The present invention relates to the technical field of cell morphology recognition, and in particular to a cell morphology recognition method and device based on multimodal fusion. Background Art

[0002] Traditional cell morphology recognition relies primarily on pathologists observing cell morphological features to diagnose and classify tumors. This method relies heavily on the physician's experience for accuracy and is inefficient. With the development of deep learning technology, significant progress has been made in medical image analysis using convolutional neural networks. However, existing technologies still have limited recognition accuracy when processing complex cell tissue images, and the results are difficult to interpret, making them unable to meet the needs of large-scale clinical cell morphology analysis.

[0003] Furthermore, existing technologies require a large amount of labeled cell information, and each time a new cell type needs to be identified, the model must be retrained. Small sample sizes can lead to low classification accuracy due to sample imbalance, and frequent training can waste computing resources.

[0004] Therefore, it is necessary to propose a cell morphology recognition method and device based on multimodal fusion to solve the above problems. Summary of the Invention

[0005] The purpose of the present invention is to provide a cell morphology recognition method and device based on multimodal fusion to solve the problem that the existing technology has limited recognition accuracy when processing complex cell tissue images, the results are weak in interpretability, and it cannot meet the needs of clinical large-scale cell morphology analysis.

[0006] In a first aspect, the present invention provides a cell morphology recognition method based on multimodal fusion, comprising:

[0007] Acquire cell images of cells to be identified;

[0008] Inputting the cell image into a pre-trained semantic segmentation network to predict cell region information;

[0009] Combining the cell region information with the cell image and inputting the information into a pre-trained convolutional neural network to extract features of the cell type and state;

[0010] Inputting the extracted features into a pre-trained text decoder for decoding, and outputting a morphological description of the cells to be identified;

[0011] The morphological descriptions are matched for similarity using a pre-built cell morphological description library to obtain the category of the cells to be identified.

[0012] Furthermore, the cell image is input into a pre-trained semantic segmentation network to predict cell region information, including:

[0013] The cell region and the cell nucleus region of the cell image are segmented by a semantic segmentation network to obtain cell region information, which is semantic segmentation information.

[0014] Furthermore, the cell region information and the cell image are combined and input into a pre-trained convolutional neural network to extract features of cell type and state, including:

[0015] The semantic segmentation information is combined with the cell image and input into a pre-trained image encoder to extract the morphological feature vector.

[0016] Furthermore, the extracted features are input into a pre-trained text decoder for decoding, and a morphological description of the cells to be identified is output, including:

[0017] The extracted morphological feature vectors are converted into specific cell description information through a text decoder.

[0018] Furthermore, similarity matching is performed on the morphological description using a pre-built cell morphological description library to obtain the category of the cell to be identified, including:

[0019] Matching the cell description information with the information in a pre-built cell morphology description library, and calculating the similarity of six pieces of information: cell size, cell shape, cell nucleus, cytoplasm, cell envelope, and special staining;

[0020] The cell category corresponding to the information with the highest similarity among the six information is used as the identified cell category.

[0021] Furthermore, the similarity is calculated as follows:

[0022] according to , calculate the similarity; wherein, is the feature vector obtained by encoding and downsampling the information in the cell morphology description library, The cell description information of the cell to be identified.

[0023] Furthermore, the cell image is input into a pre-trained semantic segmentation network, and in the step of predicting cell region information, the semantic segmentation network is trained based on the category annotation of local cell data, and the local cell data has descriptive information of specific cells, and the descriptive information includes: cell size, cell shape, cell nucleus, cytoplasm, cell capsule and special staining descriptive information, wherein the contour information of the cell nucleus and cytoplasm is used to train the semantic segmentation network.

[0024] Furthermore, in the step of inputting the extracted features into a pre-trained text decoder for decoding and outputting a morphological description of the cells to be identified, the text decoder is trained as follows:

[0025] The UNet model is used to extract cell semantic information. The target image X is input and the channel number Y is obtained. The first channel Y0 represents the cell area and the second channel Y1 represents the cell nucleus area.

[0026] The loss is calculated by the ratio of the overlapping area between the semantic label L of the target image X and the model output Y:

[0027] ;

[0028] Let Z represent the combination of cell semantic information and cell image X&Y, and extract features from Z through the image encoder. The feature vector length is 1024 and is recorded as a;

[0029] The text description T of the corresponding cell image is converted into a word vector t using the word2vector tool, and then feature extracted by the text encoder and recorded as b;

[0030] Calculate the cosine similarity Sim between features a and b;

[0031] ;

[0032] At the same time, the text decoder is trained by cross entropy CE and KL divergence. is the word vector obtained by the decoder, Indicates codec;

[0033] .

[0034] Furthermore, the method further comprises:

[0035] When a new cell type needs to be added, the cell description information of the new cell type is added to the cell morphology description library.

[0036] In a second aspect, the present invention provides a cell morphology recognition device based on multimodal fusion, comprising:

[0037] an acquisition unit, used for acquiring cell images of cells to be identified;

[0038] a segmentation unit, configured to input the cell image into a pre-trained semantic segmentation network to predict cell region information;

[0039] An extraction unit, configured to combine the cell region information and the cell image, input the information into a pre-trained convolutional neural network, and extract features of the cell type and state;

[0040] a decoding unit, configured to input the extracted features into a pre-trained text decoder for decoding, and output a morphological description of the cells to be identified;

[0041] The matching unit is used to perform similarity matching on the morphological description using a pre-built cell morphological description library to obtain the category of the cell to be identified.

[0042] The beneficial effects of the present invention are as follows: The present invention provides a cell morphology recognition method and device based on multimodal fusion, which obtains a cell image of a cell to be identified, inputs the cell image into a pre-trained semantic segmentation network, predicts cell region information, combines the cell region information and the cell image, and inputs them into a pre-trained convolutional neural network, extracts features of cell type and state, inputs the extracted features into a pre-trained text decoder for decoding, outputs a morphological description of the cell to be identified, performs similarity matching on the morphological description through a pre-constructed cell morphology description library, and obtains the category of the cell to be identified. It can efficiently and accurately identify different types of pathological cell images, has strong interpretability, and can increase the number of recognition categories by manually adding descriptions of new targets, thereby reducing the number of times the model is retrained. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0044] Figure 1 This is a flow chart of a cell morphology recognition method based on multimodal fusion according to the present invention;

[0045] Figure 2 This is a diagram of the semantic segmentation network structure based on Unet;

[0046] Figure 3 It is the feature fusion network structure diagram;

[0047] Figure 4 It is a schematic diagram describing the comparison module;

[0048] Figure 5 This is a schematic diagram of a cell morphology recognition device based on multimodal fusion according to the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention. The technical solutions provided by each embodiment of the present invention are described in detail below in conjunction with the drawings.

[0050] See also Figures 1 to 4 , an embodiment of the present invention provides a cell morphology recognition method based on multimodal fusion, comprising:

[0051] S101, obtaining a cell image of a cell to be identified.

[0052] S102, inputting the cell image into a pre-trained semantic segmentation network to predict cell region information.

[0053] Specifically, the semantic segmentation network is trained based on the category annotations of local cell data. The local cell data contains descriptive information about specific cells, including cell size, cell shape, nucleus, cytoplasm, cell envelope, and special staining. The nucleus and cytoplasm outline information is used to train the semantic segmentation network. The semantic segmentation network is used to segment the cell region and nucleus region of the cell image to obtain cell region information, which serves as semantic segmentation information.

[0054] A semantic segmentation network is a deep learning model used to assign each pixel in an image or video to a specific category. It is commonly used for computer vision tasks such as object detection, segmentation, and recognition. The basic structure of a semantic segmentation network consists of a convolutional neural network, upsampling or deconvolution layers, and fully connected layers. Convolutional neural networks are used to extract features from the input image, upsampling or deconvolution layers are used to scale the feature maps to the same size as the original image, and fully connected layers are used to classify each pixel. U-Net is a classic semantic segmentation network consisting of a contracting path (encoder) and an expanding path (decoder), similar to a U-shaped structure. It uses skip connections to retain more detailed information, thereby improving segmentation accuracy.

[0055] S103: combining the cell region information and the cell image, and inputting the information into a pre-trained convolutional neural network to extract features of the cell type and state.

[0056] Specifically, the semantic segmentation information is combined with the cell image and input into a pre-trained image encoder to extract the morphological feature vector.

[0057] S104: Input the extracted features into a pre-trained text decoder for decoding, and output a morphological description of the cells to be identified.

[0058] Specifically, the text decoder training method is as follows:

[0059] The UNet model is used to extract cell semantic information. The target image X is input and the channel number Y is obtained. The first channel Y0 represents the cell area and the second channel Y1 represents the cell nucleus area.

[0060] The loss is calculated by the ratio of the overlapping area between the semantic label L of the target image X and the model output Y:

[0061] ;

[0062] This loss is used for training Figure 2 The segmentation network is used to provide prior knowledge (the regions of the cell nucleus and cytoplasm) to the subsequent multimodal model, which facilitates further alignment with the text description information.

[0063] Let Z represent the combination of cell semantic information and cell image X&Y, and extract features from Z through the image encoder. The feature vector length is 1024 and is recorded as a;

[0064] The text description T of the corresponding cell image is converted into a word vector t using the word2vector tool. This is then used to extract features using a text encoder, denoted as b. Word2vector is a natural language processing tool that calculates the distance between words based on a set of input words. It converts words into vectors for vector operations in vector space and calculates similarity in the vector space to represent the semantic similarity of the text.

[0065] Calculate the cosine similarity Sim between features a and b;

[0066] Cosine similarity (Sim) is used to train the multimodal model, serving as a loss to align image and text features, achieving a one-to-one correspondence between images and text. This allows for subsequent input of cell images and output of corresponding text descriptions. The multimodal model consists of an image encoder, a text encoder, and a decoder.

[0067] At the same time, the text decoder is trained by cross entropy CE and KL divergence. is the word vector obtained by the decoder, Indicates codec;

[0068] .

[0069] Cross-entropy is a key concept in Shannon's information theory, primarily used to measure the difference between two probability distributions. In machine learning, cross-entropy is a commonly used loss function that describes the discrepancy between a model's prediction and the true value. In language models, cross-entropy measures the difficulty of the model's prediction and can also be viewed as the compression ratio of the model's representation of text. Cross-entropy represents the difficulty of the model in recognizing text, or, from a compression perspective, the average number of bits required to encode each word. Cross-entropy is also commonly used in classification problems. In multi-classification problems, cross-entropy is used as a loss function to optimize the difference between the model's prediction and the true label. In binary classification problems, the cross-entropy loss function is used to assess the model's ability to distinguish between positive and negative examples. The Kullback-Leibler divergence (KLD) is a measure of relative entropy, also known as information divergence or information gain. It measures the difference between two probability distributions. In information theory, relative entropy is equivalent to the difference in the information entropy of the two probability distributions.

[0070] Through the text decoder, the extracted morphological feature vectors are converted into specific cell description information, which can be provided to professional physicians as a reference for interpretable analysis.

[0071] S105 , performing similarity matching on the morphological description using a pre-built cell morphological description library to obtain the category of the cell to be identified.

[0072] Specifically, the cell description information is matched with the information in the pre-built cell morphology description library, and the similarity of six pieces of information, namely, cell size, cell shape, cell nucleus, cytoplasm, cell envelope, and special staining, is calculated respectively;

[0073] The cell category corresponding to the information with the highest similarity among the six information is used as the identified cell category.

[0074] In this embodiment, the similarity is calculated as follows:

[0075] according to , calculate the similarity; wherein, is the feature vector obtained by encoding and downsampling the information in the cell morphology description library, The cell description information of the cell to be identified.

[0076] like Figure 4 As shown in Figure 2, the description F is the information in the cell information library, and F' is the information of the cell to be identified. To increase the speed, the description F is encoded and downsampled to obtain a feature vector when it is recorded into the information library.

[0077] When a new cell category needs to be added, the model can be fine-tuned without using traditional deep learning methods. The same effect can be achieved by adding the cell description information of the new cell category to the cell morphology description library.

[0078] See also Figure 5 , an embodiment of the present invention provides a cell morphology recognition device based on multimodal fusion, comprising:

[0079] An acquisition unit 51 is used to acquire a cell image of a cell to be identified;

[0080] a segmentation unit 52, configured to input the cell image into a pre-trained semantic segmentation network to predict cell region information;

[0081] An extraction unit 53 is used to combine the cell region information and the cell image, input the information into a pre-trained convolutional neural network, and extract features of the cell type and state;

[0082] A decoding unit 54 is configured to input the extracted features into a pre-trained text decoder for decoding, and output a morphological description of the cells to be identified;

[0083] The matching unit 55 is configured to perform similarity matching on the morphological description using a pre-built cell morphological description library to obtain the category of the cell to be identified.

[0084] Embodiments of the present invention further provide a storage medium storing a computer program that, when executed by a processor, implements some or all of the steps of each embodiment of the multimodal fusion-based cell morphology recognition method provided herein. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0085] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software and a necessary general-purpose hardware platform. Based on this understanding, the technical solutions in the embodiments of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium such as ROM / RAM, a magnetic disk, or an optical disk, and includes instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in various embodiments of the present invention, or portions thereof.

[0086] In this specification, the same or similar parts between the various embodiments can be referred to each other. In particular, for the embodiment of the cell morphology recognition device based on multimodal fusion, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the description in the method embodiment.

[0087] The above-described embodiments of the present invention do not limit the protection scope of the present invention.

Claims

1. A cell morphology recognition method based on multimodal fusion, characterized in that: include: Acquire cell images of cells to be identified; Inputting the cell image into a pre-trained semantic segmentation network to predict cell region information; Combining the cell region information with the cell image and inputting the information into a pre-trained convolutional neural network to extract features of the cell type and state; The extracted features are input into a pre-trained text decoder for decoding, and a morphological description of the cells to be identified is output; the text decoder is trained as follows: Use the UNet model to extract cell semantic information, input the target image X, and obtain Y with 2 channels. The first channel Indicates cell area, second channel Indicates the nuclear region; The loss is calculated by the ratio of the overlapping area between the semantic label L of the target image X and the model output Y: ； Let Z represent the combination of cell semantic information and cell image X&Y, and extract features from Z through the image encoder. The feature vector length is 1024 and is recorded as a; The text description T of the corresponding cell image is converted into a word vector t using the word2vector tool, and then feature extracted by the text encoder and recorded as b; Calculate the cosine similarity Sim between features a and b; ； At the same time, the text decoder is trained by cross entropy CE and KL divergence. is the word vector obtained by the decoder, Indicates codec; ； The morphological descriptions are matched for similarity using a pre-built cell morphological description library to obtain the category of the cells to be identified.

2. The cell morphology recognition method based on multimodal fusion according to claim 1, characterized in that: The cell image is input into a pre-trained semantic segmentation network to predict cell region information, including: The cell region and the cell nucleus region of the cell image are segmented by a semantic segmentation network to obtain cell region information, which is semantic segmentation information.

3. The cell morphology recognition method based on multimodal fusion according to claim 2, characterized in that: The cell region information and the cell image are combined and input into a pre-trained convolutional neural network to extract features of the cell type and state, including: The semantic segmentation information is combined with the cell image and input into a pre-trained image encoder to extract the morphological feature vector.

4. The cell morphology recognition method based on multimodal fusion according to claim 3, characterized in that: The extracted features are input into a pre-trained text decoder for decoding, and a morphological description of the cells to be identified is output, including: The extracted morphological feature vectors are converted into specific cell description information through a text decoder.

5. The cell morphology recognition method based on multimodal fusion according to claim 4, characterized in that: The morphological description is matched by similarity using a pre-built cell morphological description library to obtain the category of the cell to be identified, including: Matching the cell description information with the information in a pre-built cell morphology description library, and calculating the similarity of six pieces of information: cell size, cell shape, cell nucleus, cytoplasm, cell envelope, and special staining; The cell category corresponding to the information with the highest similarity among the six information is used as the identified cell category.

6. The cell morphology recognition method based on multimodal fusion according to claim 5, characterized in that: The similarity is calculated as follows: according to , calculate the similarity; wherein, is the feature vector obtained by encoding and downsampling the information in the cell morphology description library, The cell description information of the cell to be identified.

7. The cell morphology recognition method based on multimodal fusion according to claim 6, characterized in that: The cell image is input into a pre-trained semantic segmentation network, and in the step of predicting cell region information, the semantic segmentation network is trained based on the category annotation of local cell data, and the local cell data has descriptive information of specific cells, and the descriptive information includes: description information of cell size, cell shape, cell nucleus, cytoplasm, cell capsule and special staining, wherein the contour information of the cell nucleus and cytoplasm is used to train the semantic segmentation network.

8. The cell morphology recognition method based on multimodal fusion according to claim 7, characterized in that: The method further comprises: When a new cell type needs to be added, the cell description information of the new cell type is added to the cell morphology description library.

9. A cell morphology recognition device based on multimodal fusion, characterized in that: include: an acquisition unit, used for acquiring cell images of cells to be identified; a segmentation unit, configured to input the cell image into a pre-trained semantic segmentation network to predict cell region information; An extraction unit, configured to combine the cell region information and the cell image, input the information into a pre-trained convolutional neural network, and extract features of the cell type and state; The decoding unit is used to input the extracted features into a pre-trained text decoder for decoding, and output a morphological description of the cells to be identified; the text decoder is trained as follows: Use the UNet model to extract cell semantic information, input the target image X, and obtain Y with 2 channels. The first channel Indicates cell area, second channel Indicates the nuclear region; The loss is calculated by the ratio of the overlapping area between the semantic label L of the target image X and the model output Y: ； Let Z represent the combination of cell semantic information and cell image X&Y, and extract features from Z through the image encoder. The feature vector length is 1024 and is recorded as a; The text description T of the corresponding cell image is converted into a word vector t using the word2vector tool, and then feature extracted by the text encoder and recorded as b; Calculate the cosine similarity Sim between features a and b; ； At the same time, the text decoder is trained by cross entropy CE and KL divergence. is the word vector obtained by the decoder, Indicates codec; ； The matching unit is used to perform similarity matching on the morphological description using a pre-built cell morphological description library to obtain the category of the cell to be identified.

Images