Bone marrow cell recognition system based on multi-modal enhancement

Abstract

The application discloses a kind of bone marrow cell identification systems based on multi-modal enhancement, comprising: cell image segmentation module, for the improved SAM model for the segmentation of bone marrow cell microscope image data;The module combines SAM model and Adapter block;Label text modification module is used to convert traditional label text into coarse and fine-grained text feature representation;Multi-modal alignment module uses an improved PMC-CLIP model to align the coarse and fine-grained text features output by the label text modification module to the image-text.The coarse-grained + fine-grained text information prompt of the present application can effectively improve the recognition accuracy of species with few cell numbers and complex cell images, solve the problem of unbalanced cell species and difficult-to-distinguish complex images, and improve the recognition accuracy of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent identification of bone marrow cells, and in particular to a bone marrow cell identification system based on multimodal enhancement. Background Art

[0002] Bone marrow cytomorphology plays an indispensable role in the diagnosis, assessment, and treatment planning of hematological diseases. This examination process includes several key steps, from bone marrow sample collection to preparation of bone marrow smears, microscopic observation after staining, and finally cell count and final diagnosis report. Microscopic observation is a crucial step, and doctors need to carefully examine cell morphology, structure, staining characteristics, and karyotype to ensure accurate identification of disease type, assessment of disease progression, and design of optimal treatment strategies.

[0003] However, traditional microscopic observation methods have several drawbacks, such as being time-consuming, tedious, and requiring a high level of physician experience. Therefore, research and development of automated bone marrow cell identification systems are crucial. Such systems could significantly reduce physician workload and improve diagnostic efficiency and accuracy. By combining advanced image processing techniques, deep learning, and artificial intelligence algorithms, bone marrow cells can be automatically identified and classified, providing physicians with timely auxiliary judgment and decision support, significantly improving patient diagnosis and treatment. This article will explore new solutions to this challenge using the latest deep learning technology.

[0004] With the rapid advancement of computing hardware, the accumulation of large-scale data, and the widespread application of large models, the integration of traditional bone marrow cell detection with large models has great potential. In deep learning-based bone marrow cell recognition systems, two approaches are generally used: one is to perform cell segmentation first and then classification; the other is to perform cell segmentation and classification simultaneously, i.e., an end-to-end object detection approach. However, neither of these approaches currently considers the issue of cell type imbalance and has low accuracy when dealing with complex cell images. This results in low accuracy when identifying a small number of cell types and low recognition rates for complex cell images. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a bone marrow cell recognition system based on multimodal enhancement. Through coarse-grained + fine-grained text information prompts, it can effectively improve the recognition accuracy of types with a small number of cells and complex cell images, solve the problems of cell type imbalance and difficulty in distinguishing complex images, and improve the recognition accuracy of the model.

[0006] To achieve the above objectives, the present invention provides a technical solution: a bone marrow cell recognition system based on multimodal enhancement, comprising:

[0007] The cell image segmentation module is an improved SAM model for segmenting bone marrow cell microscopy image data. This module combines the SAM model and the Adapter block. By embedding the ViT block of the SAM model into the Adapter block, it extracts and adjusts the features of the image data. This module freezes the other parameters of the SAM model and only updates the Adapter block. It also adds segmentation hints and mask processing to improve the accuracy and robustness of the model for bone marrow cell microscopy image data, thereby obtaining an embedded representation of the segmented cell image.

[0008] The label text transformation module is used to convert traditional label text into coarse-grained and fine-grained text feature representations. Its operation is as follows: first, coarse-grained text information is constructed using a text generator. A text generator is a program that generates text and can generate corresponding text according to the established content. The label text is segmented and semantically analyzed to extract coarse-grained semantic information to form a coarse-grained text feature set. Then, fine-grained text information is constructed using natural language processing or manual processing. The fine-grained text information provides a detailed text description of the cell image based on the characteristics of the cell image to form a fine-grained text feature set. Then, the coarse-grained text information and the fine-grained text information are transmitted to the BERT text encoder, and the coarse-grained text information and the fine-grained text information are mapped to a low-dimensional semantic space, thereby converting them into coarse-grained text features and fine-grained text features.

[0009] The multimodal alignment module uses an improved PMC-CLIP model to achieve image-text alignment of the coarse- and fine-grained text features output by the label text transformation module. The improved PMC-CLIP model adds coarse- and fine-grained text features to the PMC-CLIP model. The cross-entropy loss is calculated and normalized using the coarse- and fine-grained text features of the cell, and the cross-entropy loss is used to backpropagate the loss to guide the model's use of fine- and coarse-grained text information. Finally, the classification result is output through the classification head.

[0010] Furthermore, the cell image segmentation module reads a microscope image from the bone marrow cell microscope image data and then performs the following process: first, the original image is randomly cropped into small blocks of 32*32, and then segmentation points and segmentation lines are obtained on the image according to the segmentation prompts. The SAM model is used to segment the cells and the background image. Then, the parameters of the Adapter block are adjusted, the loss function is optimized, and the noise points and incomplete cells are removed. The remaining contours with large areas are the cell contours. All the segmented cell images of the microscope image are combined into a segmented cell image dataset; all microscope images are generated again according to the above process to obtain a segmented bone marrow cell dataset.

[0011] Furthermore, the improved SAM model includes an image encoder, a prompt encoder, a mask encoder, a mask decoder and an Adapter block; the image encoder refers to the input microscope image being convolved and rearranged by 3×3, and then the rearranged result and the position embedding are input into the Transformer block to generate an image embedding, and finally the image embedding is convolved and added with the result output by the mask encoder to form a new image embedding, and then the box segmentation prompt and point segmentation prompt input by the user are input into the prompt encoder to generate a prompt embedding, and then the prompt embedding and the new image embedding are input into the mask decoder to generate a prediction; the Adapter block is composed of a high-order fully connected layer, a RELU activation function layer and a low-order fully connected layer. Since the Adapter block is added to each layer for parameter adjustment, the SAM model will better segment the image; the Mask encoder includes a convolution layer, a normalization layer and a GELU activation function layer for generating a mask embedding to provide more information for the image encoder; the prompt encoder includes a point prompt encoder and a box prompt encoder. The point prompt encoder generates a point embedding by adding the point segmentation prompt provided by the user through coordinate normalization and pixel label encoding, and then The pixel labels are weighted and added to the box label embedding. The box prompt encoder generates box embedding by rearranging and normalizing the box segmentation prompts provided by the user, and finally weightedly adding it to the point label embedding. The mask substring and confidence substring with initial values ​​are then spliced ​​together to predict the image. The spliced ​​result passes through the self-attention layer and the normalization layer and then enters the mask decoder; the mask decoder includes a cross attention layer, a normalization layer, a multi-layer perceptron layer and a transposed convolution layer. The multi-layer perceptron layer consists of an input layer, a hidden layer and an output layer. The number of neural nodes in the hidden layer is 1024. The mask decoder performs the following operations: First, After the results of the prompt encoder and the image encoder pass through the cross-attention layer, they learn each other's features and are then standardized. Due to the different dimensions, the results of the prompt encoder need to go through a multi-layer perceptron layer, and then enter a cross-attention layer to learn each other's information. After that, the mask substring and confidence substring are taken out from the result of the prompt encoder, and then enter a three-layer multi-layer perceptron layer respectively. The result of the mask substring needs to be multiplied with the result of the transposed convolution of the image encoder, so that the confidence prediction and mask prediction are obtained, so as to perform the next back propagation to optimize the prediction result and finally obtain the segmented image embedding.

[0012] Furthermore, the label text transformation module performs the following operations:

[0013] 1) Read the source image and corresponding label information from the bone marrow cell dataset;

[0014] 2) generating detailed textual description content of the cell according to the image content of the source image and professional cell description knowledge;

[0015] 3) inputting the generated coarse-grained and fine-grained text information into a BERT text encoder, the BERT text encoder comprising text preprocessing and a Transformer encoder; the text preprocessing is first sentence tokenization, then sentence sub-stringization and input into an embedding layer to generate embedding, and then the embedding is added with position embedding and segment embedding and input into the Transformer encoder after rearrangement, and finally the rearrangement and normalization are performed to obtain the output result of the unified dimension, and finally the coarse-grained and fine-grained text feature representation is obtained; the Transformer encoder comprises a normalization layer, a multi-head attention layer and a multi-layer perception layer, and is used to map the text information to a low-dimensional semantic space.

[0016] Further, the multi-modal alignment module is an improved PMC-CLIP model, comprising a ViT image encoder, an image-text alignment module, an improved Loss function module and a back propagation module, wherein:

[0017] The ViT image encoder is to generate corresponding microscope image embedding by backbone network from input microscope image, and the operation is: first, the input microscope image is first subjected to image enhancement, here random cropping and combined enhancement are adopted, random cropping means randomly selecting a region from the original image and cropping it to produce a new image, combined enhancement means combining a plurality of different enhancement methods to produce more rich and diverse image data, the enhancement methods include contrast enhancement, color enhancement, scale transformation, rotation and flip and noise addition, then batch division and projection transformation are performed, and then a image embedding layer is entered for feature extraction, the extracted features and the position embedding with initial value are reorganized into a feature sequence, then the feature sequence is taken as input and processed through a series of Transformer modules, each Transformer module comprises a normalization layer, a multi-head attention layer and a multi-head perception layer, which is used to capture the global dependence between features and perform nonlinear transformation of features, after processing, the embedding representation corresponding to the input microscope image is generated, which is used for subsequent tasks;

[0018] The image-text alignment module aligns coarse-grained and fine-grained text similarity embeddings with random image similarity embeddings, while adding a scaling factor to transform the dimensions of text features and image embeddings. This results in text similarity embeddings and image similarity embeddings that are aligned in a high-dimensional semantic space. The goal is to maximize the semantic relevance between image and text, thereby obtaining an image-text pairing matrix. Finally, the resulting image-text pairing matrix is ​​used for various cross-modal applications.

[0019] The improved Loss function module is used to calculate the Loss function of the image-text feature matrix. Due to the addition of coarse-grained text information, the Loss function needs to be improved. The improvement is: total =L graph +λ((1-a)L coarse +aL fine ), where L total Represents the total loss, L coarse represents the coarse-grained text-image loss, L fine represents the fine-grained text-image loss, L graph Represents the real image-prediction image loss, L graph , L coarse , L fine It is calculated using the cross entropy function, where λ and a are hyperparameters;

[0020] The back-propagation module is used to optimize the model and optimize the model structure according to the result of the loss function. It calculates the gradient of the loss function and uses the gradient descent algorithm to update the model parameters in the direction of the gradient, so that the output of the model is closer to the true label, that is, the correct image-text pairing is closer, and the incorrect pairing is further away. Finally, the classification result is obtained through the classification head.

[0021] Furthermore, the multimodal alignment module learns useful knowledge from the bone marrow cell dataset and coarse and fine-grained text features to train an improved PMC-CLIP model, including the following steps:

[0022] 1) Set the number of training times, initial learning rate and batch size, and start training;

[0023] 2) The images segmented by the improved SAM model are used as the input image set, and the coarse-grained text features are used as the input text feature set. A batch-sized image subset A is randomly selected from the image set, and the corresponding text feature subset B is selected from the text feature set corresponding to the image subset A. Subsets A and B are aligned one-to-one in the high-dimensional semantic space, with one as the horizontal axis and the other as the vertical axis.

[0024] 3) Calculate the cross-entropy loss so that the difference between the corresponding image and text features is as small as possible, thereby achieving better image-text alignment, while the calculated loss for the rest of the image and text features is as large as possible;

[0025] 4) Similar to the coarse-grained text, the same alignment and loss calculation steps are performed for the fine-grained text to ensure semantic consistency and alignment between the image and the fine-grained text;

[0026] 5) The improved PMC-CLIP model calculates the loss Loss and updates the weights and parameters of the model by backpropagation to minimize the loss function; wherein Loss is represented as:

[0027]

[0028]

[0029] L coarse =-(L sup (v,l)+L sup (l,v)))

[0030] L fine =-(L sup (v,l)+L sup (l,v)))

[0031] L total =L graph +λ((1-a)L coarse +aL fine )

[0032] In the above formula, L total represents the total loss, L graph represents the real image-predicted image loss, L ce represents the cross-entropy loss, L sup represents the text feature-image feature loss, L coarse represents the coarse-grained text-image loss, and L fine represents the fine-grained text-image loss; in L graph , N represents the total number of samples, y i represents the one-hot encoding of the label information x i , C(x i ) represents the predicted result, and the image loss is calculated by L graph ; in L sup , L coarse , and L fine , L sup represents the text feature-image feature loss, L coarse represents the coarse-grained text-image loss, and Lfine represents the fine-grained text-image loss, i is the index, m and n represent the embedded features, represents the transposed feature of m embeddings at the ith index, Indicates the transposition of the i-th index n embedding feature; in L ce In, p i Represents the predicted probability of the i-th cell; in L sup In (m,n), P n (i) represents the positive sample set of the i-th batch, n represents the embedded features, A n (i) represents the negative sample set of the i-th batch, |P(i)| represents the total number of positive samples in the i-th batch, Represents one of the samples in a positive sample set, Represents one of the samples in a negative sample set, τ represents the temperature parameter in softmax, this parameter will be trained during the training process, through L sup Can define L coarse and L fine ; in L coarse and L fine In the example, l and v represent the text feature sample set and image feature sample set respectively, P l (i) represents the positive sample set of the i-th batch of text features, P v (i) represents the positive sample set of the i-th batch of image features, P(i) represents the positive sample set of the i-th batch, A l (i) represents the negative sample set of the i-th batch of text features, A v (i) represents the negative sample set of the i-th batch of image features, and |P l (i)|=|P v (i)|=|P(i)|; During the optimization process, the image features and text features belonging to the same category are placed in P l (i) and P v In (i), put the out-of-class features into A v (i) and A l (i) In; at the same time L coarse and L fine It will make the features of samples of the same type closer and the features of samples of different types farther apart; L total The λ in is a hyperparameter used for balanced alignment; a is used to control the contribution rate of coarse-grained text features and fine-grained text features, which is also a hyperparameter.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. The present invention uses a cell image segmentation module to generate a segmented bone marrow cell dataset based on the bone marrow cell microscope image data, assisting the model in obtaining more features about the bone marrow cells.

[0035] 2. The present invention uses a label text transformation module to enable the model to fully utilize text information to assist classification, add relevant information to the data set in disguise, increase the accuracy of model recognition, and solve the problem of unclear text descriptions in the data set.

[0036] 3. The present invention improves the classification ability of the PMC-CLIP model for bone marrow cells by adding coarse and fine-grained text features, so that the model uses the coarse and fine-grained text features of cells and the cross-entropy loss of the segmented image to guide the model to capture more cell classification information, thereby improving the model recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 It is a schematic diagram of the relationship between various modules of the system of the present invention.

[0038] Figure 2 It is a structural diagram of the cell image segmentation model.

[0039] Figure 3 It is a structural diagram of the label text transformation module.

[0040] Figure 4 It is a structural diagram of the multimodal alignment module. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to specific embodiments.

[0042] This embodiment provides a bone marrow cell recognition system based on multimodal enhancement. It is a bone marrow cell recognition system developed using Python language and can be run on multiple platforms. The relationship between the various modules of the system is as follows: Figure 1 As shown, it includes:

[0043] The cell image segmentation module is an improved SAM model for segmenting bone marrow cell microscopy image data. This module combines the SAM model with the Adapter block. By embedding the SAM model's ViT (Vision Transformer) block into the Adapter block, it extracts and adjusts the image data's features. This module freezes the other parameters of the SAM model and only updates the Adapter block. It also incorporates segmentation hints and mask processing to improve the model's segmentation accuracy and robustness for bone marrow cell microscopy image data, thereby obtaining an embedded representation of the segmented cell image.

[0044] The label text transformation module is used to convert traditional label text into coarse-grained and fine-grained text feature representations. Its operation is as follows: first, coarse-grained text information is constructed using a text generator. A text generator is a program that generates text and can generate corresponding text according to the established content. The label text is segmented and semantically analyzed to extract coarse-grained semantic information. Then, fine-grained text information is constructed using natural language processing or manual processing. The fine-grained text information provides a detailed text description of the cell image based on the characteristics of the cell image. Then, the coarse-grained text information and fine-grained text information are transmitted to the BERT text encoder, and the coarse-grained text information and fine-grained text information are mapped to a low-dimensional semantic space, thereby converting them into coarse-grained text features and fine-grained text features.

[0045] The multimodal alignment module uses an improved PMC-CLIP model to achieve image-text alignment of the coarse- and fine-grained text features output by the label text transformation module. The improved PMC-CLIP model adds coarse- and fine-grained text features to the PMC-CLIP model. The cross-entropy loss is calculated and normalized using the coarse- and fine-grained text features of the cell, and the cross-entropy loss is used to backpropagate the loss to guide the model's use of fine- and coarse-grained text information. Finally, the classification result is output through the classification head.

[0046] Specifically, if Figure 2 As shown, the cell image segmentation module reads a microscope image from the bone marrow cell microscope image data and then performs the following process: first, the original image is randomly cropped into small blocks of 32*32, and then segmentation points and segmentation lines are obtained on the image according to the segmentation prompts. The SAM model is used to segment the cells and the background image. Then, the parameters of the Adapter block are adjusted, the loss function is optimized, and the noise points and incomplete cells are removed. The contours with large areas left are the cell contours. All the segmented cell images of the microscope image are combined into a segmented cell image dataset. All microscope images are generated again according to the above process to obtain the segmented bone marrow cell dataset.

[0047] Specifically, the improved SAM model includes an image encoder, a prompt encoder, a mask encoder, a mask decoder and an Adapter block; the image encoder refers to the input microscope image being convolved and rearranged by 3×3, and then the rearrangement result and position embedding are input into the Transformer block to generate an image embedding, and finally the image embedding is convolved and added with the result output by the mask encoder to form a new image embedding, and then the box segmentation prompt and point segmentation prompt input by the user are input into the prompt encoder to generate a prompt embedding, and then the prompt embedding and the new image embedding are input into the mask decoder to generate a prediction; the Adapter block is composed of a high-order fully connected layer, a RELU activation function layer and a low-order fully connected layer. Since the Adapter block is added to each layer for parameter adjustment, the SAM model will better segment the image; the Mask encoder includes a convolution layer, a normalization layer and a GELU activation function layer for generating a mask embedding to provide more information for the image encoder; the prompt encoder includes a point prompt encoder and a box prompt encoder. The point prompt encoder generates a point embedding by adding the point segmentation prompt provided by the user through coordinate normalization and pixel label encoding, and then The pixel labels are weighted and added to the box label embedding. The box prompt encoder generates box embedding by rearranging and normalizing the box segmentation prompts provided by the user, and finally weightedly adding it to the point label embedding. The mask substring and confidence substring with initial values ​​are then spliced ​​together to predict the image. The spliced ​​result passes through the self-attention layer and the normalization layer and then enters the mask decoder; the mask decoder includes a cross attention layer, a normalization layer, a multi-layer perceptron layer and a transposed convolution layer. The multi-layer perceptron layer consists of an input layer, a hidden layer and an output layer. The number of neural nodes in the hidden layer is 1024. The mask decoder performs the following operations: First, After the results of the prompt encoder and the image encoder pass through the cross-attention layer, they learn each other's features and are then standardized. Due to the different dimensions, the results of the prompt encoder need to go through a multi-layer perceptron layer, and then enter a cross-attention layer to learn each other's information. After that, the mask substring and confidence substring are taken out from the result of the prompt encoder, and then enter a three-layer multi-layer perceptron layer respectively. The result of the mask substring needs to be multiplied with the result of the transposed convolution of the image encoder, so that the confidence prediction and mask prediction are obtained, so as to perform the next back propagation to optimize the prediction result and finally obtain the segmented image embedding.

[0048] Specifically, if Figure 3 As shown, the label text transformation module performs the following operations:

[0049] 1) Read the source image and corresponding label information from the bone marrow cell dataset;

[0050] 2) Generate detailed text descriptions of cells based on the image content of the source image and professional cell description knowledge;

[0051] 3) The generated coarse-grained and fine-grained text information is input into the BERT text encoder, which includes text preprocessing and a Transformer encoder; the text preprocessing is to first tokenize the sentence, then substring the sentence and input it into the embedding layer to generate an embedding, which is then added to the position embedding and segment embedding and then rearranged and input into the Transformer encoder. Finally, after rearrangement and normalization, an output result of unified dimension is obtained, and finally coarse-grained and fine-grained text feature representation is obtained; the Transformer encoder includes a normalization layer, a multi-head attention layer and a multi-layer perceptron layer, which is used to map the text information into a low-dimensional semantic space.

[0052] Specifically, the multimodal alignment module is an improved PMC-CLIP model, including a ViT image encoder, an image-text alignment module, an improved Loss function module, and a back-propagation module, wherein:

[0053] The ViT image encoder generates a corresponding microscope image embedding by passing the input microscope image through the backbone network. Its operation is: first, the input microscope image is first subjected to image enhancement. Here, two strategies, random cropping and combined enhancement, are adopted. Random cropping refers to randomly selecting an area from the original image and cropping it to generate a new image. Combined enhancement refers to combining multiple different enhancement methods to produce richer and more diverse image data. These enhancement methods include contrast enhancement, color enhancement, scale transformation, rotation and flipping, and noise addition. After batch division and projection transformation, it enters an image embedding layer for feature extraction. The extracted features and the position embedding with initial values ​​are reorganized into a feature sequence. This feature sequence is then used as input and processed by a series of Transformer modules. Each Transformer module contains a normalization layer, a multi-head attention layer, and a multi-head perceptron layer to capture the global dependency between features and perform nonlinear transformations of features. After processing, an embedding representation corresponding to the input microscope image is generated for subsequent tasks.

[0054] The image-text alignment module aligns coarse-grained and fine-grained text similarity embeddings with random image similarity embeddings, while adding a scaling factor to transform the dimensions of text features and image embeddings. This results in text similarity embeddings and image similarity embeddings that are aligned in a high-dimensional semantic space. The goal is to maximize the semantic relevance between image and text, thereby obtaining an image-text pairing matrix. Finally, the resulting image-text pairing matrix is ​​used for various cross-modal applications.

[0055] The improved Loss function module is used to calculate the Loss function of the image-text feature matrix. Due to the addition of coarse-grained text information, the Loss function needs to be improved. The improvement is: total =L graph +λ((1-a)L coarse +aL fine ), where L total Represents the total loss, L coarse represents the coarse-grained text-image loss, L fine represents the fine-grained text-image loss, L graph Represents the real image-prediction image loss, L graph , L coarse , L fine It is calculated using the cross entropy function, where λ and a are hyperparameters;

[0056] The back-propagation module is used to optimize the model and optimize the model structure according to the result of the loss function. It calculates the gradient of the loss function and uses the gradient descent algorithm to update the model parameters in the direction of the gradient, so that the output of the model is closer to the true label, that is, the correct image-text pairing is closer, and the incorrect pairing is further away. Finally, the classification result is obtained through the classification head.

[0057] Specifically, if Figure 4 As shown, the multimodal alignment module learns useful knowledge from the bone marrow cell dataset and coarse and fine-grained text features to train the improved PMC-CLIP model, including the following steps:

[0058] 1) Set the number of training times, initial learning rate and batch size, and start training;

[0059] 2) The images segmented by the improved SAM model are used as the input image set, and the coarse-grained text features are used as the input text feature set. A batch-sized image subset A is randomly selected from the image set, and the corresponding text feature subset B is selected from the text feature set corresponding to the image subset A. Subsets A and B are aligned one-to-one in the high-dimensional semantic space, with one as the horizontal axis and the other as the vertical axis.

[0060] 3) Calculate the cross entropy loss so that the difference between the corresponding image and text features is as small as possible, thereby achieving better image-text alignment. At the same time, for the remaining image and text features, the calculated loss is as large as possible.

[0061] 4) Similar to the coarse-grained text, the same alignment and loss calculation steps are performed on the fine-grained text to ensure semantic consistency and alignment between the image and the fine-grained text;

[0062] 5) The improved PMC-CLIP model calculates the loss and back-propagates the updated weights to update the model weights and parameters to minimize the loss function; where Loss is expressed as:

[0063]

[0064] L coarse =-(L sup (v,l)+L sup (l,v)))

[0065] L fine =-(L sup (v,l)+L sup (l,v)))

[0066] L total =L graph +λ((1-a)L coarse +aL fine )

[0067] In the above formula, L total Represents the total loss, L graph represents the real image-prediction image loss, L ce represents the cross entropy loss, L sup Represents the loss of text features-image features, L coarse represents the coarse-grained text-image loss, L fine represents the fine-grained text-image loss; in L graph In the example, N represents the total number of samples, y i Represents the label information x i One-hot encoding, C(x i ) represents the prediction result, through L graph To calculate the image loss; in L sup , L coarse , L fine , in, L sup Represents the loss of text features-image features, L coarse represents the coarse-grained text-image loss, L fine represents the fine-grained text-image loss, i is the index, m and n represent the embedded features, represents the transposed feature of m embeddings at the ith index, Indicates the transposition of the i-th index n embedding feature; in L ce In, p i Represents the predicted probability of the i-th cell; in L sup In (m,n), P n (i) represents the positive sample set of the i-th batch, n represents the embedded features, A n (i) represents the negative sample set of the i-th batch, |P(i)| represents the total number of positive samples in the i-th batch, Represents one of the samples in a positive sample set, Represents one of the samples in a negative sample set, τ represents the temperature parameter in softmax, this parameter will be trained during the training process, through L sup Can define L coarse and L fine ; in L coarse and L fine In the example, l and v represent the text feature sample set and image feature sample set respectively, P l (i) represents the positive sample set of the i-th batch of text features, P v (i) represents the positive sample set of the i-th batch of image features, P(i) represents the positive sample set of the i-th batch, A l (i) represents the negative sample set of the i-th batch of text features, A v (i) represents the negative sample set of the i-th batch of image features, and |P l (i)|=|P v (i)|=|P(i)|; During the optimization process, the image features and text features belonging to the same category are placed in P l (i) and P v In (i), put the out-of-class features into A v (i) and A l (i) In; at the same time L coarse and L fine It will make the features of samples of the same type closer and the features of samples of different types farther apart; L total The λ in is a hyperparameter used for balanced alignment; a is used to control the contribution rate of coarse-grained text features and fine-grained text features, which is also a hyperparameter.

[0068] The above-described embodiments are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, any changes made based on the shape and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. A bone marrow cell recognition system based on multimodal enhancement, characterized in that: include: The cell image segmentation module is an improved SAM model for segmenting bone marrow cell microscopy image data. This module combines the SAM model and the Adapter block. By embedding the ViT block of the SAM model into the Adapter block, it extracts and adjusts the features of the image data. This module freezes the other parameters of the SAM model and only updates the Adapter block. It also adds segmentation hints and mask processing to improve the accuracy and robustness of the model for bone marrow cell microscopy image data, thereby obtaining an embedded representation of the segmented cell image. The label text transformation module is used to convert traditional label text into coarse-grained and fine-grained text feature representations. Its operation is as follows: first, coarse-grained text information is constructed using a text generator. A text generator is a program that generates text and can generate corresponding text according to the established content. The label text is segmented and semantically analyzed to extract coarse-grained semantic information to form a coarse-grained text feature set. Then, fine-grained text information is constructed using natural language processing or manual processing. The fine-grained text information provides a detailed text description of the cell image based on the characteristics of the cell image to form a fine-grained text feature set. Then, the coarse-grained text information and the fine-grained text information are transmitted to the BERT text encoder, and the coarse-grained text information and the fine-grained text information are mapped to a low-dimensional semantic space, thereby converting them into coarse-grained text features and fine-grained text features. The multimodal alignment module uses an improved PMC-CLIP model to achieve image-text alignment of the coarse- and fine-grained text features output by the label text transformation module. The improved PMC-CLIP model adds coarse- and fine-grained text features to the PMC-CLIP model. The cross-entropy loss is calculated and normalized using the coarse- and fine-grained text features of the cell, and the cross-entropy loss is used to backpropagate the loss to guide the model's use of fine- and coarse-grained text information. Finally, the classification result is output through the classification head.

2. The bone marrow cell identification system based on multimodal enhancement according to claim 1, characterized in that: The cell image segmentation module reads a microscope image from the bone marrow cell microscope image data and then performs the following process: first, the original image is randomly cropped into small 32*32 blocks, then segmentation points and segmentation lines are obtained on the image according to the segmentation prompts, the SAM model is used to segment the cells and background images, and then the parameters of the Adapter block are adjusted and the loss function is optimized to remove noise points and incomplete cells. The remaining large outlines are cell outlines. All the segmented cell images of the microscope image are combined into a segmented cell image dataset. All microscope images are generated again according to the above process to obtain a segmented bone marrow cell dataset.

3. The bone marrow cell identification system based on multimodal enhancement according to claim 2, characterized in that: The improved SAM model includes an image encoder, a prompt encoder, a mask encoder, a mask decoder and an adapter block; the image encoder refers to the input microscope image being convolved and rearranged by 3×3, and then the rearranged result and position embedding are input into the Transformer block to generate an image embedding, and finally the image embedding is convolved and added with the result output by the mask encoder to form a new image embedding, and then the box segmentation prompt and point segmentation prompt input by the user are input into the prompt encoder to generate a prompt embedding, and then the prompt embedding and the new image embedding are input into the mask decoder to generate a prediction; the adapter block is composed of a high-order fully connected layer, a RELU activation function layer and a low-order fully connected layer. Since the adapter block is added to each layer for parameter adjustment, the SAM model will better segment the image; the mask encoder includes a convolution layer, a normalization layer and a GELU activation function layer for generating a mask embedding to provide more information for the image encoder; the prompt encoder includes a point prompt encoder and a box prompt encoder. The point prompt encoder generates a point embedding by adding the point segmentation prompt provided by the user through coordinate normalization and pixel label encoding, and then the pixel label is The box hint encoder generates a box embedding by weighted labeling and then normalizing the box segmentation hint provided by the user, and finally adds the box embedding to the point label embedding. The mask substring and confidence substring with the initial value are then spliced ​​together to predict the image. The spliced ​​result passes through the self-attention layer and the normalization layer and then enters the mask decoder; the mask decoder includes a cross attention layer, a normalization layer, a multi-layer perceptron layer and a transposed convolution layer. The multi-layer perceptron layer consists of an input layer, a hidden layer and an output layer. The number of neural nodes in the hidden layer is 1024. The mask decoder performs the following operations: First, the provided After the results of the hint encoder and the image encoder pass through the cross-attention layer, they learn each other's features and are then standardized. Due to the different dimensions, the result of the hint encoder needs to go through a multi-layer perceptron layer, and then enter a cross-attention layer to learn each other's information. After that, the mask substring and confidence substring are taken out from the result of the hint encoder, and then enter a three-layer multi-layer perceptron layer respectively. The result of the mask substring needs to be multiplied with the result of the transposed convolution of the image encoder, so that the confidence prediction and mask prediction are obtained, so as to perform the next back propagation to optimize the prediction result and finally obtain the segmented image embedding.

4. The bone marrow cell identification system based on multimodal enhancement according to claim 3, characterized in that: The label text transformation module performs the following operations: 1) Read the source image and corresponding label information from the bone marrow cell dataset; 2) Generate detailed text descriptions of cells based on the image content of the source image and professional cell description knowledge; 3) Inputting the generated coarse-grained and fine-grained text information into the BERT text encoder, which includes text preprocessing and Transformer encoder; The text preprocessing is to first tokenize the sentences, then substring the sentences and input them into the embedding layer to generate embeddings, which are then added to the position embedding and segment embedding and rearranged before being input into the Transformer encoder. Finally, the output results are rearranged and normalized to obtain a unified dimension, ultimately obtaining coarse and fine-grained text feature representations. The Transformer encoder includes a normalization layer, a multi-head attention layer, and a multi-layer perceptron layer, which are used to map text information into a low-dimensional semantic space.

5. The bone marrow cell identification system based on multimodal enhancement according to claim 4, characterized in that: The multimodal alignment module is an improved PMC-CLIP model, which includes a ViT image encoder, an image-text alignment module, an improved Loss function module, and a back-propagation module, wherein: The ViT image encoder generates a corresponding microscope image embedding by passing the input microscope image through the backbone network. Its operation is: first, the input microscope image is first subjected to image enhancement. Here, two strategies, random cropping and combined enhancement, are adopted. Random cropping refers to randomly selecting an area from the original image and cropping it to generate a new image. Combined enhancement refers to combining multiple different enhancement methods to produce richer and more diverse image data. These enhancement methods include contrast enhancement, color enhancement, scale transformation, rotation and flipping, and noise addition. After batch division and projection transformation, it enters an image embedding layer for feature extraction. The extracted features and the position embedding with initial values ​​are reorganized into a feature sequence. This feature sequence is then used as input and processed by a series of Transformer modules. Each Transformer module contains a normalization layer, a multi-head attention layer, and a multi-head perceptron layer to capture the global dependency between features and perform nonlinear transformations of features. After processing, an embedding representation corresponding to the input microscope image is generated for subsequent tasks. The image-text alignment module aligns coarse-grained and fine-grained text similarity embeddings with random image similarity embeddings, while adding a scaling factor to transform the dimensions of text features and image embeddings. This results in text similarity embeddings and image similarity embeddings that are aligned in a high-dimensional semantic space. The goal is to maximize the semantic relevance between image and text, thereby obtaining an image-text pairing matrix. Finally, the resulting image-text pairing matrix is ​​used for various cross-modal applications. The improved Loss function module is used to calculate the Loss function of the image-text feature matrix. Due to the addition of coarse-grained text information, the Loss function needs to be improved. The improvement is: total =L graph +λ((1-a)L coarse +aL fine ), where L total Represents the total loss, L coarse represents the coarse-grained text-image loss, L fine represents the fine-grained text-image loss, L graph Represents the real image-prediction image loss, L graph 、L coarse 、L fine It is calculated using the cross entropy function, where λ and a are hyperparameters; The back-propagation module is used to optimize the model and optimize the model structure according to the result of the loss function. It calculates the gradient of the loss function and uses the gradient descent algorithm to update the model parameters in the direction of the gradient, so that the output of the model is closer to the true label, that is, the correct image-text pairing is closer, and the incorrect pairing is further away. Finally, the classification result is obtained through the classification head.

6. The bone marrow cell identification system based on multimodal enhancement according to claim 5, characterized in that: The multimodal alignment module learns useful knowledge from the bone marrow cell dataset and coarse and fine-grained text features to train the improved PMC-CLIP model, including the following steps: 1) Set the number of training times, initial learning rate and batch size, and start training; 2) The images segmented by the improved SAM model are used as the input image set, and the coarse-grained text features are used as the input text feature set. A batch-sized image subset A is randomly selected from the image set, and the corresponding text feature subset B is selected from the text feature set corresponding to the image subset A. Subsets A and B are aligned one-to-one in the high-dimensional semantic space, with one as the horizontal axis and the other as the vertical axis. 3) Calculate the cross entropy loss so that the difference between the corresponding image and text features is as small as possible, thereby achieving better image-text alignment. At the same time, for the remaining image and text features, the calculated loss is as large as possible. 4) Similar to the coarse-grained text, the same alignment and loss calculation steps are performed on the fine-grained text to ensure semantic consistency and alignment between the image and the fine-grained text; 5) The improved PMC-CLIP model calculates the loss and back-propagates the updated weights to update the model weights and parameters to minimize the loss function; where Loss is expressed as: L coarse =-(L sup (v,l)+L sup (l,v))) L fine =-(L sup (v,l)+L sup (l,v))) L total =L graph +λ((1-a)L coarse +aL fine ) In the above formula, L total Represents the total loss, L graph represents the real image-prediction image loss, L ce represents the cross entropy loss, L sup Represents the loss of text features-image features, L coarse represents the coarse-grained text-image loss, L fine represents the fine-grained text-image loss; in L graph In the example, N represents the total number of samples, y i Represents the label information x i One-hot encoding, C(x i ) represents the prediction result, through L graph To calculate the image loss; in L sup 、L coarse 、L fine , in, L sup Represents the loss of text features-image features, L coarse represents the coarse-grained text-image loss, L fine represents the fine-grained text-image loss, i is the index, m and n represent the embedded features, represents the transposed feature of m embeddings at the ith index, Indicates the transposition of the i-th index n embedding feature; in L ce In, p i Represents the predicted probability of the i-th cell; in L sup In (m,n), P n (i) represents the positive sample set of the i-th batch, n represents the embedded features, A n (i) represents the negative sample set of the i-th batch, |P(i)| represents the total number of positive samples in the i-th batch, Represents one of the samples in a positive sample set, Represents one of the samples in a negative sample set, τ represents the temperature parameter in softmax, this parameter will be trained during the training process, through L sup Can define L coarse and L fine ; in L coarse and L fine In the example, l and v represent the text feature sample set and image feature sample set respectively, P l (i) represents the positive sample set of the i-th batch of text features, P v (i) represents the positive sample set of the i-th batch of image features, P(i) represents the positive sample set of the i-th batch, A l (i) represents the negative sample set of the i-th batch of text features, A v (i) represents the negative sample set of the i-th batch of image features, and |P l (i)|=|P v (i)|=|P(i)|; During the optimization process, the image features and text features belonging to the same category are placed in P l (i) and P v In (i), the out-of-class features are placed in A v (i) and A l (i) In; at the same time L coarse and L fine It will make the features of samples of the same type closer and the features of samples of different types farther apart; L total The λ in is a hyperparameter used for balanced alignment; a is used to control the contribution rate of coarse-grained text features and fine-grained text features, which is also a hyperparameter.

Images