Cell state detection method, device and storage medium based on spatiotemporal feature fusion

Abstract

The present invention discloses a cell state detection method, device and storage medium based on spatiotemporal feature fusion, the method comprising the following steps: collecting multiple cell images within a set acquisition time period, using a target detection network to identify individual cells in each cell image, and extracting cell features of all individual cells; assigning the same time code to all cell features in each cell image according to the acquisition order of the cell images within the acquisition time period; inputting the cell features with the time code into a self-attention network Transformer, using the time code as the position code of the cell features in the Transformer network, and obtaining the predicted probability of the state of each cell image in the multiple cell images. The method of the present invention combines the temporal features and spatial features of cells, provides more comprehensive cell state information, and can more accurately predict the changes in the cell division state.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and specifically relates to a cell state detection method, device and storage medium based on spatiotemporal feature fusion. Background Art

[0002] Cell division refers to the process of proliferation of living cells, that is, one cell divides into two cells. The cell before division is called the mother cell, and the new cell formed after division is called the daughter cell. In some cases, cells divide abnormally, that is, the number of daughter cells obtained from one cell division is greater than two. Another case is reverse division, that is, two cells merge into one cell. Detecting the state of cell division is helpful to understand the laws of cell growth and reproduction.

[0003] The cell proliferation process can be recorded periodically by using a time-difference incubator. Cytologists can judge whether cell division is normal by observing the time-series images of cells and combining their own experience. With the development of artificial intelligence, it has become a feasible method to use image feature learning methods to judge the division state of cells. When using the image feature learning method, the position and number of cells in the image are first detected by the target detection network, and then the changes in the number of cells in the image at different times are used to judge whether the cells are abnormally dividing or reversely dividing.

[0004] However, the image processing models used in existing image feature learning methods often extract features from different spatial regions of a single image. However, the correlation between cell images at different times is equally important, because some cells may be formed by cell division at a previous time. If this part of information is ignored, the recognition accuracy of cell status may be reduced. Summary of the invention

[0005] The present invention proposes a cell state detection method, device and storage medium based on spatiotemporal feature fusion, which solves the problem that the existing image feature learning method ignores the temporal information between different cell images.

[0006] In order to solve the above technical problems, the present invention provides a cell state detection method based on spatiotemporal feature fusion, comprising the following steps:

[0007] Step S1: Collect multiple cell images within a set collection time period, use the target detection network to identify individual cells in each cell image, and extract the cell features of all individual cells;

[0008] Step S2: assigning the same time code to all cell features in each cell image according to the acquisition order of the cell images within the acquisition time period;

[0009] Step S3: Input the cell features with time coding into the self-attention network Transformer, use the time coding as the position coding of the cell features in the Transformer network, and obtain the predicted probability of the cell division state in the acquisition time period.

[0010] Preferably, the step S1 of extracting the cell features of all single cells comprises the following steps:

[0011] Step S11: extracting the features of each cell image to obtain several global feature maps;

[0012] Step S12: identifying individual cells in each global feature map through the target detection network, and extracting cell features of all individual cells;

[0013] Step S13: Fusing each of the cell features with the corresponding global features to obtain fused features of all individual cells.

[0014] Preferably, the step S12 of identifying a single cell in each global feature map by using a target detection network comprises the following steps:

[0015] Step S121: Divide the global feature map into a number of grids of equal size, and the target detection network calculates the confidence score of each grid containing the target, and predicts the position of the bounding box that may contain the cell;

[0016] Step S122: Delete the bounding boxes whose confidence scores are lower than the set threshold, randomly select two bounding boxes A and B from the remaining bounding boxes, calculate the intersection of two and two, and if the intersection of two and two exceeds the set intersection of two and two threshold, remove the bounding box with lower confidence in A and B, repeat the above operation, and output the remaining bounding boxes.

[0017] Preferably, step S13 uses a multi-layer perceptron to fuse each of the cell features with the corresponding global feature, and the expression of the fusion process is:

[0018] ;

[0019] In the formula, For the i Fusion features of individual cell characteristics; is the weight of the multi-layer perceptron; is the fusion function; For the i Cell characteristics; It is a global feature.

[0020] Preferably, in step S3, a Transformer network based on a multi-head attention mechanism is used to output the cell division state within the collection time period, comprising the following steps:

[0021] Step S31: Input all fusion features separately n Transformer networks with different weights, we get n For each head, calculate the attention weight of each head;

[0022] Step S32: concatenate the attention weights of all heads, perform linear transformation on the concatenated attention weights, and obtain a comprehensive feature representation;

[0023] Step S33: further process the feature representation and output a predicted value of the cell division state within the acquisition time period.

[0024] Preferably, step S31 includes the following steps:

[0025] Step S311: Input the fused features in each head into three linear layers with different weights to obtain an input vector for each head, which includes the query vector Q , key vector K Sum value vector V ;

[0026] Step S312: pad the input vector of each head to the same length;

[0027] Step S312: Calculate the attention weight of each head:

[0028] ;

[0029] In the formula, For the j head; is the self-attention function; , , Respectively j The query vector, key vector and value vector of each head; is the activation function; is the length of the key vector.

[0030] Preferably, step S32 inputs the concatenated attention weights into a linear layer to obtain a comprehensive feature representation, wherein the expression of the feature representation is:

[0031] ;

[0032] In the formula, It is a multi-head attention mechanism; is the fusion function; For the h head, h is the total number of heads; is the weight of the linear layer.

[0033] Preferably, the target detection network in step S1 is a convolutional neural network (CNN).

[0034] The present invention also provides an electronic device, comprising: a memory, a processor and a computer program, wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the above-mentioned cell state detection method based on spatiotemporal feature fusion.

[0035] The present invention further provides a computer-readable storage medium, in which a computer program is stored. The computer program is executed by a processor to implement the above-mentioned cell state detection method based on spatiotemporal feature fusion.

[0036] The benefits of the present invention include at least:

[0037] 1. The present invention extracts complete target-level features based on cell detection, avoiding the problem of target fragmentation during feature extraction, and combines the local features of cells with the global features of images. Through feature fusion, the network's complete and comprehensive understanding of cell status is enhanced;

[0038] 2. The present invention combines the temporal characteristics and spatial characteristics of cells to provide more comprehensive cell state information, which can more accurately predict the temporal dynamic division state changes of cells, such as the transition from normal division to abnormal division;

[0039] 3. When processing cell image sequences, using time encoding instead of the original Transformer network's position encoding can better simulate and analyze patterns and trends in time series data, simplifying the model design and training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention;

[0041] Figure 2 Schematic diagram of a method for fusing global information of a cell image and cell characteristics of a single cell in an embodiment of the present invention;

[0042] Figure 3 Schematic diagram of inputting cell features with time coding into a Transformer network in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0044] Embodiment 1

[0045] like Figure 1 As shown, an embodiment of the present invention provides a cell state detection method based on spatiotemporal feature fusion, comprising the following steps:

[0046] Step S1: Construct a dataset of cell images.

[0047] Step S11: Use a time difference incubator to take a picture of a cell sample every 15 minutes to record the cell proliferation process. When taking pictures, control the temperature and humidity in the incubator to have appropriate parameters, and configure the magnification, light source, filter, etc. of the microscope to ensure that the cells can be clearly photographed.

[0048] Step S12: Label the collected cell images using the image labeling software LabelImg. For each cell in the image, use the annotation tool to draw a rectangular bounding box to surround it. This bounding box should fit the edge of the cell as closely as possible. After labeling, the coordinates, width, and height information of the cell center point are obtained, and an ID value is assigned to the cells in each image. For example, if image a contains two cells, the two cells are assigned ID values ​​a1 and a2.

[0049] Step S13: Record the state of each cell, which includes the undivided state, normal division state, abnormal division state and reverse division state. Use four 0 / 1 numbers to record each cell state. For example, if the cell with ID a1 in cell image a is in the undivided state, a1 is marked as (1, 0, 0, 0); if a1 is in the normal division state, it is marked as (0, 1, 0, 0); if a1 is in the abnormal division state, it is marked as (0, 0, 1, 0); if a1 is in the reverse division state, it is marked as (0, 0, 0, 1).

[0050] Step S14: After all cell images are labeled, some cell images are extracted for manual inspection to correct erroneous labels and update the labeling file to ensure accurate labeling.

[0051] Step S15: Divide the labeled data set into a training set, a validation set, and a test set according to a 6:2:2 ratio.

[0052] Step S2: Cell image preprocessing.

[0053] The purpose of cell image preprocessing is to reduce noise interference in the image, highlight cell features and standardize the input. In the image collected by the optical microscope, the cells are concentrated inside the circular culture medium. In order to filter out the background noise information outside the culture dish, the image needs to be preprocessed to improve the generalization ability and accuracy of the model, including the following steps:

[0054] Step S21: Through the Canny edge detection algorithm, morphological operations such as corrosion and dilation, the contour search method detects the cell area in the cell image, highlights the edge features of the cell image, makes the image clearer in visualization and analysis, and superimposes the edge-processed image and the original image as input.

[0055] Step S22: Determine the position of the circular culture medium based on the detected cell edge information, and then use a fixed rectangular frame to crop the cell image, and the center point of the cropped area corresponds to the center point of the detected cell area to ensure that the cell is centered in the cropped image, so that the cell culture dish areas in all input images have similar sizes and positions.

[0056] Step S3: Cell target detection.

[0057] Step S31: Use convolutional neural network CNN to input n The cell images are extracted and output n The feature map corresponding to the cell image. CNN includes multiple convolutional layers, pooling layers, and activation functions, which are used to extract useful feature information from the image.

[0058] Step S32: Divide the feature map into The confidence score of each grid containing the target is calculated. At the same time, the predicted bounding box coordinates are offset from the grid cell , , , Convert to coordinates relative to the original image to get the center point coordinates of the predicted bounding box , And width and height information , :

[0059] ;

[0060] ;

[0061] ;

[0062] ;

[0063] In the formula, , are the horizontal and vertical coordinates of the upper left corner of the grid cell respectively; , are the length and width of the grid cell respectively; is the activation function.

[0064] Step S33: Use the non-maximum suppression method to remove duplicate detection boxes. First, remove the bounding boxes whose confidence scores are lower than the set confidence threshold, and then perform non-maximum suppression on the remaining bounding boxes. For two bounding boxes A and B, if their intersection-over-union ratio (IOU) exceeds the set threshold, which is set to 0.5 in this embodiment of the present invention, remove the bounding boxes with lower confidence in bounding boxes A and B, repeat the above operation, and output the remaining bounding boxes.

[0065] Step S34: Calculate the Focaler-IoU between the predicted target frame and the true target frame, which is expressed as:

[0066] ;

[0067] In the formula, d represents the lower threshold and u represents the upper threshold. That is, when the intersection-over-union ratio IoU is less than a lower threshold d, Focaler-IoU is 0; when IoU is greater than an upper threshold u, Focaler-IoU is 1; when IoU is between d and u, Focaler-IoU is a function that increases linearly according to the IoU value.

[0068] During training, the Focaler-IoU is multiplied by -1 as the regression loss of the bounding box location.

[0069] Step S4: Cell space feature extraction.

[0070] In order to make the cell features extracted by the network more representative, the embodiment of the present invention extracts the features of the entire cell image as global features, and extracts the features of a single cell as local features, and then fuses the local features of each cell with the global features of the cell image. Figure 2 As shown, the specific process is as follows:

[0071] Step S41: Extract the feature map of the entire cell image through the ResNet network, and then obtain the global features of the cell image through the maximum pooling operation .

[0072] Step S42: extract the features of the cell position detected by the target detection network to obtain a feature map of a single cell, perform a maximum pooling operation on the cell feature map to obtain the cell feature, and then i The cell characteristics corresponding to each cell are expressed as .

[0073] Step S43: Combine the global features of the cell with the cell features and input them into the The fusion features are obtained from the multi-layer perceptron , while combining the position information of the cell's bounding box as a positional code.

[0074] Step S5: Correlation of spatiotemporal characteristics of cells.

[0075] like Figure 3 As shown, the embodiment of the present invention uses a multi-head attention mechanism to associate the temporal features of cells. This structure allows each attention mechanism to optimize different feature parts of each cell, thereby balancing the deviations that may be generated by the same attention mechanism and allowing the meaning of words to have multiple expressions. The following steps are included:

[0076] Step S51: Use digital to perform time encoding on the fusion features obtained in step S4: n All cells in the first image of the cell image are coded as 1, all cells in the second image of the cell image are coded as 2, and so on. n All cells in a cell image are encoded as n , realizing the combination of time coding and cell characteristics.

[0077] Step S52: The input several fusion features are respectively n Transformer networks with different weights, we get n For each head, the attention weight is calculated.

[0078] Assume j The head includes n Fusion features These fusion features are weighted by three factors: , , The linear layer of , gets the input vector. The input vector includes the query vector , key vector and the value vector ,in , , .

[0079] The attention weight between fused features is calculated by the following formula, where Represents the key vector dimensions:

[0080] ;

[0081] In the formula, For the jhead; is the self-attention function; , , Respectively j The query vector, key vector and value vector of each head; is the activation function.

[0082] Step S53: Different heads contain different numbers of cells. In order to associate heads containing different numbers of cells, it is necessary to ensure that the number of input vectors of each head is consistent. This can be achieved using the pad function of the nn.functional module in PyTorch, which allows each input to be padded to the same length, where the padded sequence is processed using a mask.

[0083] Step S54: Merge the attention weights of each head into a vector with weights The linear layer gets the output, and the first value of the output is used for the classification task to learn the model.

[0084] ;

[0085] In the formula, It is a multi-head attention mechanism; is the fusion function; For the h head, h is the total number of heads; is the weight of the linear layer.

[0086] Step S55: Add the fused features with time coding to the Transformer network for cell feature association, and the Transformer network outputs the probability values ​​(q1, q2, q3, q4) of the four cell states of each cell image.

[0087] Step S56: Subtract the predicted probability value (q1, q2, q3, q4) from the actual cell state value (q1, q2, q3, q4) item by item to get the loss Constrained Network Learning:

[0088] .

[0089] When the existing image feature learning method is applied to the task of cell state detection, there is a problem of loss of cell structure information due to inappropriate division method. The existing image feature association network often divides the image neatly into multiple square areas of the same size for association. However, this may cause some cells to be divided into multiple different areas, making it difficult for the network to capture complete cell information, which may lead to the loss of important structural information. The method of the present invention extracts complete target-level features on the basis of cell detection, avoids the problem of target fragmentation during feature extraction, and combines the local features of the cell with the global features of the image, and enhances the network's complete and comprehensive understanding of the cell state through feature fusion. When realizing the association of the spatiotemporal features of the cell, a multi-head attention mechanism is used to associate the cell features. This structural design allows the model to focus on different parts of multiple cell features at the same time, thereby realizing information interaction between cells and better modeling the kinship relationship during cell division. Through network training, end-to-end prediction of cell state is achieved.

[0090] In addition, existing image feature association networks often only extract features from different parts of a single cell image, ignoring the temporal correlation information between different cell images. The temporal correlation between different cell images is equally important, because some cells are formed by cell division at a previous time. Ignoring this part of information may lead to a decrease in the accuracy of cell state recognition. The method of the present invention effectively integrates the time series information of cell states by replacing position coding with time coding in Transformer, enabling the prediction network to capture dynamic changes during cell division and reverse division. By using a multi-head attention mechanism, the prediction network can focus on different aspects of cell features at the same time, enhancing the network's ability to characterize cell states, thereby improving the accuracy of state recognition.

[0091] Embodiment 2

[0092] An embodiment of the present invention provides an electronic device, including: a memory, a processor and a computer program, wherein the computer program is stored in the memory and is configured to be executed by the processor to implement the above-mentioned cell state detection method based on spatiotemporal feature fusion.

[0093] Embodiment 3

[0094] An embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. The computer program is executed by a processor to implement the above-mentioned cell state detection method based on spatiotemporal feature fusion.

[0095] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. As long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification.

[0096] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A cell state detection method based on spatiotemporal feature fusion, characterized in that: The following steps are involved: Step S1: collecting multiple cell images within a set collection time period, extracting features of each cell image, obtaining several global feature maps, identifying individual cells in each global feature map through a target detection network, extracting cell features of all individual cells, fusing each cell feature with the corresponding global feature, and obtaining fused features of all individual cells; Step S2: assigning the same time code to the fusion features of all individual cells in each cell image according to the acquisition order of the cell images within the acquisition time period; Step S3: Input the fused features of all individual cells with time codes into the self-attention network Transformer, use the time codes as the position codes of the fused features in the Transformer network, and obtain the predicted probability of the cell division state during the acquisition time period.

2. A cell state detection method based on spatiotemporal feature fusion according to claim 1, characterized in that: The step S1 of identifying individual cells in each global feature map by the target detection network includes the following steps: Step S11: Divide the global feature map into a number of grids of equal size, and the target detection network calculates the confidence score of each grid containing the target, and predicts the position of the bounding box that may contain the cell; Step S12: Delete the bounding boxes whose confidence scores are lower than the set threshold, randomly select two bounding boxes A and B from the remaining bounding boxes, calculate the intersection of two and three boxes A and B, and if the intersection of two and three boxes exceeds the set intersection of two and three boxes threshold, remove the bounding box with lower confidence in A and B, repeat the above operation, and output the remaining bounding boxes.

3. The cell state detection method based on spatiotemporal feature fusion according to claim 1, characterized in that: Step S1 uses a multi-layer perceptron to fuse each of the cell features with the corresponding global features. The expression of the fusion process is: s i =W·Concate(f i ,g); In the formula, s i is the fusion feature of the i-th cell feature; W is the weight of the multi-layer perceptron; Concate() is the fusion function; f i is the i-th cell feature; g is the global feature.

4. The cell state detection method based on spatiotemporal feature fusion according to claim 1, characterized in that: In step S3, a Transformer network based on a multi-head attention mechanism is used to output the cell division state within the collection time period, including the following steps: Step S31: Input all fused features into n Transformer networks with different weights to obtain n heads, and calculate the attention weight of each head respectively; Step S32: concatenate the attention weights of all heads, perform linear transformation on the concatenated attention weights, and obtain a comprehensive feature representation; Step S33: further process the feature representation and output a predicted value of the cell division state within the acquisition time period.

5. The cell state detection method based on spatiotemporal feature fusion according to claim 4, characterized in that: Step S31 includes the following steps: Step S311: Input the fused features in each head into three linear layers with different weights to obtain an input vector for each head, where the input vector includes a query vector Q, a key vector K, and a value vector V; Step S312: pad the input vector of each head to the same length; Step S312: Calculate the attention weight of each head: In the formula, head j is the jth head; Attention() is the self-attention function; Q j , K j 、V j are the query vector, key vector and value vector of the jth head respectively; softmax() is the activation function; d k is the length of the key vector.

6. The cell state detection method based on spatiotemporal feature fusion according to claim 5, characterized in that: Step S32 inputs the concatenated attention weights into the linear layer to obtain a comprehensive feature representation, the expression of which is: MultiHead(Q,K,V)=Concate(head1,...,head h )W o ; In the formula, MultiHead() is a multi-head attention mechanism; Concate() is a fusion function; head h is the hth head, h is the total number of heads; W o is the weight of the linear layer.

7. The cell state detection method based on spatiotemporal feature fusion according to claim 1, characterized in that: The target detection network in step S1 is a convolutional neural network CNN.

8. An electronic device, comprising: A memory, a processor and a computer program, characterized in that: the computer program is stored in the memory and is configured to be executed by the processor to implement a cell state detection method based on spatiotemporal feature fusion as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement a cell state detection method based on spatiotemporal feature fusion as described in any one of claims 1 to 7.

Images