A temporal cell counting method based on bidirectional classification long short-term memory network
Through bidirectional classification of long and short-term memory networks, different state characteristics in cell images are identified and processed, and time sequence information is combined, the problem of insufficient cell count accuracy in the prior art is solved, significantly improving the accuracy of cell count.
Patent Information
- Application Number
- CN202411313178.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-20
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-09-20
AI Technical Summary
Existing deep learning-based cell counting techniques have insufficient in processing different cell state and timing information, resulting in the impact of counting accuracy.
Bidirectional classification long and short-term memory network (Bi-CLSTM) is used to introduce classification phylums and feature learning units to identify and learn the division state and stable state characteristics in cell images, and to predict cell counts based on forward and backward timing information. and status.
It improves the adaptability and understanding of cell morphological changes and dynamic processes, improves the accuracy of cell counts and the overall performance of the model.
Smart Images

Figure CN118823028B_ABST
Abstract
Description
Technical Field
[0001] The invention belongs to the field of artificial intelligence, and in particular relates to a time series cell counting method based on a two-way classification long short-term memory network. Background Art
[0002] Cell counting is an important means of evaluating cell quality. With the development of technology, traditional manual counting methods have gradually been replaced by automated computer vision algorithms, which use artificial intelligence, especially deep learning technology, to improve the efficiency and accuracy of cell counting. Automatic calculation and analysis through deep learning can greatly improve the efficiency of cell counting.
[0003] Automatic cell counting methods based on deep learning overcome the subjective, time-consuming and laborious shortcomings of traditional morphological scoring. They have the advantages of high accuracy, objectivity and high efficiency, and have gradually become an important means of medical image analysis. At present, automatic cell counting technology based on deep learning has been applied in many fields, including but not limited to oncology, stem cell research and drug screening. These technologies usually use convolutional neural networks (CNNs) to identify and distinguish cells, and recurrent neural networks (RNNs) to process temporal information in image sequences. These models can provide higher accuracy and less manual intervention than traditional methods.
[0004] Although existing deep learning-based cell counting techniques have made significant progress, they still face some challenges in practical applications:
[0005] 1. Cells have different morphological characteristics in different physiological states, such as stable state and division state. Existing algorithms often use a unified network structure to process cells in all states, which may lead to insufficient sensitivity to cell morphological changes and affect the accuracy of counting.
[0006] 2. Cell images usually exist in the form of sequences, and there is a temporal relationship between these images. If the algorithm only focuses on a single image and ignores this temporal information, it may lead to incomplete capture of the dynamic changes of cells, thereby reducing the accuracy of counting. Summary of the invention
[0007] The present invention proposes a time-series cell counting method based on a bidirectional classification long short-term memory network, which solves the problems that the existing methods are insufficiently adaptable to cell morphological changes under different states and ignore the time-series information in the cell image sequence.
[0008] In order to solve the above technical problems, the present invention provides a time-series cell counting method based on a bidirectional classification long short-term memory network, comprising the following steps:
[0009] Step S1: Using the cell dual-state counting unit CLSTM to replace the long short-term memory unit LSTM in the bidirectional long short-term memory network Bi-LSTM, and constructing a bidirectional classification long short-term memory network Bi-CLSTM;
[0010] The CLSTM unit includes a classification gate and two groups of feature learning units, wherein the classification gate is used to identify the cell state in the cell image, and the two groups of feature learning units respectively learn the features of cell images in different states, wherein the cell state includes a division state and a stable state;
[0011] Step S2: acquiring multiple cell images at continuous time, annotating the number of cells and cell states in each cell image, and using the annotated cell images to train the Bi-CLSTM network;
[0012] Step S3: Input a set of cell image sequences to be identified into the trained Bi-CLSTM network to obtain the cell state and cell number at each time step of the cell image sequence.
[0013] Preferably, the Bi-CLSTM network includes a forward CLSTM layer, a backward CLSTM layer and an output layer, wherein the forward CLSTM layer and the backward CLSTM layer include multiple CLSTM units, the forward CLSTM layer and the backward CLSTM layer respectively acquire the timing information in the cell image sequence from two different time directions, and the output layer merges the timing information in the two directions at each time step of the cell image sequence, and uses the merged timing information to predict the cell state and cell number of the cell image at that time step.
[0014] Preferably, the feature learning unit includes a forget gate, an input gate and an output gate, each gate of each feature learning unit corresponds to a different weight, the classification gate outputs the probability that the cell image is in a split state, the weight of each gate is adjusted according to the output of the classification gate, and the outputs of the forget gate, input gate and output gate of the two feature learning units are combined to obtain the comprehensive output of each gate.
[0015] Preferably, the probability that the cell image output by the classification gate is in the division state The expression is:
[0016] ;
[0017] In the formula, Represents the Sigmoid activation function; is the classification gate network weight; for t -1 The feature vector output by the CLSTM unit at time; is the feature vector of the t-th cell image; are the learnable parameters of the classification gate.
[0018] Preferably, the expression of the comprehensive output of the forget gate is:
[0019] ;
[0020] In the formula, For the Gate of Oblivion; is the probability that the cells in the cell image are in the division state; , are the weights of the forget gates of the two groups of feature learning units; for t -1 The feature vector output by the CLSTM unit at time; For the t The feature vector of the cell image; is the learnable parameter of the forget gate;
[0021] The expression of the comprehensive output of the input gate is:
[0022] ;
[0023] In the formula, is the output vector of the input gate; , are the weights of the input gates of the two groups of feature learning units; is the learnable parameter of the input gate;
[0024] The expression of the comprehensive output of the output gate is:
[0025] ;
[0026] In the formula, is the output vector of the output gate; , are the weights of the output gates of the two groups of feature learning units; is the learnable parameter of the output gate.
[0027] Preferably, the CLSTM unit outputs comprehensive state information of cells in the cell image, and the expression of the comprehensive state information is:
[0028] ;
[0029] ;
[0030] In the formula, is the output vector of the CLSTM unit; is the memory vector of the current layer; is the memory vector of the previous layer; tanh() is the tanh activation function.
[0031] Preferably, the overall loss of the Bi-CLSTM network includes classification loss and counting loss, wherein the classification loss is the predicted probability value of the cell being in the division state. With the true value The difference between the count loss and the predicted cell number The actual number of cells The difference between.
[0032] Preferably, the overall loss is expressed as:
[0033] ;
[0034] ;
[0035] ;
[0036] In the formula, for overall loss; , They are the weights of classification loss and counting loss in the overall loss respectively; is the classification loss; is the counting loss; S is the number of input cell images;
[0037] Preferably, in step S3, preprocessing the input cell image sequence and extracting the feature vector of the preprocessed cell image sequence as the input of the Bi-CLSTM network comprises the following steps:
[0038] Step S31: Detect the cell area in the cell image by using an edge detection algorithm, and superimpose the edge-processed cell image and the original cell image as an output image;
[0039] Step S32: using a rectangular frame of the same size to crop all output images, and during cropping, the center point of the rectangular frame corresponds to the center point of the cell area;
[0040] Step S33: using an image feature extraction network to extract the spatial features of each cell image in the cropped cell image sequence to obtain multiple feature maps;
[0041] Step S34: downsampling the feature map to reduce the size of the feature map;
[0042] Step S35: Perform global average pooling on the downsampled feature map to convert the feature map into a feature vector.
[0043] Preferably, the expression for performing global average pooling on the downsampled feature map in step S35 is:
[0044] ;
[0045] In the formula, For the t The feature vector of the cell image; , are the height and width of the feature map respectively; is the first k On each channel The pixel position.
[0046] The benefits of the present invention include at least:
[0047] 1. The present invention constructs a bidirectional classification long short-term memory network Bi-CLSTM that can identify and distinguish between cell division state and stable state. The classification gate introduced is used to identify the cell state in the cell image, ensuring that cell images of different states can be correctly input into the corresponding feature learning unit, so that the network can perform special feature learning for cells of different states, thereby improving the adaptability to cell morphological changes;
[0048] 2. The bidirectional learning mechanism of the Bi-CLSTM network enables the model to simultaneously consider the previous and next information of cell images in the time series, improving the ability to understand the dynamic process of cells. At the same time, by combining the spatial and temporal characteristics of cell images, the accuracy of the model is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] Figure 1 A comparison of cells in a stable state and cells in a dividing state;
[0050] Figure 2 A schematic diagram of the structure of a bidirectional classification long short-term memory network Bi-CLSTM according to an embodiment of the present invention;
[0051] Figure 3 This is a schematic diagram of the structure of a cell dual-state counting unit CLSTM according to an embodiment of the present invention;
[0052] Figure 4 A schematic diagram of a method flow chart of an embodiment of the present invention;
[0053] Figure 5 Schematic diagram of the structure of the feature extraction network of an embodiment of the present invention. DETAILED DESCRIPTION
[0054] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.
[0055] like Figure 1 As shown in the figure, cells are often in a relatively stable state when not dividing, and in a relatively active state when dividing. There are obvious differences in the characteristics of cells in these two states. When stable, cells tend to be elliptical, and when dividing, cells are irregular in shape and more difficult to count. Existing methods often use the same network to count these two states, which may cause the output results to be affected by the difference between the two states. In addition, existing methods often count single cell images, but there is temporal correlation information between cell frame images. Ignoring this part of information may reduce the accuracy of cell counting.
[0056] In view of the above problems, an embodiment of the present invention integrates a classification task branch into a bidirectional long short-term memory network Bi-LSTM, and proposes a bidirectional classification long short-term memory network Bi-CLSTM.
[0057] like Figure 2 As shown in FIG. 1 , the Bi-CLSTM network includes a forward CLSTM layer and a backward CLSTM layer, each of which includes multiple CLSTM units. The forward CLSTM layer processes the input sequence from front to back. For each time step in the sequence, , the forward CLSTM layer receives the input of the current time step , and according to the hidden state of the previous time step , cell status and predicted probability To update the hidden state of the current time step , cell status and predicted probability ; The backward CLSTM layer processes the input sequence from back to front. For each time step in the sequence , the backward CLSTM layer receives the input of the current time step , and according to the hidden state of the next time step , cell status and predicted probability values To update the hidden state of the current time step , cell status and predicted probability At each time step, the Bi-CLSTM network transforms the forward and backward hidden states and , and Merge to form a comprehensive representation and obtain the feature vector for each time step and predicted probability For each time step , the merged feature vector and predicted probability The result is sent to a multi-layer perceptron (MLP) to predict the cell state and cell number of the cell image at that time step.
[0058] like Figure 3 As shown, unlike the traditional LSTM unit, the CLSTM unit of the present invention includes a classification gate and two groups of feature learning units, and the two groups of feature learning units respectively learn the features of the cell images in the split state and the stable state. Each group of feature learning units includes a forget gate, an input gate and an output gate, and each gate of each group of feature learning units corresponds to a different weight. The classification gate is composed of an MLP, which outputs the probability that the cells in the cell image are in the split state. The weight of each gate is dynamically adjusted according to the output of the classification gate. Finally, the outputs of the two forget gates, the two input gates and the two output gates are combined to obtain the comprehensive output of each gate.
[0059] Based on the constructed Bi-CLSTM network, the embodiment of the present invention proposes a time-series cell counting method based on a bidirectional classification long short-term memory network, such as Figure 4 As shown, the following steps are included:
[0060] Step S1: Construct a cell image dataset.
[0061] Step S11: Use a time-difference incubator to record the cell proliferation process, ensure that the incubator has appropriate temperature and humidity control, configure the microscope's magnification, light source, filter and other parameters to ensure that the cells can be clearly photographed. Take a cell sample every 15 minutes to obtain a cell image.
[0062] Step S12: Use the image annotation tool to manually circle each cell in each image, record the number of cells in each image, and save the annotation file.
[0063] Step S13: The characteristics of the cell division state are determined by cytology experts. These characteristics include cell morphological changes, cell tearing, etc. If the cells in the cell image are in a division state, the image is marked as 1, otherwise it is marked as 0.
[0064] Step S14: After all cell images are labeled, some images are extracted for manual inspection to correct erroneous labels and update the labeling files to ensure accurate labeling.
[0065] Step S15: Divide the labeled data set into a training set, a validation set, and a test set according to a 6:2:2 ratio.
[0066] Step S2: Cell image preprocessing. The purpose of cell image preprocessing is to reduce noise interference, highlight cell features, and standardize input. In the images collected by the optical microscope, the cells are concentrated inside the circular culture medium. In order to filter out the background noise information outside the culture dish, the image needs to be preprocessed to improve the generalization ability and accuracy of the model.
[0067] Step S21: Through the Canny edge detection algorithm, morphological operations such as corrosion and dilation, the contour search method detects the cell area in the cell image, highlights the edge features of the cell image, makes the cell image clearer in visualization and analysis, and superimposes the edge-processed cell image and the original image as input.
[0068] Step S22: Determine the position of the circular culture medium based on the detected cell edge information, and then use a fixed rectangular frame to crop the cell image. The center point of the cropped area corresponds to the center point of the detected cell area to ensure that the cell is in the center of the cropped cell image. The fixed rectangular frame is used to ensure that the cell culture dish areas in all input images have similar sizes and positions.
[0069] Step S3: Extraction of spatial features of cell images.
[0070] like Figure 5 As shown, the present invention uses the image feature extraction network ResNet50 to obtain the input S The spatial features of the continuous-time cell image are output S Features include the following steps:
[0071] Step S31: Input n A continuous cell image is taken from a cell and local features of different dimensions, such as edges, textures, etc., are extracted through multiple convolutional neural networks of different sizes. A convolution kernel of a specific size is slid on the input image and the dot product is calculated at each position to obtain a feature map.
[0072] Step S32: After each convolution operation, a nonlinear activation function is applied to the feature map, usually ReLU, so that the model can learn more complex features, and then the input feature map is directly added to the output feature map through jump connections. This structure effectively alleviates the problem of gradient vanishing in the network and makes the network easier to optimize.
[0073] Step S33: downsample the output feature map to reduce the size of the feature map while retaining important spatial features.
[0074] Step S34: After multiple convolution operations, the output feature map is globally averaged pooled to convert the feature map into a feature vector of fixed length:
[0075] ;
[0076] In the formula, is the feature vector of the t-th cell image; , are the height and width of the feature map respectively; is the kth channel in the feature map The pixel position.
[0077] Step S4: Use a bidirectional classification long short-term memory network Bi-CLSTM to identify the cell state and cell number at each time step of the cell image sequence.
[0078] Step S41: The input of the Bi-CLSTM network is S The feature vector corresponding to the cell image ,in Indicates t The Bi-CLSTM network includes multiple cell dual-state counting units CLSTM, each of which includes a classification gate, two forget gates, two input gates and two output gates.
[0079] The classification gate is used to determine whether there are dividing cells in the current image:
[0080] ;
[0081] In the formula, Represents the probability value of the cell in the division state output by the classification gate; Represents the Sigmoid activation function; is the classification gate network weight; for t -1 The feature vector output by the CLSTM unit at time; is the feature vector of the t-th cell image; are the learnable parameters of the classification gate.
[0082] The forget gate selects the cell state at the previous moment and determines how much of the cell state at the previous moment is retained to the current moment:
[0083] ;
[0084] In the formula, For the Gate of Oblivion; is the probability that the cells in the cell image are in the division state; , are the weights of the two forget gates respectively; for t -1 The feature vector output by the CLSTM unit at time; For the t The feature vector of the cell image; is the learnable parameter of the forget gate.
[0085] The input gate determines the input of the CLSTM unit at the current moment, that is, how much is saved to the unit state:
[0086] ;
[0087] In the formula, is the output vector of the input gate; , are the weights of the two input gates respectively; are the learnable parameters of the input gate.
[0088] The output gate controls the cell state, that is, how many are used as the output value of the CLSTM cell:
[0089] ;
[0090] In the formula, is the output vector of the output gate; , are the weights of the two output gates respectively; is the learnable parameter of the output gate.
[0091] The cell state at this time step is determined by the historical memory and the current input. is the memory vector of the current layer, is the memory vector of the previous layer:
[0092] ;
[0093] The final output of the CLSTM unit at this time step is:
[0094] ;
[0095] Where tanh() is the tanh activation function; Represents the output vector of the CLSTM unit.
[0096] Then the basic formula of the forward CLSTM unit is:
[0097] ;
[0098] is the feature vector output by the forward CLSTM unit; is the probability of the cell being in the division state output by the forward CLSTM unit.
[0099] The basic formula of the backward CLSTM unit is:
[0100] ;
[0101] is the feature vector output by the backward CLSTM unit; is the probability that the cell is in the division state output by the backward CLSTM unit.
[0102] Step S42: Connect the forward and backward outputs of each time step into a vector, and use the multi-layer perceptron MLP to predict the number of cells in the cell image at that moment. and the probability that the cell is in a dividing state , is a fusion function, which means merging two vectors into one vector:
[0103] ;
[0104] ;
[0105] Step S43: The embodiment of the present invention improves the accuracy of cell counting by considering whether the cells are in a stable or dividing state. Therefore, the network loss is divided into two parts, namely, the classification loss With counting loss The classification loss is the predicted probability value of the cell in the division state With the true value The difference between the count loss and the number of cells predicted by the model The actual number of cells The difference between The calculation of is as follows:
[0106] ;
[0107] ;
[0108] ;
[0109] in and is a hyperparameter that represents the weight of classification loss and counting loss in the overall loss.
[0110] The proposed method of sequential cell counting based on bidirectional classification long short-term memory network can simultaneously realize classification and regression tasks, and use the classification results to guide more accurate regression results. The probability of cells being in a division state is obtained by the classifier. , so as to effectively judge whether the input cell image is in a division state or a stable state, and then extract the feature vectors in the two states through fully connected layers with different weights. By combining the spatial and temporal features of the cell image, it is possible to effectively detect whether the cells in the time series image are in a division state, thereby more accurately predicting the number of cells.
[0111] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is more specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. As long as there is no contradiction in the combination of these technical features, they should be considered as the scope recorded in this specification.
[0112] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. A time-series cell counting method based on a bidirectional classification long short-term memory network, characterized in that: The following steps are involved: Step S1: Using the cell dual-state counting unit CLSTM to replace the long short-term memory unit LSTM in the bidirectional long short-term memory network Bi-LSTM, and constructing a bidirectional classification long short-term memory network Bi-CLSTM; The CLSTM unit includes a classification gate and two groups of feature learning units, wherein the classification gate is used to identify the cell state in the cell image, and the two groups of feature learning units respectively learn the features of cell images in different states, wherein the cell state includes a division state and a stable state; Step S2: acquiring multiple cell images at continuous time, annotating the number of cells and cell states in each cell image, and using the annotated cell images to train the Bi-CLSTM network; Step S3: input a set of cell image sequences to be identified into the trained Bi-CLSTM network to obtain the cell state and cell number at each time step of the cell image sequence; The Bi-CLSTM network includes a forward CLSTM layer, a backward CLSTM layer and an output layer, wherein the forward CLSTM layer and the backward CLSTM layer include a plurality of CLSTM units, the forward CLSTM layer and the backward CLSTM layer respectively acquire the time series information in the cell image sequence from two different time directions, and the output layer merges the time series information in the two directions at each time step of the cell image sequence, and uses the merged time series information to predict the cell state and cell number of the cell image at the time step; The feature learning unit includes a forget gate, an input gate and an output gate, each gate of each feature learning unit corresponds to a different weight, the classification gate outputs the probability that the cell image is in a split state, the weight of each gate is adjusted according to the output of the classification gate, and the outputs of the forget gate, input gate and output gate of the two feature learning units are combined to obtain a comprehensive output of each gate; The probability k that the cell image output by the classification gate is in a split state t The expression is: k t =σ(w k ·[h (t-1) ,x t ]+b k ); Where σ represents the Sigmoid activation function; w k is the classification gate network weight; h (t-1) is the feature vector output by the CLSTM unit at time t-1; x t is the feature vector of the t-th cell image; b k is the learnable parameter of the classification gate; The expression of the comprehensive output of the forget gate is: f t =σ(k t ·w f1 ·[h (t-1) ,x t ]+(1-k t )·w f2 ·[h (t-1) ,x t ]+b f ); Where, σ represents the Sigmoid activation function; f t is the forget gate; k t is the probability that the cells in the cell image are in the division state; w f1 、w f2 are the weights of the forget gates of the two groups of feature learning units; h (t-1) is the feature vector output by the CLSTM unit at time t-1; x t is the feature vector of the t-th cell image; b f is the learnable parameter of the forget gate; The expression of the comprehensive output of the input gate is: i t =σ(k t ·w i1 ·[h (t-1) ,x t ]+(1-k t )·w i2 ·[h (t-1) ,x t ]+b i ); In the formula, i t is the output vector of the input gate; w i1 、w i2 are the weights of the input gates of the two groups of feature learning units; b i is the learnable parameter of the input gate; The expression of the comprehensive output of the output gate is: o t =σ(k t ·w o1 ·[h (t-1) ,x t ]+(1-k t )·w o2 ·[h (t-1) ,x t ]+b o ); In the formula, o t is the output vector of the output gate; w o1 、w o2 are the weights of the output gates of the two groups of feature learning units; b o is the learnable parameter of the output gate; The CLSTM unit outputs comprehensive state information of cells in the cell image, and the expression of the comprehensive state information is: h t =o t ·tanh(c t ); c t =f t ·c t-1 +i t ·x t ; In the formula, h t is the feature vector output by the CLSTM unit at time t; c t is the memory vector of the current layer; c t-1 is the memory vector of the previous layer; tanh() is the tanh activation function.
2. The method for sequential cell counting based on a bidirectional classification long short-term memory network according to claim 1, characterized in that: The overall loss of the Bi-CLSTM network includes classification loss and counting loss. The classification loss is the predicted probability value K of the cell in the division state. t and the true value K t ', the count loss is the predicted cell number N t The actual number of cells N t '.
3. The method for sequential cell counting based on a bidirectional classification long short-term memory network according to claim 2, characterized in that: The expression of the overall loss is: L=γ k L k +g n L n ; Where L is the total loss; γ k , γ n are the weights of classification loss and counting loss in the overall loss respectively; L k is the classification loss; L n is the counting loss; S is the number of input cell images.
4. The method for sequential cell counting based on a bidirectional classification long short-term memory network according to claim 1, characterized in that: In step S3, the input cell image sequence is preprocessed, and the feature vector of the preprocessed cell image sequence is extracted as the input of the Bi-CLSTM network, which includes the following steps: Step S31: Detect the cell area in the cell image by using an edge detection algorithm, and superimpose the edge-processed cell image and the original cell image as an output image; Step S32: using a rectangular frame of the same size to crop all output images, and during cropping, the center point of the rectangular frame corresponds to the center point of the cell area; Step S33: using an image feature extraction network to extract the spatial features of each cell image in the cropped cell image sequence to obtain multiple feature maps; Step S34: downsampling the feature map to reduce the size of the feature map; Step S35: Perform global average pooling on the downsampled feature map to convert the feature map into a feature vector.
5. The method for sequential cell counting based on a bidirectional classification long short-term memory network according to claim 4, characterized in that: The expression for performing global average pooling on the downsampled feature map in step S35 is: In the formula, f t is the feature vector of the t-th cell image; H and W are the height and width of the feature map respectively; y i,j,k is the pixel at position (i, j) on the kth channel in the feature map.
