A method for detecting the cell growth state based on image segmentation

Abstract

The present invention provides a method for detecting the cell growth state based on image segmentation, belonging to the technical field of cell image processing and segmentation. Cell images are collected in real time, and the collected cell images are input into a hierarchical division feature encoder to obtain coarse-grained features of cell blocks and fine-grained features of cell boundaries; the coarse-grained features of cell blocks and the fine-grained features of cell boundaries are input into a cascaded feature fusion network to obtain deeply fused features; the deeply fused features are input into a decoder to obtain the instance segmentation result of the cell image; the number and survival rate of target cells are calculated according to the instance segmentation result of the cell image. The newly designed image extraction and segmentation network of the present invention, combined with a dual-task collaborative training strategy, combines the cell image instance segmentation task with the cell image semantic segmentation task, realizes the real-time detection of the cell growth state, and avoids the time-consuming and low efficiency of manual observation in traditional methods, as well as the contamination and damage that may be caused by manual sampling.

Description

Technical Field

[0001] The present invention belongs to the technical field of cell image processing and segmentation, and particularly relates to a method for detecting the growth state of cells based on image segmentation. Background Art

[0002] During the process of cell culture, it is necessary to observe the growth state of cells in real time, and adjust operations in a timely manner according to the growth state of cells to ensure the smooth progress of cell culture experiments. At present, there are two main methods for observing the growth state of cells: the first method uses a microscope to observe the growth state of cells by human eyes; the second method evaluates data such as the number and survival rate of cells in a sample by manually opening the lid to take a sample and then using a flow cytometer. The observation results obtained by the first method, such as cell survival rate, have large errors, consume a lot of manpower, and have low efficiency. The second method is prone to contaminating and damaging the cells in the existing culture vessels during the process of manual sampling and counting, which affects the normal growth process of cells to a certain extent. With the development of artificial intelligence technology, machine learning algorithms and deep learning algorithms have gradually been applied to the field of cell growth state detection.

[0003] The Chinese invention patent with the application number CN202410755124.4 discloses a method, device, equipment, medium and product for segmenting tumor tissue cell images. The specific steps are as follows: Step 1, obtain a tumor tissue cell image; Step 2, input the tumor tissue cell image into a segmentation model, and the segmentation model outputs a boundary map and a foreground map of the tumor tissue cells. Among them, the segmentation model includes a feature encoder, a context extractor and a feature decoder. The skip connection between the feature encoder and the feature decoder adds an ASPP module and / or a CBAM attention. The feature decoder splices and upsamples the feature maps extracted by the feature encoder and the context extractor, and the attention-weighted feature maps calculated by the CBAM attention to obtain the boundary map and the foreground map of the tumor tissue cells; Step 3, fuse the boundary map and the foreground map to obtain the segmentation result of the tumor tissue cell image. Although this method improves the segmentation accuracy by introducing the ASPP module and the CBAM attention mechanism, its feature extraction process still encodes at a single scale, making it difficult to fully consider the cell block structure and edge details, and lacking the ability to hierarchically model different granularity information. At the same time, this method does not consider the semantic task-assisted optimization strategy, which limits the training efficiency and generalization ability of the model.

[0004] The Chinese invention patent with the application number CN202410275835.1 discloses a method for segmenting cervical liquid-based thin-layer cell images based on the SAM segmentation model, which specifically includes the following steps: Step 1, for the prompt information required for cell segmentation in cervical liquid-based thin-layer cell images, after determining the cell region, construct a color brightness histogram of the cell region in a sampling manner to obtain the seed points of the cell nucleus. At the same time, obtain the auxiliary points of the cytoplasm around the seed points of the cell nucleus, and use the seed points of the cell nucleus and their corresponding auxiliary points as prompt points; Step 2, use the SAM segmentation model to process the cervical liquid-based thin-layer cell images to obtain the mask region corresponding to the cell region, and further optimize and update the mask region according to the contour of the mask region to remove holes and noise, so that the segmented cell region is complete. However, this method relies on manually constructing prompt points and performs segmentation guided by the seed points; therefore, there are problems of strong dependence on prompt information and high manual participation, which limits the automation degree and real-time performance of the system. At the same time, the mask optimization process repairs the contour based on heuristic rules, lacking deep semantic understanding and context modeling, which affects the accuracy and robustness of the segmentation results in complex images. Summary of the Invention

[0005] In view of the above problems, the present invention proposes a method for detecting the cell growth state based on image segmentation, including the following processes:

[0006] S1, Real-time collect cell growth state images;

[0007] S2, Input the image into the trained hierarchical division feature encoder to obtain the coarse-grained features of the cell block and the fine-grained features of the cell boundary;

[0008] S3, Input the coarse-grained features of the cell block and the fine-grained features of the cell boundary into the trained cascaded feature fusion network to obtain the deep interaction fusion features; the cascaded feature fusion network includes a first feature fusion network and a second feature fusion network; the first feature fusion network fuses the coarse-grained features of the cell block and the fine-grained features of the cell boundary in the hierarchical division features output by the hierarchical division feature encoder to obtain the fusion features; the second feature fusion network fuses the fusion features, the coarse-grained features of the cell block, and the fine-grained features of the cell boundary again to obtain the deep interaction fusion features;

[0009] S4, Input the deep interaction fusion features into the trained decoder to perform instance segmentation on the cell images to obtain the instance segmentation results of the cell images; the decoder introduces a dilated convolutional layer on the basis of the classical decoder to increase the receptive field without reducing the resolution of the feature map;

[0010] S5, Calculate the number and survival rate of target cells according to the instance segmentation results of the cell images.

[0011] Preferably, the hierarchical division feature encoder includes a cell block feature roughly extraction network and a cell boundary feature fine-grained encoding network; the cell block feature roughly extraction network divides the image into 16*16 sub-blocks, and based on the self-attention layer and MixFNN performs feature extraction to obtain the coarse-grained features of the cell block; the cell boundary feature fine-grained encoding network then performs refined extraction on the cell boundary, divides the image into 64*64 sub-blocks, and then further extracts the fine-grained features through the encoding layer based on Transformer and the feed-forward neural network to obtain the fine-grained features of the cell boundary; finally, the outputs of the two networks are concatenated to obtain the hierarchical division feature encoder output hierarchical division features.

[0012] Preferably, the process of constructing the dataset for training the hierarchical division feature encoder, the cascaded feature fusion network and the decoder includes: first, collecting image data containing target cells, and performing denoising and normalization processing to obtain standard cell image data as ;

[0013] Using the LabelMe instance segmentation tool to label the target cells to obtain an instance segmentation dataset; that is, drawing an instance bounding box for each cell instance, for each cell instance, assigning a unique instance ID to all pixels within the labeled area, and marking the instance ID label of each cell; marking the non-target cell area as the "background" label, and annotating it with the RGB value 255, 255, 255; for the initial image annotating to obtain the instance label result , and obtaining the instance data ; the initial image and the instance data together constitute the instance segmentation dataset;

[0014] Based on the instance-labeled image, perform script processing to generate a semantic segmentation image, and construct a semantic segmentation dataset together with the initial image; that is, process the RGB of the labeled instances in the image through OpenCV to be the consistent RGB pixel value 0, 0, 0; thus obtaining the semantic label map , the semantic label data and the initial image together constitute the semantic segmentation dataset.

[0015] Preferably, the specific data processing process of the cell block feature roughly extraction network includes:

[0016] Based on the cell block feature roughly extraction network, the input image is roughly divided into 16*16 sub-blocks, and then each 16*16 sub-block will undergo operations of the Conv3 convolutional layer and the max pooling layer to extract the low-level features of the image, including texture and color;

[0017] After that, a self-attention layer is adopted to learn the relationships between different regions in the image, that is:

[0018] ;

[0019] Among them, , and represent matrices obtained by linearly processing through fully connected layers respectively, is the normalization coefficient, represents the softmax function; is the feature extracted after self-attention;

[0020] After that, a mixed feed-forward neural network MixFNN is used to perform a non-linear mapping on the input features to obtain enhanced features ; It includes processing with multi-scale convolutional layers of Conv3, Conv5 and Conv7, and using Resnet residual connections to avoid the problem of gradient disappearance, that is:

[0021] ;

[0022] ;

[0023] Among them, and represent convolution operations with convolution kernels of 5*5 and 7*7 respectively, represents the channel splicing operation; represents the ReLU activation function; is the multi-scale residual feature;

[0024] After that, to strengthen the feature extraction and encoding capabilities, multiple self-attention layers and MixFNN layers are repeatedly stacked to achieve deeper feature learning and encoding; the output of each layer will be used as the input of the next layer for processing; after passing through multiple self-attention layers and MixFNN layers, a cell block coarse-grained feature containing rich information is finally formed .

[0025] Preferably, the data processing process of the cell boundary feature fine-grained encoding network includes:

[0026] First of all, the input image will be divided into 64*64 sub-blocks, and the fine-grained features in the sub-blocks will be preliminarily extracted through the Conv5 convolution operation and the average pooling layer Avgpool:

[0027] ;

[0028] Among them, is the extracted feature, It is a convolution layer function with a convolution kernel of 5*5, is the bias term, is the average pooling operation;

[0029] After that, the encoder layer of the Transformer is used to further refine and extract features:

[0030] ;

[0031] Among them, is the extracted feature representation, represents the processing function of the encoding layer of the Transformer architecture;

[0032] Subsequently, the features extracted by the encoding layer are further non-linearly mapped through a feed-forward neural network FNN, and each layer of the feed-forward neural network consists of two fully connected layers and a ReLU activation function;

[0033] Finally, the fine-grained features of the cell boundary and the coarse-grained features of the cell block are concatenated to provide a more complete feature representation:

[0034] ;

[0035] Among them, is the concatenated feature and also the output result of the hierarchical division feature encoder.

[0036] Preferably, the specific data processing process of the cascade feature fusion network is as follows:

[0037] The first feature fusion network performs feature fusion on the coarse-grained features of the cell block and the fine-grained features of the cell boundary . The first feature fusion network includes a bilinear interpolation upsampling layer, a channel concatenation layer, a 1×1 convolution layer, a ReLU activation function layer, and a batch normalization layer; First, the coarse-grained features of the cell block are input into the bilinear interpolation upsampling layer to perform bilinear interpolation on the feature channels of to upsample to the same size as to obtain the padded coarse-grained features of the cell block ; Secondly, and are input into the channel concatenation layer for feature concatenation to obtain the concatenated feature ; Then, is input into the 1×1 convolution layer to reduce the number of channels to reduce the computational complexity to obtain the dimensionality-reduced feature , and then The ReLU activation function layer and the batch normalization layer are sequentially input to perform batch normalization operations on the feature channels to obtain fused features ;

[0038] The second feature fusion network processes the fused features output by the first feature fusion network as well as the coarse-grained features of the cell blocks and the fine-grained features of the cell boundaries for depth feature fusion; the second feature fusion network includes a bilinear interpolation upsampling layer, a channel concatenation layer, a SENet layer, a 3×3 convolutional layer, a ReLU activation function layer, and a batch normalization layer; first, the bilinear interpolation upsampling layer is used again to upsample the coarse-grained features of the cell blocks to the same size as the fine-grained features ; secondly, the features after the first fusion , the upsampled coarse-grained features of the cell blocks and the fine-grained features of the cell boundaries are concatenated to obtain the concatenated features ; the SENet module is applied to the concatenated features to adaptively re-weight the importance of each channel feature to obtain the features ; is input into a 3×3 convolutional layer to further integrate the feature information and reduce the number of channels to obtain the features , and then is output through the ReLU activation function and batch normalization to obtain the depth fusion feature map .

[0039] Preferably, the decoder network is designed based on a classical encoder structure, including a transposed convolutional layer and an activation function layer, and a dilated convolutional layer is introduced. The convolutional kernels of the transposed convolution and the dilated convolution are both 3×3, and the number of channels is gradually reduced from 256 to 64; the specific data processing process includes:

[0040] Input into the first transposed convolutional layer for upsampling to obtain the downsampled features with a feature map size of W / 8×H / 8 , and the number of channels of is 128; then is input into the ReLU activation function layer for non-linear transformation to obtain the non-linear features ; again, is input into the dilated convolutional layer with a dilation rate of 2, and the output depth features with a size of H / 8×W / 8 are obtained ; again, , the second transposed convolutional layer is responsible for restoring the size of the feature map; again, The input is fed into the Mask branch convolutional layer to generate an instance cell image segmentation mask, that is, the probability that each pixel in the cell image belongs to a different cell instance. What the Mask branch convolutional layer outputs is a three-dimensional tensor containing the probabilities corresponding to each pixel for each cell instance. The specific calculation formula is as follows:

[0041] ;

[0042] Among them, represents the probability that the pixel at the position ( i, j ) in the cell image belongs to the k-th cell instance, is the unnormalized score corresponding to the k-th cell instance of the pixel at the position ( ) in the cell image after the Mask branch convolutional layer processes i, j ;

[0043] Finally, is fed into the sigmoid activation function layer to obtain the predicted value of the cell instance category to which the pixel at the position ( i, j ) in the cell image belongs. The predicted values of the cell instance categories to which all the pixels at all positions in the cell image belong form a set .

[0044] Preferably, during the training process of the network model, an auxiliary transfer optimization training strategy is designed. A cell image semantic segmentation task that is simpler than this task is matched as an auxiliary task for the cell image instance segmentation task, so that the model simultaneously solves the cell image instance segmentation task and the cell image semantic segmentation task during training, and the two tasks share a dual-task loss function for parameter update; the dual-task loss function is specifically as follows:

[0045] ;

[0046] Among them is the loss function of the cell image instance segmentation task, is the classification loss function of the cell image semantic segmentation, and are the weights of and respectively;

[0047] The specific expression of

[0048] is:

[0049] is the pixel scale in the cell image; represents the The weight of a pixel position; Indicating In the Predicted value of the cell instance category to which the Indicates the cell instance category to which the pixel point in the cell image belongs;

[0050] The specific expression of is:

[0051] ;

[0052] Wherein is the number of cell categories, is the pixel scale in the cell image, is one-hot encoding. If the predicted category of the pixel point in the cell image is the same as the cell category , then , otherwise it is , is the probability that the pixel point belongs to the cell category .

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. Improve the accuracy of cell growth state detection: By designing a hierarchical feature encoder, the present invention can extract the coarse-grained features of cell blocks and the fine-grained features of cell boundaries respectively. This multi-level feature extraction method helps to more accurately identify cell morphology and boundaries, thereby improving the accuracy of cell growth state detection;

[0055] 2. Enhance the robustness of the model: The use of a cascaded feature fusion network enables the model to integrate feature information of different scales, improves the adaptability to scale changes of cell images, and enhances the robustness of the model. This is particularly important for processing cell images of different sizes and morphologies;

[0056] 3. Achieve efficient and accurate instance segmentation of cell images: By introducing a dilated convolutional layer in the decoder part, the receptive field of the network is effectively expanded, while maintaining the spatial resolution of the feature map, improving the model's ability to capture detail information and context information, thereby achieving efficient and accurate instance segmentation of cell images;

[0057] 4. Accelerate the training convergence speed of the model: By designing a dual-task collaborative training strategy, combining the cell image instance segmentation task with the cell image semantic segmentation task, and using the context information provided by the semantic segmentation task to assist the instance segmentation task, the training convergence speed of the model is accelerated, and the training efficiency is improved;

[0058] 5. Automation and real-time performance: Once the method of the present invention is trained and deployed in the background, it can achieve real-time detection of the cell growth state. This avoids the time-consuming and inefficient manual observation in the traditional method, as well as the contamination and damage that may be caused by manual sampling, and improves the automation level and efficiency of cell culture experiments;

[0059] 6. Wide application prospects: This method is not only applicable to the detection of the growth state of specific types of cells such as tumor cells and cervical liquid-based thin-layer cells, but can also be extended to other types of cell culture experiments. Brief Description of the Drawings

[0060] In order to more clearly illustrate the technical solutions of the present invention or the prior art, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the following description is only one embodiment of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0061] Figure 1 It is the overall flowchart of the present invention.

[0062] Figure 2 It is the structural diagram of the hierarchical division feature encoder network of the present invention.

[0063] Figure 3 It is the structural diagram of the cascaded feature fusion network of the present invention.

[0064] Figure 4 It is the structural diagram of the decoder network of the present invention.

[0065] Figure 5 It is the example diagram of cell image preprocessing in the embodiment of the present invention.

[0066] Figure 6 It is the experimental result diagram in the embodiment of the present invention. Detailed Embodiments

[0067] The overall process of a method for detecting the cell growth state based on image segmentation proposed by the present invention is as Figure 1 shown. The specific implementation includes the following processes:

[0068] Dataset construction; First, collect image data containing target cells, and perform denoising and normalization processing; Use the LabelMe instance segmentation tool to label the target cells to obtain an instance segmentation dataset; Based on the instance-labeled images, perform script processing to generate semantic segmentation images, and construct a semantic segmentation dataset together with the initial images;

[0069] Design hierarchical division feature encoder; including a rough cell block feature extraction network and a fine-grained cell boundary feature encoding network; the rough cell block feature extraction network divides the image into 16*16 sub-blocks, and based on the self-attention layer and MixFNN perform feature extraction to obtain rough cell block features; the fine-grained cell boundary feature encoding network then performs refined extraction on the cell boundary, divides the image into 64*64 sub-blocks, and then further extracts fine-grained features through an encoding layer based on Transformer and a feed-forward neural network to obtain fine-grained cell boundary features; finally, the outputs of the two networks are concatenated to obtain the hierarchical division feature encoder output hierarchical division features;

[0070] Design a cascaded feature fusion network; including a first feature fusion network and a second feature fusion network; the first feature fusion network fuses the rough cell block features and the fine-grained cell boundary features in the hierarchical division features output by the hierarchical division feature encoder to obtain fusion features; the second feature fusion network fuses the fusion features, the rough cell block features, and the fine-grained cell boundary features again to obtain deep interaction fusion features;

[0071] Design a decoder; introduce a dilated convolution layer on the basis of a classical decoder to significantly increase the receptive field without reducing the resolution of the feature map, and achieve accurate instance segmentation of cell images;

[0072] Design a dual-task collaborative training strategy to train the hierarchical division feature encoder, the cascaded feature fusion network, and the decoder to improve the convergence speed of the model during training;

[0073] Model deployment; deploy the trained hierarchical division feature encoder, cascaded feature fusion network, and decoder in the background to achieve real-time detection of the cell growth state.

[0074] The following further illustrates the specific implementation process of the present invention in combination with specific embodiments.

[0075] I. Construction of cell image dataset

[0076] First, collect initial image data using an optical microscope , a total of groups of images are collected, thus obtaining an initial cell image dataset ; and ;

[0077] Then, use median filtering to denoise the image. For each pixel point in the image , the denoised result is:

[0078] ;

[0079] Among them, represents the neighborhood pixel values centered on the pixel point ; is the median operation;

[0080] The pixel values of the image are linearly mapped to the range through Min - Max normalization, that is:

[0081] ;

[0082] Among them, is the value after normalization of the pixel point , and are the minimum - value and maximum - value functions respectively; denoising and normalization processing are performed on all the images in to obtain the standard cell image data as ;

[0083] Use the professional annotation tool LabelMe to annotate each cell instance in the image; specifically, the operator needs to draw an instance bounding box for each cell instance. For each cell instance, all pixels within the annotation area are assigned a unique instance ID, and the instance ID label of each cell is marked (such as "Cell 1", "Cell 2", and different ID labels are annotated by different RGB values); in addition, the non - target cell area is marked with the "background" label and annotated by the RGB value 255, 255, 255; therefore, for the initial image the instance label result is obtained, and the instance data is obtained; the initial image and the instance data together constitute the instance dataset;

[0084] Based on the obtained instance data , the RGB of the labeled instances in the image is processed by OpenCV to be the consistent RGB pixel value 0, 0, 0; thus, the semantic label map , the semantic label data and the initial image together constitute the semantic dataset.

[0085] II. Design of the hierarchical division feature encoder network structure

[0086] The hierarchical division feature encoder respectively performs rough feature division and refined feature encoding on the image, so that the model can perform feature encoding on the image to be segmented at different levels; specifically, the present invention designs a rough extraction network for cell block features and a fine-grained encoding network for cell boundary features; so that the two branch networks can extract features at different levels, thereby improving the efficiency and accuracy of feature encoding; its network structure is as Figure 2 shown, and the specific steps include:

[0087] 1. Design a rough extraction network for cell block features:

[0088] The input image is roughly divided into 16*16 sub-blocks, and then each 16*16 sub-block will go through the Conv3 convolutional layer and the max pooling layer Maxpool operation to extract the low-level features of the image (including information such as texture and color); this process is expressed as:

[0089] ;

[0090] Among them, is the extracted feature, is the convolutional layer function with a convolutional kernel of 3*3, is the bias term, is the max pooling operation;

[0091] After that, in order to capture the overall division information of the block and the context information between cells, and enhance the long-range dependence relationship in the image, the present invention uses a self-attention layer to learn the relationship between different regions in the image, that is:

[0092] ;

[0093] Among them, , and represent the matrices obtained by linearly processing through the fully connected layer respectively, is the normalization coefficient, represents the softmax function; is the feature after self-attention extraction;

[0094] The mixed feedforward neural network (MixFNN) can perform non-linear mapping on the input features, so as to effectively transfer the position information of the image sub-blocks and obtain the enhanced features ; specifically, this process includes processing with multi-scale convolutional layers of Conv3, Conv5 and Conv7, and using the Resnet residual connection to avoid the problem of gradient disappearance, that is:

[0095] ;

[0096] ;

[0097] Among them, and respectively represent the convolution operations with convolution kernels of 5*5 and 7*7, represents the channel concatenation operation; represents the ReLU activation function; is the multi-scale residual feature;

[0098] After that, in order to further enhance the feature extraction and encoding capabilities, multiple self-attention layers and MixFNN layers are repeatedly stacked to achieve deeper feature learning and encoding. The output of each layer will be processed as the input of the next layer; after passing through multiple self-attention layers and MixFNN layers, a cell block coarse-grained feature containing rich information is finally formed , providing a basis for subsequent feature fusion and instance segmentation tasks.

[0099] 2. Design a refined feature encoding network:

[0100] Its goal is to refine the extraction of features related to cell boundaries in the image; first, the input image will be divided into 64*64 sub-blocks, and the fine-grained features in the sub-blocks will be initially extracted through the Conv5 convolution operation and the average pooling layer Avgpool:

[0101] ;

[0102] Among them, is the extracted feature, is the convolution layer function with a convolution kernel of 5*5, is the bias term, is the average pooling operation;

[0103] After that, taking advantage of the effectiveness of the encoder layer of the Transformer in learning cross-view image details and context information, and finally enhancing the representation of features in global information, the encoding layer of the Transformer architecture is adopted in the present invention to further refine the extraction of features:

[0104] ;

[0105] Among them, is the extracted feature representation, represents the processing function of the encoding layer of the Transformer architecture;

[0106] The features extracted by the encoding layer are further non-linearly mapped through a feed-forward neural network (FNN), and each layer of the feed-forward neural network consists of two fully connected layers and a ReLU activation function. This process includes:

[0107] ;

[0108] Among them, represents the fully connected operation, is the deep feature after non-linear mapping;

[0109] In order to improve the accuracy and depth of the cell boundary features, the present invention stacks multiple self-attention layers and feed-forward neural network layers to form a deep network structure. The output of each layer is processed as the input of the next layer. Repeatedly stacking these layers helps to extract more complex and fine-grained features and improve the performance of the model, and finally obtain the fine-grained cell boundary features output by the refined feature encoding network ;

[0110] Finally, the fine-grained cell boundary features and the coarse-grained cell block features are concatenated to provide a more complete feature representation:

[0111] ;

[0112] Among them, is the concatenated feature and also the output result of the hierarchical division feature encoder.

[0113] III. Design of cascaded feature fusion network

[0114] The cascaded feature fusion network performs cascaded feature fusion on the coarse-grained cell block features and the fine-grained cell boundary features output by the hierarchical division feature encoder, so that the model can more accurately capture and understand the multi-level boundary information in the cell image, thereby improving the cell boundary segmentation accuracy of the model and the robustness to the scale change of the cell image through multi-scale information integration;

[0115] Specifically, the cascaded feature fusion network includes a first feature fusion network and a second feature fusion network. Its network structure is as Figure 3 shown. The steps of inputting the coarse-grained cell block features and the fine-grained cell boundary features into the cascaded feature fusion network are as follows:

[0116] The first feature fusion network processes the coarse-grained cell block features and the fine-grained cell boundary features Perform feature fusion. The first feature fusion network includes a bilinear interpolation upsampling layer, a channel concatenation layer, a 1×1 convolutional layer, a ReLU activation function layer, and a batch normalization layer. First, input the coarse-grained features of the cell block into the bilinear interpolation upsampling layer to perform bilinear interpolation on the feature channels of and upsample them to the same size as to obtain the padded coarse-grained features of the cell block ; Second, input and into the channel concatenation layer for feature concatenation to obtain the concatenated feature ; Second, input into the 1×1 convolutional layer to reduce the number of channels to reduce the computational complexity and obtain the dimensionality-reduced feature , and then input into the ReLU activation function layer and the batch normalization layer in sequence to perform batch normalization operations on the feature channels to obtain the fused feature ;

[0117] The second feature fusion network performs deep feature fusion on the fused feature output by the first feature fusion network, the coarse-grained features of the cell block and the fine-grained features of the cell boundary ; The second feature fusion network includes a bilinear interpolation upsampling layer, a channel concatenation layer, a SENet layer, a 3×3 convolutional layer, a ReLU activation function layer, and a batch normalization layer. First, use the bilinear interpolation upsampling layer again to upsample the coarse-grained features of the cell block to the same size as the fine-grained feature ; Second, concatenate the features after the first fusion , the upsampled coarse-grained features of the cell block and the fine-grained features of the cell boundary to obtain the concatenated feature ; Apply the SENet module to the concatenated feature to adaptively re-weight the importance of each channel feature to obtain the feature ; Input into a 3×3 convolutional layer to further integrate the feature information and reduce the number of channels to 256 to obtain the feature , and then input through the ReLU activation function and batch normalization to output the deep fusion feature map .

[0118] IV. Decoder Network Design

[0119] The classical encoder structure is mainly composed of a transposed convolutional layer and an activation function layer, which is difficult to capture context information. To solve this problem, the present invention introduces a dilated convolutional layer in the classical decoder, which allows the network to effectively expand the receptive field of the network without reducing the spatial resolution of the feature map, and effectively improves the model's ability to capture detailed information and context information. In the present invention, the convolutional kernels of the transposed convolution and the dilated convolution are both 3×3, and the number of channels is gradually reduced from 256 to 64. The network structure is as Figure 4 shown;

[0120] The obtained depth interaction fusion feature map is input into the decoder network. First, is input into the first transposed convolutional layer for upsampling to obtain a reduced-dimension feature with a feature map size of W / 8×H / 8 , and the number of channels is 128; subsequently, is input into the ReLU activation function layer for non-linear transformation to obtain a non-linear feature ; again, is input into the dilated convolutional layer with a dilation rate of 2. Since the padding is also 2, the dilated convolution does not change the size of the feature map, and the output size is the depth feature of H / 8×W / 8 ; again, is sequentially input into the ReLU activation function layer and the second transposed convolutional layer to obtain a high-dimension feature with a size of W×H , and the second transposed convolutional layer is responsible for restoring the size of the feature map. Again, is input into the Mask branch convolutional layer to generate an instance cell image segmentation mask, that is, the probability that each pixel point in the cell image belongs to a different cell instance. What the Mask branch convolutional layer outputs is a three-dimensional tensor containing the probability corresponding to each pixel for each cell instance. The specific calculation formula is as follows:

[0121] ;

[0122] where, represents the probability that the pixel at the position ([[]] i, j ) in the cell image belongs to the k-th cell instance, is the unnormalized score corresponding to the k-th cell instance of the pixel at the position ([[]] ) in the cell image after the Mask branch convolutional layer processes i, j ;

[0123] Finally, is input into the sigmoid activation function layer to obtain the probability at the position ([[]] i, jThe predicted value of the cell instance category to which the pixel on [ [ ID = 0 ] ] belongs. The predicted values of the cell instance categories to which all pixel points at all positions on the cell image belong form a set. .

[0124] V. Training Strategy Design

[0125] Since the cell image instance segmentation task to be solved by the present invention requires accurate segmentation of different types of cells and also segmentation of cells of different instances of the same type, and due to the large number of cell instances in the cell image and the situation of cell boundary intersections, the difficulty of the cell image instance segmentation task is increased. The traditional training strategy based on the cross - entropy loss function is difficult to make the model converge quickly during training. To accelerate the convergence speed of the model during training, the present invention proposes an auxiliary transfer optimization training strategy, which matches a cell image semantic segmentation task simpler than this task as an auxiliary task for the cell image instance segmentation task, so that the model simultaneously solves the cell image instance segmentation task and the cell image semantic segmentation task during training, and the two tasks share a dual - task loss function for parameter update. Semantic segmentation provides context information about the content of the cell image by distinguishing pixels of different cell categories in the cell image, and this context information helps instance segmentation to more accurately identify and segment different instances of cells of the same category.

[0126] Dual - task loss function Specifically, it is as follows:

[0127] ;

[0128] Where is the loss function of the cell image instance segmentation task, is the classification loss function of the cell image semantic segmentation, and are respectively and weights of;

[0129] The specific expression of is:

[0130] ;

[0131] is the pixel scale in the cell image; represents the weight of the th pixel position; represents the predicted value of the cell instance category to which the th pixel point in belongs; ; represents the cell instance category to which the th pixel point in the cell image belongs; ;

[0132] The specific expression is:

[0133] ;

[0134] where is the number of cell types, is the pixel scale in the cell image, is one - hot encoding. If the predicted class of the th pixel point in the cell image is the same as the cell class , then , otherwise , is the probability that the th pixel point belongs to the cell class .

[0135] VI. Model Deployment and Experiments

[0136] Deploy the trained hierarchical division feature encoder, cascaded feature fusion network, and decoder in the background;

[0137] Input the collected cell images into the hierarchical division feature encoder to obtain the coarse - grained features of cell blocks and the fine - grained features of cell boundaries;

[0138] Input the coarse - grained features of cell blocks and the fine - grained features of cell boundaries into the cascaded feature fusion network to obtain the deep - fusion features;

[0139] Input the deep - fusion features into the decoder to obtain the instance segmentation results of cell images;

[0140] Calculate the number and survival rate of target cells according to the instance segmentation results of cell images;

[0141] To verify the performance of the method proposed in the present invention, the inventors created a dataset containing 6,000 cell images. The training set and the test set were divided in a ratio of 7:3. Among them, the training set contained 4,200 cell images, and the test set contained 1,800 cell images. The method proposed in the present invention was compared with YOLOV8 and Mask-RNN, and the comparison of the MPA (mean pixel accuracy of classes) of different models is shown in Table 1. In the training set, the accuracy of the method of the present invention for instance segmentation of cell images was significantly better than that of YOLOV8 and Mask-RNN (P < 0.002, P < 0.002); in the test set, the MPA of the method of the present invention was still the highest, consistent with its performance in the training set, and significantly better than YOLOV8 and Mask-RNN (P < 0.002, P < 0.002). It is worth noting that the difference between the MPA of the method of the present invention on the test set and the training set was small, while YOLOV8 and Mask-RNN performed better on the training set but worse on the test set, indicating that YOLOV8 and Mask-RNN had weak generalization ability on the test set, and the method of the present invention had strong generalization ability on the test set.

[0142] Table 1 Comparison of MPA among different models

[0143]

[0144] The experimental results that can be demonstrated are as Figure 5 and Figure 6 shown. According to Figure 5 it can be observed that after image preprocessing, the line features at the edges of cells become clearer, which makes the subsequent model segmentation work more efficient and accurate. Through image preprocessing, standardized and denoised cell image data are obtained, thus greatly improving the performance and reliability of the cell segmentation task.

[0145] From Figure 6 it can be seen that the instance segmentation method proposed in the present invention can accurately perform precise instance segmentation on different types of target cells, and provides reliable data support for subsequent medical staff to detect the growth state of cells. This method not only improves the accuracy of segmentation, but also ensures the accurate identification of cell states, providing strong technical support for clinical applications.

[0146] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, various changes and modifications can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0147] Although the specific implementation manners of the present invention are described above, they are not limitations on the protection scope of the present invention. Those skilled in the art should understand that various modifications or deformations that can be made without creative efforts on the basis of the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A method for detecting the cell growth state based on image segmentation, characterized in that, It includes the following processes: S1, Collect cell growth status images in real time; S2, Input the images into the trained hierarchical division feature encoder to obtain coarse-grained features of cell blocks and fine-grained features of cell boundaries; S3, Input the coarse-grained features of cell blocks and the fine-grained features of cell boundaries into the trained cascaded feature fusion network to obtain deep interaction fusion features; The cascaded feature fusion network includes a first feature fusion network and a second feature fusion network; The first feature fusion network fuses the coarse-grained features of cell blocks and the fine-grained features of cell boundaries in the hierarchical division features output by the hierarchical division feature encoder to obtain fusion features; The second feature fusion network fuses the fusion features, the coarse-grained features of cell blocks, and the fine-grained features of cell boundaries again to obtain deep interaction fusion features; S4, Input the deep interaction fusion features into the trained decoder to perform instance segmentation on the cell images to obtain the instance segmentation results of the cell images; The decoder introduces a dilated convolution layer on the basis of the classical decoder to increase the receptive field without reducing the resolution of the feature map; S5, Calculate the number and survival rate of target cells according to the instance segmentation results of the cell images.

2. The method for detecting the cell growth state based on image segmentation according to claim 1, wherein: The hierarchical division feature encoder includes a rough cell block feature extraction network and a fine-grained cell boundary feature encoding network; the rough cell block feature extraction network divides the image into 16*16 sub-blocks, and based on the self-attention layer and MixFNN performs feature extraction to obtain the coarse-grained features of cell blocks; the fine-grained cell boundary feature encoding network then performs refined extraction of cell boundaries, divides the image into 64*64 sub-blocks, and then further extracts fine-grained features through an encoding layer based on Transformer and a feed-forward neural network to obtain the fine-grained features of cell boundaries; finally, the outputs of the two networks are concatenated to obtain the hierarchical division features output by the hierarchical division feature encoder.

3. A method for detecting cell growth status based on image segmentation according to claim 1, characterized in that: The dataset construction process for training the hierarchical classification feature encoder, cascaded feature fusion network, and decoder includes: First, collect image data containing target cells, and perform denoising and normalization processing to obtain standard cell image data as ; Use the LabelMe instance segmentation tool to label the target cells to obtain an instance segmentation dataset; that is, draw an instance bounding box for each cell instance, assign a unique instance ID to all pixels within the labeled area for each cell instance, and mark the instance ID label of each cell; label the non-target cell area as the "background" label and annotate it with the RGB value 255, 255, 255; for the initial image The instance label result is obtained by annotation and the instance data is obtained ; the initial image and the instance data together constitute the instance segmentation dataset; Based on the instance-annotated image, perform script processing to generate a semantic segmentation image, and construct a semantic segmentation dataset together with the original image; that is, process the instance RGB with labels in the image through OpenCV to be the consistent RGB pixel value of 0, 0, 0; thereby obtaining a semantic label map , semantic label data and the original image together constitute a semantic segmentation dataset.

4. The method for detecting the cell growth state based on image segmentation according to claim 2, wherein, The specific data processing process of the cell block feature rough extraction network includes: Based on the cell block feature rough extraction network, the input image is roughly divided into 16*16 sub-blocks, and then each 16*16 sub-block is subjected to Conv3 convolutional layer and max pooling layer operations to extract low-level features of the image, including texture and color; After that, a self-attention layer is used to learn the relationship between different regions in the image, that is: ； Among them, , and represent matrices obtained by performing fully connected layer linear processing on respectively, is the normalization coefficient, represents the softmax function; is the feature after self-attention extraction; After that, the input features are nonlinearly mapped using the MixFNN (Mixed Feedforward Neural Network) to obtain enhanced features. ; including processing using multi-scale convolutional layers of Conv3, Conv5, and Conv7, and using Resnet residual connections to avoid the problem of gradient vanishing, that is: ； ； Among them, and respectively represent convolution operations with convolution kernels of 5*5 and 7*7, represents the channel concatenation operation; represents the ReLU activation function; is the multi-scale residual feature; After that, to enhance the feature extraction and encoding capabilities, multiple self-attention layers and MixFNN layers are repeatedly stacked to achieve deeper feature learning and encoding; the output of each layer is processed as the input of the next layer; after passing through multiple self-attention layers and MixFNN layers, a coarse-grained feature of the cell block containing rich information is finally formed .

5. The method for detecting the cell growth state based on image segmentation according to claim 2, wherein: The data processing process of the cell boundary feature fine-grained encoding network includes: First, the input image is divided into 64*64 sub-blocks, and the fine-grained features in the sub-blocks are initially extracted through Conv5 convolutional operations and the average pooling layer Avgpool: ； Among them, is the extracted feature, is the convolutional layer function with a convolutional kernel of 5*5, is the bias term, is the average pooling operation; After that, the encoder layer of the Transformer is used to refine the extraction of features: ； Among them, is the extracted feature representation, represents the encoding layer processing function of the Transformer architecture; Subsequently, the features extracted by the encoding layer are further non-linearly mapped through a feed-forward neural network FNN, and each layer of the feed-forward neural network consists of two fully connected layers and a ReLU activation function; Finally, the fine-grained features of the cell boundaries are concatenated with the coarse-grained features of the cell blocks to provide a more complete feature representation: ； Among them, is the stitched feature and also the output result of the hierarchical division feature encoder.

6. The method for detecting the cell growth state based on image segmentation according to claim 1, wherein, The specific data processing process of the cascaded feature fusion network is: The first feature fusion network fuses the coarse-grained features of cell blocks and the fine-grained features of cell boundaries for feature fusion. The first feature fusion network includes a bilinear interpolation upsampling layer, a channel concatenation layer, a 1×1 convolutional layer, a ReLU activation function layer, and a batch normalization layer. First, the coarse-grained features of cell blocks are input into the bilinear interpolation upsampling layer to perform bilinear interpolation on the feature channels of and upsample them to the same size as to obtain the padded coarse-grained features of cell blocks ; Secondly, and are input into the channel concatenation layer for feature concatenation to obtain the concatenated feature ; Then, is input into the 1×1 convolutional layer to reduce the number of channels to reduce the computational complexity and obtain the dimensionality-reduced feature . Next, is successively input into the ReLU activation function layer and the batch normalization layer to perform batch normalization operations on the feature channels to obtain the fused feature ; The second feature fusion network performs deep feature fusion on the fused features output by the first feature fusion network and the coarse-grained features of the cell blocks and the fine-grained features of the cell boundaries ; The second feature fusion network includes a bilinear interpolation upsampling layer, a channel concatenation layer, a SENet layer, a 3×3 convolutional layer, a ReLU activation function layer, and a batch normalization layer; first, the bilinear interpolation upsampling layer is used again to upsample the coarse-grained features of the cell block to fine-grained features of the same size; secondly, the features after the first fusion , the upsampled coarse-grained features of the cell block , and the fine-grained features of the cell boundary are concatenated to obtain the concatenated features ; Apply the SENet module to the concatenated feature to adaptively re-weight the importance of each channel feature to obtain the feature ; Input into a 3×3 convolutional layer to further integrate the feature information and reduce the number of channels to obtain the feature , and then output the depth fusion feature map through the ReLU activation function and batch normalization .

7. The method for detecting the cell growth state based on image segmentation according to claim 1, wherein: The decoder network is designed based on the classical encoder structure, including a transposed convolutional layer and an activation function layer, and a dilated convolution layer is introduced. The convolution kernels of the transposed convolution and the dilated convolution are both 3×3, and the number of channels is gradually reduced from 256 to 64; The specific data processing process includes: Input into the first transposed convolutional layer for upsampling to obtain a dimensionality-reduced feature with a feature map size of W / 8×H / 8 , and the number of channels is 128; subsequently, input into the ReLU activation function layer for non-linear transformation to obtain a non-linear feature ; again, input into the dilated convolutional layer with a dilation rate of 2, and output a depth feature with a size of H / 8×W / 8 ; again, input sequentially into the ReLU activation function layer and the second transposed convolutional layer to obtain a high-dimensional feature with a size of W×H , and the second transposed convolutional layer is responsible for restoring the size of the feature map; again, input into the Mask branch convolutional layer to generate an instance cell image segmentation mask, that is, the probability that each pixel point in the cell image belongs to different cell instances. What the Mask branch convolutional layer outputs is a three-dimensional tensor containing the probabilities corresponding to each pixel for each cell instance. The specific calculation formula is as follows: ； Among them, represents the probability that the pixel located at the cell image position ( i,j ) belongs to the k-th cell instance, is the unnormalized score corresponding to the k-th cell instance of the pixel at the cell image position ( ) after the Mask branch convolutional layer processes i,j ; Finally, input it into the sigmoid activation function layer to obtain the predicted value of the cell instance category to which the pixel located at the cell image position ( i,j ) belongs. The predicted values of the cell instance categories to which all the pixel points at all positions on the cell image belong form a set .

8. A method for detecting cell growth status based on image segmentation according to claim 1, characterized in that: During the training process of the network model, an auxiliary transfer optimization training strategy is designed. A cell image semantic segmentation task that is simpler than this task is matched as an auxiliary task for the cell image instance segmentation task, so that the model simultaneously solves the cell image instance segmentation task and the cell image semantic segmentation task during training, and the two tasks share a dual-task loss function for parameter update; the dual-task loss function is specifically as follows: ； Among them is the loss function for the instance segmentation task of cell images, is the classification loss function for the semantic segmentation of cell images, and are respectively and the weights of; The specific expression is as follows: ； is the pixel scale in the cell image; represents the weight of the th pixel position; represents in the th pixel belongs to the predicted value of the cell instance category; represents the cell instance category to which the th pixel in the cell image belongs; The specific expression is: ； where is the number of cell types, is the pixel scale in the cell image, is one-hot encoding. If the predicted class of the -th pixel point in the cell image is the same as the cell type , then , otherwise it is . is the probability that the -th pixel point belongs to the cell type .

Images