A method and system for calculating the number and coverage of label-free fibroblast cells

The ViTSeg model, which combines a ViT encoder and a CNN decoder, along with edge enhancement and ridge regression techniques, solves the problem of poor segmentation and recognition of fibroblast cells, and achieves label-free, high-precision calculation of cell number and coverage rate.

CN118115991BActive Publication Date: 2026-11-17XI AN JIAOTONG UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410287235.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-03-13
Publication Date
2026-11-17
Estimated Expiration
2044-03-13

AI Technical Summary

Technical Problem

Existing automated cell segmentation methods are not effective at segmenting and identifying fibroblasts, resulting in low accuracy in calculating cell number and confluence. Furthermore, traditional methods require staining or fluorescent labeling, which can affect cell viability.

Method used

The ViTSeg model, which combines a ViT encoder and a CNN decoder, is used to segment label-free fibroblast cell images by incorporating an edge enhancement loss function and a ridge regression model. Through vector flow field and semantic segmentation mask image prediction, combined with post-processing operations, it adapts to changes in cell size at different culture stages.

Benefits of technology

It improves the segmentation accuracy of fibroblast cell images, reduces the error in calculating cell number and coverage rate, and achieves high-precision automatic calculation without staining or fluorescent labeling.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118115991B_ABST
    Figure CN118115991B_ABST
Patent Text Reader

Abstract

This invention discloses a method and system for calculating the number and coverage rate of label-free fibroblast cells. A deep network model, ViTSeg, is built to learn the mapping from cell images to their vector flow field images and semantic segmentation mask images. Then, gradient flow tracing is performed on the semantic segmentation mask image using the vector flow field to obtain the instance segmentation results of the cell image, thereby statistically determining the number and coverage rate of cells in the image. A Vision Transformer is used as the encoder, and edge detection operators are used to emphasize the inner and outer boundaries of the cell instance mask to guide the model in preserving the shape characteristics of fibroblast cells. For images of different fibroblast cell types showing significant differences in cell instance size, the size vector is obtained from the encoder part of the ViTSeg network, and a cell average size prediction model is constructed based on ridge regression to further improve the segmentation effect of fibroblast cell images. This invention achieves an advanced level of automatic segmentation of label-free fibroblast cell images, improving the accuracy of cell number and coverage rate calculations.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of medical image processing technology, specifically relating to a method and system for calculating the number and coverage of label-free fibroblasts. Background Technology

[0002] Because fibroblasts have a unique shape, their size and density vary significantly at different culture stages. Previous automated cell segmentation methods have been ineffective at segmenting and identifying microscopic images obtained throughout the entire cell culture process, resulting in low accuracy in calculating cell number and confluence. This has hindered further research in tissue engineering, drug screening, wound repair, and disease treatment. Specifically: First, fibroblasts have inwardly concave edges and significant irregularities in their boundaries, which has led to poor performance of previous automatic cell segmentation methods and systems, resulting in large relative errors in the calculation of cell number and coverage rate. Second, as fibroblasts grow, due to the interaction between cells, they tend to adhere to each other and cluster together. Previous cell segmentation methods and systems cannot accurately identify closely contacted fibroblast cell instances, and therefore cannot accurately calculate the cell number and coverage rate. Third, during the culture of fibroblast cells, the cell size increases from small to large and the cell distribution density increases from low to high, which increases the difficulty of accurate automatic segmentation. For cell images in the later stages of culture, previous automatic cell segmentation methods and systems are prone to missing cell instances, resulting in the calculated values ​​of cell number and coverage rate being far lower than the actual values.

[0003] In addition, many previous methods and systems for automatically calculating cell number and coverage rate required staining or fluorescent labeling of cell images. However, staining or fluorescent labeling can alter cell physiological functions and affect cell viability. Moreover, the operation steps are complex, require a large amount of contrast reagents, and the experimental process is time-consuming and labor-intensive. Summary of the Invention

[0004] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing a method and system for calculating the number and coverage of unlabeled fibroblast cells. This method utilizes skip connections to merge a ViT-based encoder and a CNN-based decoder, combined with a loss function including edge enhancement, to construct a ViTSeg model that predicts the vector flow field image and semantic segmentation mask image corresponding to the cell image. Post-processing operations are then used to obtain the instance segmentation result. This addresses the technical problems of low segmentation accuracy and large errors in the automatic calculation of cell number and coverage of unlabeled or fluorescently labeled fibroblast images. Simultaneously, a cell mean size prediction model is constructed based on ridge regression to enable the ViTSeg model to better adapt to changes in cell size at different culture stages, further improving the segmentation effect of fibroblast images and reducing the errors in the automatic calculation of cell number and coverage.

[0005] The present invention adopts the following technical solution: A method for calculating the number and coverage of label-free fibroblasts includes the following steps: S1. Based on the LiveCell dataset and the Sartorius cell instance segmentation dataset, a fibrous cell dataset is constructed. The instance annotations and cell images in the fibrous dataset are preprocessed, and the preprocessed dataset is divided into training set, validation set and test set. S2. Calculate the average cell size of each image based on the labeled mask image in the training set obtained in step S1. Adjust the image size to make the average cell size of the training images the same. Then, perform image augmentation to obtain a new training set. S3. Referring to the Cellpose model's cell instance representation based on vector flow field, calculate the corresponding vector flow field map and semantic mask map based on the labeled mask map in the new training set obtained in step S2. Input the vector flow field map, semantic mask map, and training image into the U-shaped deep network model ViTSeg built on the ViT encoder and CNN decoder architecture, and use the loss function with edge enhancement to guide the training of the U-shaped deep network model ViTSeg. S4. Input the cell images in the training set obtained in step S1 into the U-shaped deep network model ViTSeg obtained in step S3 to obtain the size vector. Train the ridge regression model to fit the relationship between the corresponding vector and the average cell size value of the training images in step S2 to obtain the cell size prediction model. S5. Using the cell size prediction model obtained in step S4 as the test set obtained in step S1, predict the average cell size for each image. Adjust the size of the test image so that the average cell size is the same as that in the training stage. Input the U-shaped deep network model ViTSeg to obtain the predicted vector flow field map and semantic mask map. Perform post-processing using the gradient flow tracing algorithm designed based on the Cellpose model to obtain the instance segmentation result of the cell image. Count the number of instances to obtain the number of cells. Calculate the proportion of the number of pixels covered by the instance to the total number of pixels in the image to obtain the coverage rate value. S6. The instance segmentation results of the test set obtained in step S1 are comprehensively evaluated using the average accuracy with an intersection-union ratio threshold of 0.5, the aggregated Jaccard index, and the panoramic quality. The relative error is used to measure the effectiveness of the cell number and coverage calculation.

[0006] Preferably, in step S1, the fibroblast cell dataset consists of A172 cell line, SH-SY5Y cell line data from the LiveCell dataset, and SH-SY5Y cell line data from the SCIS dataset. Each cell image is converted from its corresponding annotation file to obtain a corresponding annotation mask. Overlapping mask pixels are represented by the category label corresponding to the nearest mask centroid. When more than 75% of a mask overlaps with other masks, the mask is removed. A linear transformation is performed on the cell image pixels in the fibroblast cell dataset. The preprocessed fibroblast cell dataset is then divided into a training set, a validation set, and a test set in an 8:1:1 ratio.

[0007] Preferably, step S2 specifically includes: S201. Define the average cell size of the image as the diameter of the circle in the corresponding mask image whose area is the average area of ​​all cell instances. Calculate the average cell size of each image based on the labeled mask of the training set in step S1. Then, adjust the image size so that the average cell size of the training images is the same. S202. Simultaneously perform image augmentation on the training image and its corresponding labeled mask, and then sample a 224×224 image from the center of the augmented image.

[0008] Preferably, step S3 specifically includes: S301. Simulate thermal diffusion within each cell instance of the labeled mask to obtain a two-dimensional energy function. Take the derivative to obtain gradient field images in the horizontal and vertical directions. Combine the gradient field images in the two directions to obtain a vector flow field image. S302. Obtain a semantic mask by assigning a value of 1 to the pixel corresponding to the cell instance in the labeled mask image and a value of 0 to the background. S303. Introduce the ViT encoder into the downsampling stage, select the feature representations output by four Transformer modules, adjust the dimension of the feature representations, and merge them with the CNN-based decoder through skip connections to construct the ViTSeg deep network, predict the corresponding vector flow field map and semantic mask map based on the cell image. The S304 and ViTSeg deep networks predict outputs horizontal and vertical gradient images and probability maps of pixels located within cells. S305. Use the momentum-driven stochastic gradient descent method to minimize the loss function of step S302 to train the ViTSeg model. Set the momentum of the SGD function in the torch toolkit to 0.9 and the weight decay to 0.00001. Train for 2000 epochs, set the batch size to 8, and set the initial learning rate to 0. In the first 10 epochs, linearly increase the learning rate from the initial value to 0.08 and then keep it. In the last 100 epochs, start the learning rate decay and reduce the learning rate to half of the previous value every 10 epochs.

[0009] More preferably, step S303 specifically includes: S3031, Input image resolution is Divide the image into non-overlapping segments. The image patches are then flattened into 768-dimensional vectors. Each vector is then projected into a 384-dimensional embedding space using a trainable linear mapping layer, while a one-dimensional learnable position embedding is introduced. The resulting vector sequence is then input into a standard Transformer encoder. The S3032 Transformer encoder consists of 12 standard Transformer modules stacked in series, and the encoder parameters are initialized using a ViT-S weight file pre-trained on 104 million 256×256 resolution histological images of the Cancer Genome Atlas. S3033, the first skip connection processes the input image using two 3×3 convolutions combined with batch normalization and the ReLU activation function. The remaining four skip connections adjust the output size of the 3rd, 6th, 9th, and 12th modules in the 12 concatenated stacked Transformer modules to... The first three layers are enlarged using 2×2 deconvolution, and the dimensions of the embedding space are adjusted using 3×3 convolution. The last layer is enlarged using only 2×2 deconvolution, and its resolution is concatenated with the output of the corresponding layer decoder. Then, the dimensions of the output result are adjusted using 1×1 convolution.

[0010] More preferably, step S304 specifically includes: S3041. Edge detection is performed on the real semantic mask image using the Laplacian of Gaussian (LoG) operator. This is implemented using the `gaussian_laplace` function in the SciPy toolkit, with the Gaussian standard deviation of the LoG convolution kernel set as the parameter. ; S3042. In the image obtained by edge detection in step S3041, the pixel positions with a pixel intensity greater than 0.001 are regarded as outer edges, and the pixel positions with a pixel intensity less than -0.001 are regarded as inner edges. S3043. Flatten the real and predicted binary semantic mask images into vectors, then flatten the pixels at the corresponding positions of the inner and outer edges of the two images into vectors again, and concatenate the three vectors in sequence, and calculate the binary cross-entropy loss between the concatenated vectors. S3044. Enlarging the actual horizontal and vertical gradient maps by a factor of 5, we obtain the final loss function as follows:

[0011] in, , , These are the horizontal and vertical gradient plots and the intracellular and extracellular probability plots output by the model, respectively. and These are the actual horizontal and vertical gradient maps, respectively. The image represents the true intracellular and extracellular binary semantic mask, and EE_Loss is the cross-entropy loss based on edge enhancement. This is the Sigmoid activation function.

[0012] Preferably, step S4 specifically includes: S401, the result obtained in step S3 based on the 12th Transformer module Each channel feature map of the dimensional tensor is subjected to global average pooling to obtain a 384-dimensional size vector, which is then normalized using the 2-norm of the size vector. S402. Input the cell images from the training set into the trained ViTSeg model, and obtain the corresponding normalized size vector according to step S401. The objective function of the ridge regression model is established, and the objective function is minimized by iterating for 10 epochs using the linear regression function in the NumPy package, thus obtaining the model parameter vector. The value is used to obtain a cell size prediction model.

[0013] Preferably, step S5 specifically includes: S501. Input the cell images of the test set in step S1 into the trained ViTSeg model encoder to obtain the size vector, and input the corresponding vector into the cell size prediction model to obtain the predicted value of the average cell size. S502. Adjust the size of the test image according to the predicted average cell size so that the average cell size is the same as that in the training stage. Input the ViTSeg model to obtain the instance segmentation result. Calculate the average cell size of the image again using the corresponding instance segmentation mask as the final average cell size value of the test image. Adjust the size of the test image according to the final average cell size value so that the average cell size is the same as that in the training stage. Use the standard bilinear interpolation function in the OpenCV package to adjust the image size. S503, Divide each test image into a group. Overlapping slices, where adjacent slices in the horizontal and vertical directions have 50% partial overlap, are predicted using the ViTSeg model for each slice image. By averaging the prediction results of all slices at each pixel, a vector flow field map and semantic mask map with the same size as the original image are obtained. S504. Obtain the instance segmentation result of the cell image through post-processing; S505. Based on the instance segmentation results of the cell image obtained in step S504, the number of cells is obtained by counting the number of instances, and the coverage rate is obtained by calculating the proportion of the number of pixels covered by the instances to the total number of pixels in the image. The two calculation results are displayed on the corresponding instance mask.

[0014] Preferably, in step S6, the average accuracy AP 0.5 The calculation formula is as follows:

[0015] in, To predict the number of masks where the intersection-union ratio (IU) between instance masks and labeled instance masks is greater than 0.5, The number of predicted instance masks for instances that do not have corresponding labeled instance masks. The number of labeled instance masks for which there is no matching predicted instance mask; The aggregate Jaccard index (AJI) is calculated as follows:

[0016] in, N To indicate the number of mask instances in the mask image, For the first i The nth instance mask, i.e., the nth i Connected components, To predict the relationship between the mask image and The connected component with the largest intersection-union ratio is given by set U, where element F is the index of the predicted mask that does not have a corresponding labeled mask. For connected components that do not intersect with the labeled mask.

[0017] The panoramic quality (PQ) is calculated as follows:

[0018] in, These represent instance masks in the prediction mask and the annotation mask, respectively. IoU is the intersection-union ratio. The threshold crossover ratio for pairing is 0.5.

[0019] Secondly, embodiments of the present invention provide a label-free fibroblast cell number and coverage rate calculation system, comprising: The data module constructs a fibrous cell dataset based on the LiveCell dataset and the Sartorius cell instance segmentation dataset. It preprocesses the instance annotations and cell images in the fibrous dataset and divides the preprocessed dataset into training set, validation set and test set. The augmentation module calculates the average cell size of each image based on the labeled mask image in the training set obtained from the data module. By adjusting the image size, the average cell size of the training images is made the same. Based on this, image augmentation is performed to obtain a new training set. The training module, referencing the cell instance representation based on vector flow field in the Cellpose model, calculates the corresponding vector flow field map and semantic mask map based on the labeled mask map in the new training set obtained by the augmentation module. The vector flow field map, semantic mask map, and training image are input into the U-shaped deep network model ViTSeg, which is built based on the ViT encoder and CNN decoder architecture. The U-shaped deep network model ViTSeg is trained using a loss function with edge enhancement. The fitting module inputs the cell images from the training set obtained by the data module into the U-shaped deep network model ViTSeg obtained by the training module to obtain the size vector. The ridge regression model is then trained to fit the relationship between the corresponding vector and the average cell size value of the training images in the augmentation module, thus obtaining the cell size prediction model. The post-processing module uses the cell size prediction model obtained from the fitting module as the data module to predict the average cell size of each image in the test set. The size of the test image is adjusted so that the average cell size is the same as that in the training stage. The U-shaped deep network model ViTSeg is input to obtain the predicted vector flow field map and semantic mask map. The gradient flow tracing algorithm designed based on the Cellpose model is used for post-processing to obtain the instance segmentation result of the cell image. The number of instances is counted to obtain the number of cells. The proportion of the number of pixels covered by the instance to the total number of pixels in the image is calculated to obtain the coverage rate value. The output module comprehensively evaluates the instance segmentation results of the test set obtained by the data module using the average accuracy with an intersection-union ratio threshold of 0.5, the aggregated Jaccard index, and the panoramic quality. The relative error is used to measure the effectiveness of the cell number and coverage calculation.

[0020] Compared with the prior art, the present invention has at least the following beneficial effects: A label-free method for calculating the number and coverage of fibroblasts employs ViT as the encoder, which, through skip connections and concatenation into a CNN-based upsampling decoder network, can acquire more contextual information while preserving fine-grained features. This improves the prediction performance of vector flow fields and semantic masks in cell images, thereby enhancing instance segmentation results and reducing the relative error in calculating cell number and coverage. Utilizing a loss function with edge enhancement to guide model training allows for more precise segmentation of irregular boundaries in fibroblasts. A cell average size prediction model can estimate the average size of cell instances in cell images, enabling the segmentation model to provide better segmentation results for cell instances of different sizes, as well as better calculation results for cell number and coverage.

[0021] Furthermore, unlabeled fibroblast cell images are typically phase-contrast microscopy images, resulting in low contrast between the background and foreground, making cell boundaries and the image background difficult to distinguish. Considering that the original images are grayscale, a linear transformation is performed on the image pixel values ​​to enhance the contrast between cells and the background, making the cells more prominent and easier to identify.

[0022] Furthermore, ViT is employed as the encoder in the deep network, and input adaptation is achieved through self-attention operations, allowing the model to focus on any region in the image to capture long-range dependencies. By directly modeling global relationships, the receptive field of the image can be expanded, more contextual information can be obtained, and the segmentation accuracy of the network for cell instances of different shapes can be improved. In addition, the encoder weight parameters are initialized with a pre-trained ViT-S weight file, alleviating the "data hunger" problem of ViT. The feature representation extracted by the encoder is merged with the CNN-based decoder through skip connections at multiple resolutions, which helps features flow freely in the network, enriches feature expression, better captures the details of the target, and reduces the loss of fine-grained information. Using a CNN-based decoder enables the model to correctly capture local information during the upsampling stage.

[0023] Furthermore, by using the ViTSeg deep network to predict the vector flow field and semantic mask of the cell image, and then combining it with post-processing operations based on gradient flow tracing to obtain the instance segmentation result, instead of directly predicting the instance mask, we can achieve more accurate segmentation of tightly adhered cells, while reducing the false negative rate of the model when the cell distribution density is high, thereby improving the accuracy of cell number and coverage calculation.

[0024] Furthermore, for the semantic mask map predicted by the ViTSeg deep network, the cross-entropy of the mask edges is emphasized in the loss calculation, guiding the model to better preserve the shape characteristics of fibroblast cells. Based on the edge detection operator, the inner and outer boundaries of the cell instance mask are given greater weight in the cross-entropy loss calculation, making the network focus on the segmentation effect of fibroblast cell boundaries.

[0025] Furthermore, stochastic gradient descent with momentum is used as the optimizer, which alleviates the model's convergence to local minima to some extent and improves the convergence speed. In order to deal with model overfitting, weight decay is also introduced into the optimizer, and an L2 regularization term for the network weight parameters is added after the loss function to prevent the network weight parameters from being too large and reduce the complexity of the model.

[0026] Furthermore, a learning rate warm-up mechanism is employed to allow the model to better adapt to the data in the early stages of training, thereby improving training stability. In the later stages of training, to mitigate over-adjustment and severe oscillations during the optimization process, learning rate decay is initiated when the model approaches the optimal solution, allowing it to explore the parameter space more fully and improve its generalization ability to new data.

[0027] Furthermore, the cell mean size prediction model is built based on ridge regression, and the L2 regularization term in the objective function can effectively reduce overfitting. Using this model, the average size of cell instances in a cell image can be estimated, allowing for image resizing to match the average cell instance size during training. This results in better segmentation results when input into the segmentation model, and more accurate calculations of cell number and coverage ratio.

[0028] It is understandable that the beneficial effects of the second aspect mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here.

[0029] In summary, this invention employs ViT as the encoder of a deep network, merging its extracted feature representations with a CNN-based decoder through multiple resolution skip connections. This improves the prediction accuracy of vector flow fields and semantic masks in cell images. Combined with post-processing operations, it enhances the segmentation performance when cells are tightly adhered and densely distributed. The model training is guided by a loss function with edge enhancement, resulting in more accurate boundary segmentation of fibroblast cells. The average cell size prediction model provides the average size value of cell instances in the cell image, allowing the model to better adapt to the significant size differences of fibroblast cells at different culture stages, further improving the segmentation performance of fibroblast cell images. Building upon the improved segmentation performance of fibroblast cell images, the relative error in calculating the cell number and coverage rate is further reduced.

[0030] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0031] Figure 1 This is the overall flowchart of the present invention; Figure 2 The original cell image and its labeled mask image; Figure 3 Original cell image and its contrast-enhanced image; Figure 4 This is a schematic diagram illustrating the process of calculating the gradient field based on the mask. Figure 5 A schematic diagram illustrating the process of reverse-engineering the instance mask using a vector flow field; Figure 6 An overview of the ViTSeg model; Figure 7 The network structure of ViTSeg; Figure 8 A segmentation outline diagram of the A172 cell line at different culture stages; Figure 9 Segmentation outlines of the SH-SY5Y cell line at different culture stages; Figure 10 A qualitative comparison chart of the segmentation prediction results for the five models; Figure 11 Box plot of the numerical distribution of the five model indicators Figure 12 A comparison chart of calculated A172 cell number and confluence at different culture stages. Figure 13 Comparison of calculated SH-SY5Y cell number and confluence at different culture stages. Figure 14 A schematic diagram of a computer device provided in an embodiment of the present invention; Figure 15 This is a block diagram of a chip provided according to an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] In the description of this invention, it should be understood that the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0034] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0035] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Additionally, the character " / " in this invention generally indicates that the preceding and following objects have an "or" relationship.

[0036] It should be understood that although terms such as first, second, third, etc., may be used in the embodiments of the present invention to describe the preset range, these preset ranges should not be limited to these terms. These terms are only used to distinguish the preset ranges from one another. For example, without departing from the scope of the embodiments of the present invention, the first preset range may also be referred to as the second preset range, and similarly, the second preset range may also be referred to as the first preset range.

[0037] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0038] The accompanying drawings illustrate various structural schematic diagrams according to embodiments disclosed in this invention. These drawings are not to scale, and some details have been enlarged for clarity, and some details may have been omitted. The shapes of the various regions and layers shown in the drawings, as well as their relative sizes and positional relationships, are merely exemplary and may deviate from reality due to manufacturing tolerances or technical limitations. Furthermore, those skilled in the art can design regions / layers with different shapes, sizes, and relative positions as needed.

[0039] This invention provides a label-free method for calculating the number and coverage rate of fibroblast cells. It extracts feature representations based on a ViT encoder, and merges these representations with a CNN-based decoder through multiple resolution skip connections to construct a deep network model, ViTSeg. This model learns the mapping from cell images to their vector flow field images and semantic segmentation mask images. Post-processing operations are then performed to obtain instance segmentation results for the cell images. The ViTSeg network is trained using a loss function with edge enhancement. Utilizing the size vectors obtained from the ViTSeg network encoder, a ridge regression-based model for predicting the average cell size is constructed, further improving the segmentation performance of fibroblast cell images. A fibroblast cell dataset was constructed to train, validate, and test the model. The evaluation results of the test experiments show that the present invention has achieved an advanced level of automatic segmentation of label-free fibroblast cell images and effectively improved the results of automatic calculation of fibroblast cell number and coverage rate. The method of learning the mapping from cell image to its vector flow field image and semantic segmentation mask image based on a deep neural network model, and then using the vector flow field to post-process the semantic segmentation mask image to obtain cell image instance segmentation, can accurately segment adhered and high-density distributed cell instances, thereby improving the results of automatic calculation of number and coverage rate, and has high practical application value.

[0040] Please see Figure 1 This invention discloses a method for calculating the number and coverage of label-free fibroblasts, comprising the following steps: S1. Select fibroblast-like cell line data from the LiveCell dataset and the Sartorius cell instance segmentation dataset (abbreviated as: SCIS dataset), merge them to obtain the fibroblast-type cell dataset, preprocess the instance annotations and cell images in the dataset, and divide the preprocessed dataset into training set, validation set and test set. S101, Dataset Acquisition This invention obtained the LiveCell public dataset and the Sartorius cell instance segmentation public dataset, selected fibroblast-like cell line data from them, and merged them to obtain the fibroblast-type cell dataset, as shown in Table 1. Table 1. Composition of the fibroblast cell image dataset

[0041] The LiveCell dataset, developed by the Sartorius Institute, is a large, hand-annotated dataset for cell instance segmentation. It contains phase-contrast microscopy images of eight diverse cell lines, along with annotations for over 1.6 million single-cell instances. For each cell line, the dataset provides images of the entire process from initial seeding to growth to complete confluence, consistent with the timeline of a typical cell biology experiment. This results in significant variations in cell size and the number of cell instances per image. The SH-SY5Y cell line consistently performs the worst in segmentation tests across the eight cell lines in the LiveCell dataset due to its unique, irregular, and concave morphology. This issue spurred the Sartorius Neural Cell Instance Segmentation Challenge. This competition provides the Sartorius Cell Instance Segmentation dataset, including phase-contrast microscopy images of the Cort, SH-SY5Y, and Astro cell lines.

[0042] S102, Annotation Preprocessing The LiveCell dataset uses COCO format for single-cell instance annotations, while the SCIS dataset uses RLE format. Therefore, we first unified both to RLE format and then converted the files into mask images. For overlapping cell instances in the original annotation files, the overlapping mask pixels were represented by the category label corresponding to the nearest mask centroid. When more than 75% of a mask overlapped with other masks, the mask was removed. Figure 2 The image shows the original cell images (left column) and the masked images obtained after annotation transformation (right column). The first and second rows of images are from the A172 and SH-SY5Y cell lines in the LiveCell dataset, respectively, and the third row is from the SH-SY5Y cell line in the SCIS dataset. In the masked images, the black areas are the background, and the different colored areas represent the annotation masks for different cell body instances.

[0043] S103, Cell Image Preprocessing The label-free phase-contrast microscopy images in the fibroblast cell image dataset exhibit low contrast between the background and foreground, making it difficult to distinguish cell boundaries from the image background. Considering that the original images are all grayscale, a linear transformation of the image pixel values ​​is performed using the following formula to enhance the contrast between cells and the background in the images.

[0044]

[0045] in, The input image contains pixel values. Output the pixel values ​​of the image. This represents the percentile corresponding to the minimum pixel intensity in an image. This represents the percentage quantile corresponding to the maximum pixel intensity of the image.

[0046] Figure 3 The image shows the original image (left column) and its contrast-enhanced image (right column). It can be seen that the cells in the enhanced image are more prominent and easier to distinguish.

[0047] S104, Dataset Partitioning Considering that the fibroblast cell image dataset basically uniformly covers cell images of A172 and SH-SY5Y cell lines from initial seeding to growth and complete confluence, the partitioning process ensured that the proportions of the two cell types in the training, validation, and test sets were roughly equal, and that images of each stage of cell culture were included in all three datasets with roughly the same proportion. The datasets were partitioned in an 8:1:1 ratio, resulting in training, validation, and test sets containing 1160, 152, and 152 cell images, respectively, along with corresponding mask images.

[0048] S2. Calculate the average cell size of each image based on the labeled mask image in the training set obtained in step S1. Adjust the image size to make the average cell size of the training images the same. Then, perform image augmentation to obtain a new training set. S201. Define the average cell size of an image as the diameter of a circle in the corresponding mask image whose area is the average area of ​​all cell instances. Based on this, calculate the average cell size of each image using the labeled mask images of the training set in S1. Then, adjust the image size so that the average cell size of the training images is the same. S202. Image augmentation operations include three types: random rotation, random translation, and horizontal flipping. These are performed simultaneously on the training image and its corresponding labeled mask. Random rotation angles are uniformly distributed from 0° to 360°. Random translation is performed along both the horizontal and vertical directions, with the horizontal translation distance not exceeding [a certain value]. The vertical translation distance does not exceed , W , H These are the width and height of the original image, respectively. Then, a 224×224 image is sampled from the center of the augmented image for model training.

[0049] S3. Referring to the Cellpose model's cell instance representation based on vector flow field, the corresponding vector flow field map and semantic mask map are calculated based on the labeled mask map in the new training set. The two and the training image are input into the U-shaped deep network model ViTSeg built on the ViT encoder and CNN decoder architecture, and the model training is guided by the loss function with edge enhancement. Please see Figures 4 to 7 The specific steps are as follows: S301, according to Figure 4 The schematic process involves simulating thermal diffusion within each cell instance of the labeled mask to obtain a two-dimensional energy function. Taking its derivative yields gradient field images in the horizontal and vertical directions. Combining these two gradient field images yields the vector flow field image. S302. A semantic mask can be obtained by assigning a value of 1 to the pixel corresponding to the cell instance in the labeled mask and a value of 0 to the background. S303. Introduce ViT into the downsampling stage, select the feature representations output by four Transformer modules, adjust the dimension of the feature representations, and merge them with the CNN-based decoder through skip connections to construct the ViTSeg model, so as to predict the corresponding vector flow field map and semantic mask map based on the cell image. Please see Figure 6 , Figure 7 The specific construction process of the ViTSeg deep network is as follows: S3031, Input image resolution is Divide the image into non-overlapping segments. The image patches are then flattened into 768-dimensional vectors. Each vector is then projected into a 384-dimensional embedding space using a trainable linear mapping layer, while a one-dimensional learnable position embedding is introduced. The resulting vector sequence is then input into a standard Transformer encoder. The S3032 Transformer encoder consists of 12 standard Transformer modules stacked in series, and the encoder parameters are initialized using a ViT-S weight file pre-trained on 104 million 256×256 resolution histological images from the Cancer Genome Atlas (TCGA). S3033, the first skip connection processes the input image using two 3×3 convolutions combined with batch normalization and the ReLU activation function. The remaining four skip connections adjust the output size of the 3rd, 6th, 9th, and 12th modules in the 12 concatenated stacked Transformer modules to... The first three layers are enlarged using 2×2 deconvolution, and the dimensions of the embedding space are adjusted using 3×3 convolution. The last layer is enlarged using only 2×2 deconvolution, and then concatenated with the output of the corresponding layer decoder. Finally, the dimensions of the output result are adjusted using 1×1 convolution. The S304 and ViTSeg deep networks predict and output horizontal and vertical gradient images, as well as a probability map of pixel location within a cell. The two gradient images are synthesized to obtain a vector flow field map. The probability map is activated using the Sigmoid function, and after setting a threshold of 0.5, it can be converted into a semantic mask. For the two gradient images, the L2 norm loss function is used to measure the distance between the model's predicted values ​​and the true values. The semantic mask can be viewed as a pixel-level binary classification problem, with loss calculated based on edge-enhanced cross-entropy. The specific process is as follows: S3041. Edge detection is performed on the real semantic mask image using the Laplacian of Gaussian (LoG) operator. This is implemented using the `gaussian_laplace` function in the SciPy toolkit, with the Gaussian standard deviation of the LoG convolution kernel set as the parameter. ; S3042. In the image obtained by edge detection in S3041, the pixel positions with a pixel intensity greater than 0.001 are regarded as outer edges, and the pixel positions with a pixel intensity less than -0.001 are regarded as inner edges. S3043. Flatten the real and predicted binary semantic mask images into vectors. Then, flatten the pixels at the corresponding positions of the inner and outer edges of the two images into vectors again. Concatenate the three vectors in sequence and calculate the binary cross-entropy loss between the concatenated vectors using the following formula:

[0050] in, p The value can be either a binary label 0 or 1. y The probability predicted by the model; S3044. To balance the relative contributions of the L2 norm loss and the edge-enhanced cross-entropy loss, the true horizontal and vertical gradient maps are enlarged by a factor of 5, resulting in the following loss function:

[0051] in, , , These represent the horizontal and vertical gradient plots and the intracellular and extracellular probability plots of the model output, respectively. and These are the actual horizontal and vertical gradient maps, respectively. This is a true binary semantic mask image showing the inside and outside of the cell; EE_Loss represents the edge-enhanced cross-entropy loss introduced in S3041 to S3043. This represents the Sigmoid activation function.

[0052] S305. The ViTSeg model is trained using the momentum-driven stochastic gradient descent method. The momentum of the SGD function in the torch toolkit is set to 0.9, the weight decay is set to 0.00001, and a total of 2000 epochs are trained. The batch size is set to 8, and the initial learning rate is set to 0. In the first 10 epochs, the learning rate is linearly increased from the initial value to 0.08 and then kept there. In the last 100 epochs, the learning rate decay is started, and the learning rate is reduced to half of the previous value every 10 epochs.

[0053] S4. Let the model trained in step S3 be the ViTSeg model. Input the cell images in the training set into the ViTSeg model to obtain the size vector. Train the ridge regression model to fit the relationship between the vector and the average cell size value of the training images in step S2 to obtain the cell size prediction model. Please see Figure 7 The specific steps are as follows: S401, to Figure 7 Based on the 12th Transformer module Each channel feature map of the dimensional tensor is subjected to global average pooling to obtain a 384-dimensional "size vector", which is then normalized using the 2-norm of the size vector. S402. Input the cell images from the training set into the trained ViTSeg model, and obtain the corresponding normalized size vector according to S401. The objective function for establishing the ridge regression model is as follows:

[0054] in, n The number of samples in the training set. The average cell size for each training image. This is a 384-dimensional model parameter vector. The coefficient of the L2 regularization term is taken in this invention. The model parameter vector is obtained by iterating through 10 epochs using the linear regression function from the NumPy package to minimize the objective function. The value is used to obtain a cell size prediction model.

[0055] S5. The cell size prediction model is used to predict the average cell size for each image in the test set. The size of the test image is adjusted so that the average cell size is the same as that in the training stage. The image is then input into the ViTSeg model to obtain the predicted vector flow field map and semantic mask map. The gradient flow tracing algorithm designed based on the Cellpose model is used for post-processing to obtain the instance segmentation result of the cell image. The specific steps are as follows: S501. Input the cell images of the test set in step S1 into the trained ViTSeg model encoder to obtain the size vector, and input the vector into the cell size prediction model to obtain the predicted value of the average cell size. S502. Adjust the size of the test image according to the predicted average cell size so that the average cell size is the same as that in the training stage. Input the ViTSeg model to obtain the instance segmentation result. Use the instance segmentation mask to recalculate the average cell size value of the image as the final average cell size value of the test image. Adjust the size of the test image according to the final average cell size value so that the average cell size is the same as that in the training stage. All the above operations of adjusting the image size are completed using the standard bilinear interpolation function in the OpenCV package. S503, Divide each test image into a group. The image is composed of overlapping slices that cover the entire image, with adjacent slices in the horizontal and vertical directions having 50% overlap. The ViTSeg model is used to predict the image for each slice, and the vector flow field map and semantic mask map of the same size as the original image are obtained by averaging the prediction results of all slices at each pixel. S504, according to Figure 5 The illustrated process is followed by post-processing to obtain instance segmentation results for the cell images. The outlines of the predicted instance masks for the A172 and SH-SY5Y cell lines are then visualized as follows: Figure 8 , Figure 9 As shown, the first, second, and third rows contain images taken during the early, middle, and late stages of cell culture, respectively. S505. Based on the cell image instance segmentation results obtained in S504, the number of cells is obtained by counting the number of instances, and the coverage ratio is calculated as the proportion of pixels covered by the instances to the total number of pixels in the image. The two calculation results are then displayed on the corresponding instance mask image, such as... Figure 12 , Figure 13 As shown.

[0056] S6. Use an average precision (AP) with an intersection-to-union (IoU) ratio threshold of 0.5. 0.5 The three commonly used evaluation metrics in the field of cell segmentation, namely Aggregate Jaccard Index (AJI) and Panoramic Quality (PQ), are used to comprehensively evaluate the instance segmentation results on the test set. The training set in step S1 is used to train four typical segmentation models: Mask R-CNN, U-Net, Cellpose, and Omnipose. The above three evaluation metrics are also used to evaluate their segmentation performance on the test set, so as to compare and analyze the performance with the ViTSeg model provided by this invention. Average precision (AP) with an intersection-union ratio threshold of 0.5 0.5 The calculation formula is as follows:

[0057] Wherein, TP refers to the number of masks with an intersection-union ratio greater than 0.5 between the predicted instance mask and the labeled instance mask, FP refers to the number of predicted instance masks that do not have a corresponding labeled instance mask, and FN refers to the number of labeled instance masks that do not have a matching predicted instance mask.

[0058] The formula for calculating the Aggregate Jaccard Index (AJI) is as follows:

[0059] in, N To indicate the number of mask instances in the mask image, It is the first one i The nth instance mask, i.e., the nth i Connected components, To predict the relationship between the mask image and Given the connected component with the largest intersection-union ratio, and the element F in set U that is the index of the predicted mask that has no corresponding labeled mask, then... For connected components that do not intersect with the labeled mask.

[0060] The formula for calculating Panoramic Quality (PQ) is as follows:

[0061] in, These represent instance masks in the prediction mask and the annotation mask, respectively. IoU is the intersection-union ratio. The crossover ratio (CRO) for pairing was set to 0.5, and the definitions of TP, FP, and FN were the same as in S601. The average values ​​of each metric across the 152 test samples are shown in Table 3. Table 3 Performance comparison of the five models on the test set

[0062] The ViTSeg model provided by this invention outperforms the other four models in all three metrics. Compared with the advanced cell segmentation model Cellpose, the ViTSeg model achieves better AP. 0.5 The performance was improved by 2.89% on the label-free fibroblast cell image, 4.3% on the label-free fibroblast cell image, and 2.61% on the label-free fibroblast cell image. This invention achieves an advanced level of automatic segmentation of label-free fibroblast cell images.

[0063] The evaluation metrics for instance segmentation results of the ViTSeg model on the test set are as follows: =0.6262, AJI=0.6868, PQ=0.5736. Compared with the state-of-the-art cell segmentation model Cellpose, the ViTSeg model has higher performance in AP. 0.5The accuracy was improved by 2.89% on the standard, 4.3% on the AJI standard, and 2.61% on the PQ standard. The relative errors in cell number and coverage calculations were also further reduced.

[0064] To visually compare the segmentation performance of the five models on fibroblast cell images, this paper selected four cell images from the test set, visualized the corresponding labeled masks and the prediction masks of the five models, and extracted representative regions from each image, as shown below. Figure 10 As shown, the automatic segmentation method for label-free fibroblast cell images provided by this invention has better cell detection and recognition performance and segmentation precision than other models, further verifying the quantitative results shown in Table 3.

[0065] Figure 11 The box plots reflect the distribution of the aforementioned metrics for each model across 152 test images. It is evident that the metric values ​​for all five models exhibit significant fluctuations, indicating large variances. Given that the cell images in the test set were taken at different stages of cell culture, it can be further inferred that the performance of each model varies considerably depending on the image taken at different stages of cell culture.

[0066] To further compare the segmentation effect of the ViTSeg model constructed in this invention on fibroblast cell images at different culture stages, this paper divides the 152 cell images in the test set into three stages: early, middle and late, based on the length of cell culture time, cell shape and cell distribution characteristics. The number of images and image characteristics of different growth stages are shown in Table 4.

[0067] Table 4 Results of subdividing the test set by cell culture stage

[0068] Given that the Cellpose model performs best among the four typical models, this paper uses it as a representative model when comparing and analyzing the segmentation effects of different culture stages on cell images. The AP values ​​of the Cellpose and ViTSeg models on cell images at different culture stages, as shown in Table 4, were calculated. 0.5 The indicators are shown in Table 5.

[0069] Table 5. AP of the model at the three culture stages 0.5 Calculation results (average) (Standard deviation)

[0070] AP is classified according to the cultivation period. 0.5The fluctuation range of the indicators narrowed. Both models performed well in segmenting cell images in the early stages of culture, but their segmentation performance declined in the middle and late stages of culture. This is mainly because the boundaries of fibroblast cells become concave in the middle and late stages of culture, with more slender branches, and the cells are tightly adhered and densely distributed. Compared with the Cellpose model, the ViTSeg model constructed in this invention improved the average segmentation performance across all three culture stages, by 2.94%, 2.13%, and 3.58%, respectively. The improvement was greatest in segmenting cell images in the late stage of culture, indicating that the ViTSeg model is better suited to the characteristics of cell adhesion and dense distribution in the late stage of culture.

[0071] To quantitatively compare the calculation performance of the ViTSeg model and the Cellpose model for cell number and coverage ratio, two cell images were randomly selected from each of the three culture stages in Table 4 to calculate the cell number and coverage ratio (one image of A172 cells and one image of SH-SY5Y cells). The calculation results were compared with the results calculated based on the corresponding labeled masks, and the relative errors are shown in Tables 6 and 7. The corresponding visualization results are as follows. Figure 12 , Figure 13 As shown.

[0072]

[0073] The predicted value refers to the number of cells and the coverage rate calculated based on the predicted instance mask, while the labeled value represents the number of cells and the coverage rate calculated based on the labeled mask.

[0074] Table 6. Calculation results of A172 cell number and confluence rate at different culture stages.

[0075] Table 7. Calculation results of SH-SY5Y cell number and confluence rate at different culture stages.

[0076] As shown in Tables 6 and 7, the relative errors of the ViTSeg model in calculating the number and coverage of A172 and SH-SY5Y cells at different culture stages are lower than those of the Cellpose model. This indicates that the ViTSeg model effectively improves the accuracy of cell number and coverage calculations and has high practical application value.

[0077] In another embodiment of the present invention, a label-free fibroblast cell number and coverage rate calculation system is provided. This system can be used to implement the above-mentioned label-free fibroblast cell number and coverage rate calculation method. Specifically, the label-free fibroblast cell number and coverage rate calculation system includes a data module, an augmentation module, a training module, a fitting module, a post-processing module, and an output module.

[0078] The data module constructs a fibrous cell dataset based on the LiveCell dataset and the Sartorius cell instance segmentation dataset. It preprocesses the instance annotations and cell images in the fibrous dataset and divides the preprocessed dataset into training set, validation set and test set. The augmentation module calculates the average cell size of each image based on the labeled mask image in the training set obtained from the data module. By adjusting the image size, the average cell size of the training images is made the same. Based on this, image augmentation is performed to obtain a new training set. The training module, referencing the cell instance representation based on vector flow field in the Cellpose model, calculates the corresponding vector flow field map and semantic mask map based on the labeled mask map in the new training set obtained by the augmentation module. The vector flow field map, semantic mask map, and training image are input into the U-shaped deep network model ViTSeg, which is built based on the ViT encoder and CNN decoder architecture. The U-shaped deep network model ViTSeg is trained using a loss function with edge enhancement. The fitting module inputs the cell images from the training set obtained by the data module into the U-shaped deep network model ViTSeg obtained by the training module to obtain the size vector. The ridge regression model is then trained to fit the relationship between the corresponding vector and the average cell size value of the training images in the augmentation module, thus obtaining the cell size prediction model. The post-processing module uses the cell size prediction model obtained from the fitting module as the data module to predict the average cell size of each image in the test set. The size of the test image is adjusted so that the average cell size is the same as that in the training stage. The U-shaped deep network model ViTSeg is input to obtain the predicted vector flow field map and semantic mask map. The gradient flow tracing algorithm designed based on the Cellpose model is used for post-processing to obtain the instance segmentation result of the cell image. The number of instances is counted to obtain the number of cells. The proportion of the number of pixels covered by the instance to the total number of pixels in the image is calculated to obtain the coverage rate value. The output module comprehensively evaluates the instance segmentation results of the test set obtained by the data module using the average accuracy with an intersection-union ratio threshold of 0.5, the aggregated Jaccard index, and the panoramic quality. The relative error is used to measure the effectiveness of the cell number and coverage calculation.

[0079] In another embodiment of the present invention, a terminal device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment can be used for the operation of a method for calculating the number and coverage rate of label-free fibroblast cells, including: A fibrous cell dataset was constructed based on the LiveCell and Sartorius cell instance segmentation datasets. Instance annotations and cell images in the fibrous dataset were preprocessed, and the preprocessed dataset was divided into training, validation, and test sets. The average cell size of each image was calculated based on the annotation masks in the training set. Image sizes were adjusted to ensure the average cell size of all training images was the same. Image augmentation was then performed to obtain a new training set. Referring to the Cellpose model's vector flow field-based cell instance representation, corresponding vector flow field maps and semantic masks were calculated based on the annotation masks in the new training set. These vector flow field maps, semantic masks, and training images were input into a U-shaped deep network model, ViTSeg, built on a ViT encoder and CNN decoder architecture. A loss function with edge enhancement was used to guide the training of the ViTSeg U-shaped deep network model. Cell images are input into the U-shaped deep network model ViTSeg to obtain size vectors. A ridge regression model is trained to fit the relationship between the corresponding vectors and the average cell size values ​​of the training images, resulting in a cell size prediction model. The cell size prediction model is used to predict the average cell size for each image in the test set. The size of the test images is adjusted so that the average cell size is the same as that in the training stage. The predicted vector flow field map and semantic mask map are then input into the U-shaped deep network model ViTSeg to obtain the predicted vector flow field map and semantic mask map. A gradient flow tracing algorithm based on the Cellpose model is used for post-processing to obtain the instance segmentation results of the cell images. The number of instances is counted to obtain the number of cells, and the coverage ratio is calculated as a percentage of the number of pixels covered by the instances to obtain the coverage ratio value. The instance segmentation results of the test set are comprehensively evaluated using the average precision with an intersection-union ratio threshold of 0.5, the aggregated Jaccard index, and the panoramic quality. The relative error is used to measure the effectiveness of the cell number and coverage ratio calculations.

[0080] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device in a terminal device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the terminal device and extended storage media supported by the terminal device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device.

[0081] One or more instructions stored in the computer-readable storage medium can be loaded and executed by the processor to implement the corresponding steps of the method for calculating the number and coverage of label-free fibroblasts in the above embodiments; one or more instructions in the computer-readable storage medium are loaded and executed by the processor in the following steps: A fibrous cell dataset was constructed based on the LiveCell and Sartorius cell instance segmentation datasets. Instance annotations and cell images in the fibrous dataset were preprocessed, and the preprocessed dataset was divided into training, validation, and test sets. The average cell size of each image was calculated based on the annotation masks in the training set. Image sizes were adjusted to ensure the average cell size of all training images was the same. Image augmentation was then performed to obtain a new training set. Referring to the Cellpose model's vector flow field-based cell instance representation, corresponding vector flow field maps and semantic masks were calculated based on the annotation masks in the new training set. These vector flow field maps, semantic masks, and training images were input into a U-shaped deep network model, ViTSeg, built on a ViT encoder and CNN decoder architecture. A loss function with edge enhancement was used to guide the training of the ViTSeg U-shaped deep network model. Cell images are input into the U-shaped deep network model ViTSeg to obtain size vectors. A ridge regression model is trained to fit the relationship between the corresponding vectors and the average cell size values ​​of the training images, resulting in a cell size prediction model. The cell size prediction model is used to predict the average cell size for each image in the test set. The size of the test images is adjusted so that the average cell size is the same as that in the training stage. The predicted vector flow field map and semantic mask map are then input into the U-shaped deep network model ViTSeg to obtain the predicted vector flow field map and semantic mask map. A gradient flow tracing algorithm based on the Cellpose model is used for post-processing to obtain the instance segmentation results of the cell images. The number of instances is counted to obtain the number of cells, and the coverage ratio is calculated as a percentage of the number of pixels covered by the instances to obtain the coverage ratio value. The instance segmentation results of the test set are comprehensively evaluated using the average precision with an intersection-union ratio threshold of 0.5, the aggregated Jaccard index, and the panoramic quality. The relative error is used to measure the effectiveness of the cell number and coverage ratio calculations.

[0082] Please see Figure 14 The terminal device is a computer device. In this embodiment, the computer device 60 includes a processor 61, a memory 62, and a computer program 63 stored in the memory 62 and executable on the processor 61. When executed by the processor 61, the computer program 63 implements the fluid composition calculation method in the reservoir stimulation wellbore of this embodiment. To avoid repetition, these details are not elaborated here. Alternatively, when executed by the processor 61, the computer program 63 implements the functions of each model / unit in the fluid composition calculation system in the reservoir stimulation wellbore of this embodiment. To avoid repetition, these details are not elaborated here.

[0083] Computer device 60 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. Computer device 60 may include, but is not limited to, a processor 61 and a memory 62. Those skilled in the art will understand that... Figure 14 This is merely an example of computer device 60 and does not constitute a limitation on computer device 60. It may include more or fewer components than shown, or combine certain components, or different components. For example, computer device may also include input / output devices, network access devices, buses, etc.

[0084] The processor 61 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0085] The memory 62 can be an internal storage unit of the computer device 60, such as a hard disk or RAM of the computer device 60. The memory 62 can also be an external storage device of the computer device 60, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on the computer device 60.

[0086] Furthermore, the memory 62 may include both internal storage units of the computer device 60 and external storage devices. The memory 62 is used to store computer programs and other programs and data required by the computer device. The memory 62 can also be used to temporarily store data that has been output or will be output.

[0087] Please see Figure 15 The terminal device is a chip. In this embodiment, the chip 600 includes a processor 622, which may be one or more, and a memory 632 for storing computer programs executable by the processor 622. The computer program stored in the memory 632 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processor 622 may be configured to execute the computer program to perform the generalizable monocular absolute depth map estimation method described above.

[0088] Additionally, chip 600 may also include a power supply component 626 and a communication component 650. The power supply component 626 can be configured to perform power management of chip 600, and the communication component 650 can be configured to enable communication of chip 600, such as wired or wireless communication. Furthermore, chip 600 may also include an input / output interface 658. Chip 600 can operate on an operating system stored in memory 632.

[0089] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0090] Compared to the advanced cell segmentation model Cellpose, the ViTSeg model further improves segmentation performance, specifically in terms of AP. 0.5 The accuracy improved by 2.89% on the surface area, 4.3% on the AJI metric, and 2.61% on the PQ metric. Furthermore, the ViTSeg model demonstrated improved accuracy in calculating cell number and coverage rate compared to the Cellpose model.

[0091] In summary, this invention provides a method and system for calculating the number and coverage of label-free fibroblast cells. It preprocesses label-free fibroblast cell images to enhance image contrast; utilizes ViTSeg as an encoder, merging its extracted feature representations with a CNN-based decoder through five-resolution skip connections; and trains the ViTSeg network using a loss function with edge enhancement, improving the prediction performance of cell image vector flow maps and semantic mask maps. Post-processing further enhances the segmentation accuracy of fibroblast cell instances. For the significant differences in cell instance size among different fibroblast cell images, the size vector is obtained from the ViTSeg network encoder, and a cell average size prediction model is constructed based on ridge regression, further improving the segmentation performance of fibroblast cell images. Comparative experiments on the test set demonstrate that this invention achieves an advanced level of automatic segmentation of label-free fibroblast cell images, effectively improving the accuracy of calculating the number and coverage of cells in relevant images.

[0092] The above content is only for illustrating the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solution based on the technical concept proposed in this invention shall fall within the scope of protection of the claims of this invention.

Claims

1. A method for calculating the number and coverage rate of label-free fibroblast cells, characterized in that, Includes the following steps: S1. Based on the LiveCell dataset and the Sartorius cell instance segmentation dataset, a fibrous cell dataset is constructed. The instance annotations and cell images in the fibrous dataset are preprocessed, and the preprocessed dataset is divided into training set, validation set and test set. S2. Calculate the average cell size of each image based on the labeled mask image in the training set obtained in step S1. Adjust the image size to make the average cell size of the training images the same. Then, perform image augmentation to obtain a new training set. S3. Referring to the Cellpose model's cell instance representation based on vector flow field, calculate the corresponding vector flow field map and semantic mask map based on the labeled mask map in the new training set obtained in step S2. Input the vector flow field map, semantic mask map, and training image into the U-shaped deep network model ViTSeg built on the ViT encoder and CNN decoder architecture, and use the loss function with edge enhancement to guide the training of the U-shaped deep network model ViTSeg. S4. Input the cell images in the training set obtained in step S1 into the U-shaped deep network model ViTSeg obtained in step S3 to obtain the size vector. Train the ridge regression model to fit the relationship between the corresponding vector and the average cell size value of the training images in step S2 to obtain the cell size prediction model. S5. Using the cell size prediction model obtained in step S4 as the test set obtained in step S1, predict the average cell size for each image. Adjust the size of the test image so that the average cell size is the same as that in the training stage. Input the U-shaped deep network model ViTSeg to obtain the predicted vector flow field map and semantic mask map. Perform post-processing using the gradient flow tracing algorithm designed based on the Cellpose model to obtain the instance segmentation result of the cell image. Count the number of instances to obtain the number of cells. Calculate the proportion of the number of pixels covered by the instance to the total number of pixels in the image to obtain the coverage rate value. S6. The instance segmentation results of the test set obtained in step S1 are comprehensively evaluated using the average accuracy with an intersection-union ratio threshold of 0.5, the aggregated Jaccard index, and the panoramic quality. The relative error is used to measure the effectiveness of the cell number and coverage calculation.

2. The method for calculating the number and coverage rate of label-free fibroblasts according to claim 1, characterized in that, In step S1, the fibroblast cell dataset consists of A172 cell line, SH-SY5Y cell line data from the LiveCell dataset, and SH-SY5Y cell line data from the SCIS dataset. The corresponding annotation files are converted into cell images to obtain corresponding annotation mask maps. The overlapping mask pixels in the annotation files are represented by the category label corresponding to the centroid of the mask closest to the corresponding pixel. When more than 75% of the mask overlaps with other masks, the mask is removed. A linear transformation is performed on the cell image pixels in the fibroblast cell dataset. The preprocessed fibroblast cell dataset was divided into training, validation, and test sets in a ratio of 8:1:

1.

3. The method for calculating the number and coverage rate of label-free fibroblasts according to claim 1, characterized in that, Step S2 is as follows: S201. Define the average cell size of the image as the diameter of the circle in the corresponding mask image whose area is the average area of ​​all cell instances. Calculate the average cell size of each image based on the labeled mask of the training set in step S1. Then, adjust the image size so that the average cell size of the training images is the same. S202. Simultaneously perform image augmentation on the training image and its corresponding labeled mask, and then sample a 224×224 image from the center of the augmented image.

4. The method for calculating the number and coverage rate of label-free fibroblasts according to claim 1, characterized in that, Step S3 is as follows: S301. Simulate thermal diffusion within each cell instance of the labeled mask to obtain a two-dimensional energy function. Take the derivative to obtain gradient field images in the horizontal and vertical directions. Combine the gradient field images in the two directions to obtain a vector flow field image. S302. Obtain a semantic mask by assigning a value of 1 to the pixel corresponding to the cell instance in the labeled mask image and a value of 0 to the background. S303. Introduce the ViT encoder into the downsampling stage, select the feature representations output by four Transformer modules, adjust the dimension of the feature representations, and merge them with the CNN-based decoder through skip connections to construct the ViTSeg deep network, predict the corresponding vector flow field map and semantic mask map based on the cell image. The S304 and ViTSeg deep networks predict outputs horizontal and vertical gradient images and probability maps of pixels located within cells. S305. Use the momentum-driven stochastic gradient descent method to minimize the loss function of step S302 to train the ViTSeg model. Set the momentum of the SGD function in the torch toolkit to 0.9 and the weight decay to 0.00001. Train for 2000 epochs, set the batch size to 8, and set the initial learning rate to 0. In the first 10 epochs, linearly increase the learning rate from the initial value to 0.08 and then keep it. In the last 100 epochs, start the learning rate decay and reduce the learning rate to half of the previous value every 10 epochs.

5. The method for calculating the number and coverage rate of label-free fibroblasts according to claim 4, characterized in that, Step S303 specifically includes: S3031, Input image resolution is Divide the image into non-overlapping segments. The image patches are then flattened into 768-dimensional vectors. Each vector is then projected into a 384-dimensional embedding space using a trainable linear mapping layer, while a one-dimensional learnable position embedding is introduced. The resulting vector sequence is then input into a standard Transformer encoder. The S3032 Transformer encoder consists of 12 standard Transformer modules stacked in series, and the encoder parameters are initialized using a ViT-S weight file pre-trained on 104 million 256×256 resolution histological images of the Cancer Genome Atlas. S3033, the first skip connection processes the input image using two 3×3 convolutions combined with batch normalization and ReLU activation; the remaining four skip connections adjust the output size of the 3rd, 6th, 9th, and 12th modules in the 12 concatenated stacked Transformer modules to... The first three layers are enlarged using 2×2 deconvolution, and the dimensions of the embedding space are adjusted using 3×3 convolution. The last layer is enlarged using only 2×2 deconvolution, and its resolution is concatenated with the output of the corresponding layer decoder. Then, the dimensions of the output result are adjusted using 1×1 convolution.

6. The method for calculating the number and coverage rate of label-free fibroblasts according to claim 4, characterized in that, Step S304 specifically includes: S3041. Edge detection is performed on the real semantic mask image using the Laplacian of Gaussian (LoG) operator. This is implemented using the `gaussian_laplace` function in the SciPy toolkit, with the Gaussian standard deviation of the LoG convolution kernel set as the parameter. ; S3042. In the image obtained by edge detection in step S3041, the pixel positions with a pixel intensity greater than 0.001 are regarded as outer edges, and the pixel positions with a pixel intensity less than -0.001 are regarded as inner edges. S3043. Flatten the real and predicted binary semantic mask images into vectors, then flatten the pixels at the corresponding positions of the inner and outer edges of the two images into vectors again, and concatenate the three vectors in sequence, and calculate the binary cross-entropy loss between the concatenated vectors. S3044. Enlarging the actual horizontal and vertical gradient maps by a factor of 5, we obtain the final loss function as follows: in, , , These are the horizontal and vertical gradient plots and the intracellular and extracellular probability plots output by the model, respectively. and These are the actual horizontal and vertical gradient maps, respectively. The image represents the true intracellular and extracellular binary semantic mask, and EE_Loss is the cross-entropy loss based on edge enhancement. This is the Sigmoid activation function.

7. The method for calculating the number and coverage rate of label-free fibroblasts according to claim 1, characterized in that, Step S4 is as follows: S401, the result obtained in step S3 based on the 12th Transformer module Each channel feature map of the dimensional tensor is subjected to global average pooling to obtain a 384-dimensional size vector, which is then normalized using the 2-norm of the size vector. S402. Input the cell images from the training set into the trained ViTSeg model, and obtain the corresponding normalized size vector according to step S401. The objective function of the ridge regression model is established, and the objective function is minimized by iterating for 10 epochs using the linear regression function in the NumPy package, thus obtaining the model parameter vector. The value is used to obtain a cell size prediction model.

8. The method for calculating the number and coverage rate of label-free fibroblasts according to claim 1, characterized in that, Step S5 is as follows: S501. Input the cell images of the test set in step S1 into the trained ViTSeg model encoder to obtain the size vector, and input the corresponding vector into the cell size prediction model to obtain the predicted value of the average cell size. S502. Adjust the size of the test image according to the predicted average cell size so that the average cell size is the same as that in the training stage. Input the ViTSeg model to obtain the instance segmentation result. Calculate the average cell size of the image again using the corresponding instance segmentation mask as the final average cell size value of the test image. Adjust the size of the test image according to the final average cell size value so that the average cell size is the same as that in the training stage. Use the standard bilinear interpolation function in the OpenCV package to adjust the image size. S503, Divide each test image into a group. Overlapping slices, where adjacent slices in the horizontal and vertical directions have 50% partial overlap, are predicted using the ViTSeg model for each slice image. By averaging the prediction results of all slices at each pixel, a vector flow field map and semantic mask map with the same size as the original image are obtained. S504. Obtain the instance segmentation result of the cell image through post-processing; S505. Based on the instance segmentation results of the cell image obtained in step S504, the number of cells is obtained by counting the number of instances, and the coverage rate is obtained by calculating the proportion of the number of pixels covered by the instances to the total number of pixels in the image. The two calculation results are displayed on the corresponding instance mask.

9. The method for calculating the number and coverage rate of label-free fibroblasts according to claim 1, characterized in that, In step S6, the average accuracy AP 0.5 The calculation formula is as follows: in, To predict the number of masks where the intersection-union ratio (IU) between instance masks and labeled instance masks is greater than 0.5, The number of predicted instance masks for instances that do not have corresponding labeled instance masks. The number of labeled instance masks for which there is no matching predicted instance mask; The aggregate Jaccard index (AJI) is calculated as follows: in, N To indicate the number of mask instances in the mask image, For the first i The nth instance mask, i.e., the nth i Connected components, To predict the relationship between the mask image and The connected component with the largest intersection-union ratio is given by set U, where element F is the index of the predicted mask that does not have a corresponding labeled mask. For connected components that do not intersect with the labeled mask; The panoramic quality (PQ) is calculated as follows: in, These represent instance masks in the prediction mask and the annotation mask, respectively. IoU is the intersection-union ratio. The threshold crossover ratio for pairing is 0.

5.

10. A label-free fibroblast cell number and coverage system, characterized in that, include: The data module constructs a fibrous cell dataset based on the LiveCell dataset and the Sartorius cell instance segmentation dataset. It preprocesses the instance annotations and cell images in the fibrous dataset and divides the preprocessed dataset into training set, validation set and test set. The augmentation module calculates the average cell size of each image based on the labeled mask image in the training set obtained from the data module. By adjusting the image size, the average cell size of the training images is made the same. Based on this, image augmentation is performed to obtain a new training set. The training module, referencing the cell instance representation based on vector flow field in the Cellpose model, calculates the corresponding vector flow field map and semantic mask map based on the labeled mask map in the new training set obtained by the augmentation module. The vector flow field map, semantic mask map, and training image are input into the U-shaped deep network model ViTSeg, which is built based on the ViT encoder and CNN decoder architecture. The U-shaped deep network model ViTSeg is trained using a loss function with edge enhancement. The fitting module inputs the cell images from the training set obtained by the data module into the U-shaped deep network model ViTSeg obtained by the training module to obtain the size vector. The ridge regression model is then trained to fit the relationship between the corresponding vector and the average cell size value of the training images in the augmentation module, thus obtaining the cell size prediction model. The post-processing module uses the cell size prediction model obtained from the fitting module as the data module to predict the average cell size of each image in the test set. The size of the test image is adjusted so that the average cell size is the same as that in the training stage. The U-shaped deep network model ViTSeg is input to obtain the predicted vector flow field map and semantic mask map. The gradient flow tracing algorithm designed based on the Cellpose model is used for post-processing to obtain the instance segmentation result of the cell image. The number of instances is counted to obtain the number of cells. The proportion of the number of pixels covered by the instance to the total number of pixels in the image is calculated to obtain the coverage rate value. The output module comprehensively evaluates the instance segmentation results of the test set obtained by the data module using the average accuracy with an intersection-union ratio threshold of 0.5, the aggregated Jaccard index, and the panoramic quality. The relative error is used to measure the effectiveness of the cell number and coverage calculation.

Citation Information

Patent Citations

  • Multi-modal microscopic image cell segmentation method based on convolutional neural network

    CN116229457A

  • Medical image segmentation method based on u-shaped network

    WO2022199143A1