Mesenchymal stem cell confluence degree calculation method and system based on image semantic segmentation

Abstract

The invention provides a mesenchymal stem cell confluence degree calculation method and system based on image semantic segmentation, and the method comprises the steps: training a cell image semantic segmentation model through a training set, and carrying out the test screening of the trained cell image semantic segmentation model through a test set, obtaining an optimal cell image semantic segmentation model; performing preprocessing operation including scale transformation and gray level normalization on the to-be-calculated mesenchymal stem cell phase contrast microimage; and segmenting the to-be-calculated mesenchymal stem cell phase contrast microscopic image into cells and a background by using the optimal cell image semantic segmentation model, and summarizing and counting the pixel proportions of the segmented cells to obtain the mesenchymal stem cell confluence degree so as to realize automatic identification and separation of the mesenchymal stem cells. The method is used for assisting cell culture quality evaluation in medical and biological research.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method and system for calculating the confluence of mesenchymal stem cells based on image semantic segmentation. Background Art

[0002] In the fields of medicine and biology, the culture quality assessment of mesenchymal stem cells (MSCs) usually relies on manual observation of cell images under a microscope.

[0003] In the process of culturing and applying mesenchymal stem cells, it is very important to accurately assess the cell confluence, which directly affects the reliability of subsequent experiments and applications. Due to the complexity of cell morphology and distribution, as well as the quality differences in imaging, traditional methods rely on manual observation and empirical judgment, which is not only time-consuming and laborious, but also difficult to avoid the influence of subjective factors, and is prone to large errors, especially for inexperienced experimenters.

[0004] How to achieve automated and accurate calculation of the confluence of mesenchymal stem cells has become a technical problem that needs to be solved urgently. However, as a common tool for observing living cells, the images taken by phase contrast microscopes have some inherent defects, which poses a great challenge to accurately calculating the confluence.

[0005] First, the boundaries between different cell regions or different cell clusters are blurred, and the relationships between and within the various categories within the image are unclear, which makes it difficult to accurately distinguish single cells or identify cell clusters. Second, the quality of images obtained from different batches of culture or different microscopes varies, and there may be problems such as uneven brightness and low contrast, which makes it impossible to have a more accurate comparison result between different pictures, making it even more difficult for methods that should be consistent to cope with such scenarios. Finally, even in the same culture batch, the cell density in different regions may vary greatly, which makes it difficult for global-based calculation methods to adapt to local changes and accuracy cannot be guaranteed.

[0006] Therefore, how to design an automated and sufficiently intelligent method that can accommodate the possible changes in cell morphology and size in cell images, can address the problem of blurred cell edges and unclear boundaries in cell images, can also identify the problem of uneven cell distribution, and can tolerate the problems and noise that occur in image capture, and ultimately be able to adapt to all batches and various cells in various morphological conditions to automatically calculate the confluence of mesenchymal stem cells in phase contrast microscopy images from all angles. Solving this problem is not only related to the quality control of stem cell culture, but also will provide important technical support for stem cell research and application. Summary of the invention

[0007] The present invention provides a method and system for calculating the confluence of mesenchymal stem cells based on image semantic segmentation, which are used to solve the problems of cell morphology and size changes, unclear cell edges and boundaries, and uneven cell distribution that may occur in existing cell images, and how to automatically calculate cell confluence more intelligently and accurately.

[0008] The present invention provides a method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation, comprising: Using the training set to train the cell image semantic segmentation model, and using the test set to test and screen the trained cell image semantic segmentation model to obtain the optimal cell image semantic segmentation model; Performing preprocessing operations including scale transformation and grayscale normalization on the phase contrast microscopic images of mesenchymal stem cells to be calculated; The optimal cell image semantic segmentation model is used to segment the image of the mesenchymal stem cell phase contrast microscopy picture to be calculated into cells and background, and the pixel ratios of the segmented cells are summarized and counted to obtain the confluence of the mesenchymal stem cells.

[0009] Furthermore, the method of training the cell image semantic segmentation model using the training set and testing and screening the trained cell image semantic segmentation model using the test set to obtain the optimal cell image semantic segmentation model further includes: Acquire a standard data set; the standard data set includes a number of cell phase contrast microscopic images; After preprocessing, the standard data set is divided into a training set and a test set.

[0010] Furthermore, the standard data set is preprocessed and divided into a training set and a test set, including: Performing preprocessing operations including scale transformation and label unification on the standard data set; and, The standard data set that has undergone scale transformation and label unification preprocessing operations is divided into a training set and a test set.

[0011] Furthermore, the standard data set is subjected to preprocessing operations including scale transformation and label unification, including: Performing a scale transformation preprocessing operation on the standard data set to transform the width and height of each picture in the standard data set to an integer multiple of 32; and The standard data set is subjected to a unified label preprocessing operation, and is converted into a corresponding binary mask image according to the original format labels of the images in the standard data set.

[0012] Furthermore, the method of training the cell image semantic segmentation model using the training set and testing and screening the trained cell image semantic segmentation model using the test set to obtain the optimal cell image semantic segmentation model includes: Using the training set, the cell image semantic segmentation model is trained for a preset number of iterations according to a preset batch processing standard; After each iteration of training, the segmentation index loss of the training set corresponding to this round is recorded.

[0013] Furthermore, the using the training set to train the cell image semantic segmentation model for a preset number of iterations according to a preset batch processing standard includes: In each training batch, the images in the training set and the corresponding binary mask images are input into the cell image semantic segmentation model according to a preset batch processing standard; Calculating the output segmentation result of the cell image semantic segmentation model on the training set through forward propagation; Using a loss function, calculating the segmentation index loss between the output segmentation result and the binary mask image input to the cell image semantic segmentation model this time; Calculating the gradient of the parameters of the cell image semantic segmentation model through back propagation; Using an adaptive moment estimation optimizer, the parameters of the cell image semantic segmentation model are updated according to the calculated gradient to gradually minimize the segmentation index loss, and the above steps are repeated until all preset iteration rounds of training are completed.

[0014] Furthermore, the preprocessing operation including scale transformation and grayscale normalization is performed on the mesenchymal stem cell phase contrast microscopy image to be calculated, including: Cutting the phase contrast microscopic image of mesenchymal stem cells to be calculated to obtain several partial images that meet the preset standard size; Calculate the mean and variance of all pixels in each local image; According to the mean and the variance of the local image, the pixel values ​​of the local image are normalized so that the mean is 0 and the variance is 1, so as to obtain the mesenchymal stem cell phase contrast microscopy image to be calculated after scale transformation and grayscale normalization preprocessing operations.

[0015] Furthermore, the optimal cell image semantic segmentation model is used to segment the image of the mesenchymal stem cell phase contrast microscopy image to be calculated into cells and background, and the pixel ratios of the segmented cells are summarized and counted to obtain the mesenchymal stem cell confluence, including: Inputting all local images of the mesenchymal stem cell phase contrast microscopy image to be calculated that have undergone scale transformation and grayscale normalization preprocessing operations into the optimal cell image semantic segmentation model to obtain an initial segmentation result; Normalizing the initial segmentation result by using a Sigmoid function to obtain a normalized segmentation result; The pixels whose normalized segmentation results are greater than the segmentation probability threshold are marked as cell parts, otherwise they are marked as background, so as to obtain a segmentation result with a mark; According to the size of the original image of the phase contrast microscopic image of the mesenchymal stem cells to be calculated that has not been preprocessed, all images in the segmentation results with the identifier are spliced, and the overlapping parts are processed by taking the intersection to obtain a global segmentation result; The pixel count of the cell part of the global segmentation result is performed, and the ratio of the pixels of all cells in the global segmentation result to the original image is calculated to serve as the cell confluence.

[0016] Furthermore, the segmentation probability threshold is set to 0.5, and the overlapping part in which only two rows or two columns of pictures are cells is marked as the cell part, and the other parts are marked as the background part.

[0017] Furthermore, the method of training the cell image semantic segmentation model using the training set and testing and screening the trained cell image semantic segmentation model using the test set to obtain the optimal cell image semantic segmentation model further includes: A cell image semantic segmentation model is constructed, wherein the cell image semantic segmentation model includes an image feature encoder, an image feature decoder, and an image semantic segmentor.

[0018] Furthermore, the construction of the cell image semantic segmentation model includes: The image feature encoder outputs image feature codes at four different depth levels through a 34-layer residual network; The image feature decoder comprises four fully connected convolution modules and four segmentation convolution modules, which sample and accumulate the feature codes of four different depths output by the image feature encoder from the deep level to the shallow level respectively; The image semantic segmentor includes a fully connected convolutional layer and an upsampling operation, and uses a linear activation function to output the image segmentation result.

[0019] Furthermore, the present application also provides a mesenchymal stem cell confluence calculation system based on image semantic segmentation, comprising: An optimal cell image semantic segmentation model training module is used to train the cell image semantic segmentation model using a training set, and to test and screen the trained cell image semantic segmentation model using a test set to obtain an optimal cell image semantic segmentation model; An image processing module, used for performing preprocessing operations including scale transformation and grayscale normalization on the mesenchymal stem cell phase contrast microscopic images to be calculated; The calculation processing module is used to segment the image of the mesenchymal stem cell phase contrast microscopy picture to be calculated into cells and background using the optimal cell image semantic segmentation model, and summarize and count the pixel proportions of the segmented cells to obtain the confluence of the mesenchymal stem cells.

[0020] The technical solution provided by the embodiment of the present invention may have the following beneficial effects: The present invention discloses an automated solution for calculating the confluence of mesenchymal stem cells. The core business scenario problem is how to automatically calculate the confluence of mesenchymal stem cells from phase contrast microscopic images accurately and efficiently, especially when the cell boundaries are unclear and the image quality is uneven.

[0021] Specifically, the present invention discloses a method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation. This method proposes an automated and accurate solution to the problem of confluence assessment during mesenchymal stem cell culture. The present invention first constructs and trains a cell image semantic segmentation model, which uses a deep residual network to extract multi-level features, performs feature fusion through upsampling and lateral connections, and finally generates a segmentation result of the cell image. On this basis, the present invention preprocesses and segments the phase-contrast microscopy images of mesenchymal stem cells for which the confluence is to be calculated, optimizes the segmentation results through post-processing steps, and finally obtains the confluence by counting the cell pixel ratio. This method not only improves the accuracy and efficiency of mesenchymal stem cell confluence calculation, but also provides strong support for automated monitoring and quality control of the cell culture process, which is of great significance for promoting stem cell research and application.

[0022] The present invention constructs a deep learning model, which extracts multi-level features based on an image feature encoder of a 34-layer residual network, fuses these features through an image feature decoder including four fully connected convolutional modules and four segmentation convolutional modules, and finally generates segmentation results by an image semantic segmentor. The model is trained and verified using a standard data set containing a variety of cell phase contrast microscopy images, and an adaptive moment estimation optimizer is used in each training batch to update the model parameters to minimize the loss of segmentation indicators. The significant technical effect of this solution is that it realizes the automation and high precision of mesenchymal stem cell confluence calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The present invention provides a flowchart of a method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation according to an embodiment of the present invention.

[0024] Figure 2 A flowchart of another method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation provided by an embodiment of the present invention.

[0025] Figure 3 A flowchart of another method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation provided by an embodiment of the present invention.

[0026] Figure 4 A flowchart of another method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation provided by an embodiment of the present invention.

[0027] Figure 5 A flowchart of another method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation provided by an embodiment of the present invention.

[0028] Figure 6 A network structure diagram of a cell image semantic segmentation model constructed in an embodiment of the present invention.

[0029] Figure 7 This is an example diagram comparing the effects of the cell image semantic segmentation model on a standard dataset demonstrated in an embodiment of the present invention.

[0030] Figure 8 This is an example diagram comparing the effects of the cell image semantic segmentation model shown in an embodiment of the present invention on umbilical cord mesenchymal stem cells photographed under a microscope.

[0031] Fig. 9 A module diagram of a mesenchymal stem cell confluence calculation system based on image semantic segmentation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with the accompanying drawings and embodiments. The present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant inventions, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the invention are shown in the accompanying drawings.

[0033] See also Figure 1 , Figure 1 The present invention provides a flowchart of a method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation. Figure 1 As shown, this embodiment provides a method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation, which specifically includes: S110, using the training set to train the cell image semantic segmentation model, and using the test set to test and screen the trained cell image semantic segmentation model to obtain the optimal cell image semantic segmentation model.

[0034] When using the training set to train the cell image semantic segmentation model, you first need to prepare a sufficient number and quality of cell image datasets. These datasets should contain cell images of various types, morphologies, and growth states, as well as the corresponding accurately labeled segmentation mask images. Usually, professional cell biologists and medical imaging experts are required to complete the collection and annotation of cell images to ensure the accuracy and representativeness of the dataset.

[0035] When constructing a semantic segmentation model for cell images, the current leading deep learning semantic segmentation network architecture can be used. The network architecture generally adopts an encoder-decoder structure, that is, a convolutional neural network is first used to extract multi-scale and multi-level features of the image, and then the feature map is restored to the resolution of the original image through deconvolution or upsampling operations, and a class probability map for each pixel is generated. During the training process, the model learns and optimizes network parameters by minimizing the loss function between the predicted segmentation result and the true segmentation mask. Commonly used loss functions include cross entropy loss, Dice loss, Focal Loss, etc., which can be selected and improved according to the characteristics and needs of specific tasks. Dice loss function (DiceLoss) is a loss function commonly used in medical image segmentation and other semantic segmentation tasks. Focal Loss is a loss function used for target detection. It effectively addresses the problem of sample imbalance by reducing the weight of easy-to-classify samples and increasing the weight of difficult-to-classify samples. In order to make full use of limited training data and improve the generalization performance of the model, data augmentation technology can be used to expand the training set. Common data enhancement methods include random cropping, translation, rotation, flipping, scaling, color transformation, etc. These operations can generate more diverse training samples, allowing the model to learn various changes and interference factors in cell images. In addition, the idea of ​​transfer learning can be used to initialize the segmentation model using a backbone network pre-trained on a large-scale natural image dataset, and then fine-tune it on the cell image dataset, thereby accelerating the convergence speed of the model and improving performance.

[0036] After the model training is completed, a separate test set is needed to evaluate and screen the model. The test set should have the same data distribution as the training set, but should not overlap with the training set to ensure the objectivity and credibility of the evaluation results. Multiple common semantic segmentation evaluation indicators, such as pixel accuracy, average IoU, Dice coefficient, etc., can be used to comprehensively measure the performance of the model on the test set. Mean IoU (Mean Intersection over Union, MIoU) is a commonly used evaluation indicator in semantic segmentation tasks, which is used to measure the segmentation performance of the model on all categories. It is obtained by calculating the IoU of each category and then taking the average. The Dice coefficient is a statistical tool for measuring the similarity between two samples. It is widely used in data analysis and image processing, especially in medical image segmentation. The value of the Dice coefficient ranges from 0 to 1, where 1 means exactly the same and 0 means completely different. In order to select the model with the best performance, multiple training experiments with different hyperparameter configurations can be performed, such as adjusting the learning rate, batch size, regularization coefficient, etc., and then comparing their evaluation indicators on the test set, and selecting the one with the best comprehensive performance as the final cell image semantic segmentation model. This process can be guided by an early stopping strategy on the validation set, that is, evaluating the performance of the model on the validation set at the end of each training cycle, and stopping training when the performance has not improved for multiple consecutive cycles to prevent overfitting.

[0037] After selecting the optimal cell image semantic segmentation model, it can be applied to actual cell image analysis tasks. Before use, the cell images to be analyzed need to be preprocessed, such as scale normalization, grayscale transformation, etc., to eliminate the differences between different images and keep them consistent with the data distribution during model training. The preprocessed cell images are forward inferred through the trained semantic segmentation model to obtain the probability value of each pixel belonging to the cell area or the background area. By setting a threshold for the probability value, the original image can be segmented into two parts: cells and background. Then, the number of cell pixels can be counted and indicators such as confluence can be calculated, providing a quantitative basis for subsequent cell analysis and medical diagnosis.

[0038] S120, performing a preprocessing operation including scale transformation and grayscale normalization on the mesenchymal stem cell phase contrast microscopic image to be calculated; The phase contrast microscopic images of mesenchymal stem cells to be calculated are preprocessed, mainly including two steps: scale transformation and grayscale normalization. Scale transformation is to uniformly scale the original microscopic images to a fixed size, such as 512x512 pixels, so that the subsequent semantic segmentation model can adapt well to images of different sizes. Through scale transformation, cell images from different sources and different magnifications can be converted to a unified size specification, which is convenient for model training and reasoning. Grayscale normalization is to uniformly map the pixel value range of the image to between 0 and 1 to eliminate the differences in image brightness and contrast. The pixel values ​​of the original microscopic images are usually integers between 0 and 255. The brightness and contrast of the images will vary greatly under different devices and different shooting conditions. In order to reduce the impact of this difference on the accuracy of semantic segmentation, it is necessary to grayscale normalize the images and divide the pixel values ​​by 255, so that the pixel value range of all images is normalized to between 0 and 1. Grayscale normalization allows the semantic segmentation model to pay more attention to the content of the image rather than low-level visual features such as brightness and contrast, thereby improving segmentation accuracy. The preprocessed cell images are input into the trained semantic segmentation model for inference, and the model will classify each pixel into two categories: cells or background. The segmentation result output by the model is a binary image, where the area with a pixel value of 1 represents cells, and the area with a pixel value of 0 represents the background. The segmentation result can accurately distinguish the cell area in the image.

[0039] In this embodiment, the phase contrast microscopic image of mesenchymal stem cells to be calculated is obtained and preprocessed, including scale transformation and grayscale normalization. In this embodiment, the cell phase contrast microscopic image to be calculated is obtained by taking local sampling of umbilical cord mesenchymal stem cells in the culture bottle under LED bright field conditions by inverted biological microscope MI52-N, and the original pixel is 4000*3000, which is a three-channel true color image. In order to be consistent with the training data, the green channel is taken as a single-channel grayscale image. Due to the uneven illumination of the original image shooting device, which is specifically manifested as bright center and dim edge of the image, it is cut into 7*6 pixels of 672*512 local images, and then the pixel value mean and variance of each local image are taken to perform zero mean normalization to reduce the illumination difference. After these preprocessing, it is input into the trained cell image semantic segmentation model for reasoning, and the semantic segmentation output of 7*6 local images can be obtained.

[0040] S130, using the optimal cell image semantic segmentation model to segment the image of the mesenchymal stem cell phase contrast microscopy image to be calculated into cells and background, and summarizing and statistically analyzing the pixel ratios of the segmented cells to obtain the mesenchymal stem cell confluence.

[0041] After obtaining the semantic segmentation results, it is necessary to further count the proportion of cell pixels in the entire image as a quantitative indicator of cell confluence. Specifically, the number of pixels with a pixel value of 1 in the segmentation result image is added up, and then divided by the total number of pixels in the image. The obtained ratio is the cell confluence. For example, for a 512x512 image, 100,000 pixels are divided into cell categories after semantic segmentation, so the confluence is 100,000 / (512x512)=38.15%. Cell confluence is an important indicator for evaluating the culture status of mesenchymal stem cells. The traditional calculation method mainly relies on manual observation of cell images under a microscope, which has the disadvantages of strong subjectivity and poor repeatability. Using image semantic segmentation to automatically calculate the confluence can avoid the subjectivity of manual observation and improve the accuracy and repeatability of the calculation. At the same time, this method can process a large number of cell images, which is suitable for large-scale cell production processes, improves the automation level of the cell culture process, and provides a powerful tool for fields such as cell therapy and regenerative medicine.

[0042] See also Figure 2 , Figure 2 A flowchart of another method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation provided by an embodiment of the present invention. Figure 2 As shown, Figure 1 Compared with the method for calculating the confluence of mesenchymal stem cells shown in the figure, the difference is that Figure 1 Before step S110, that is, the cell image semantic segmentation model is trained using the training set, and the trained cell image semantic segmentation model is tested and screened using the test set to obtain the optimal cell image semantic segmentation model, the following steps S101 and S102 are also included: S101, acquiring a standard data set; the standard data set includes a number of cell phase contrast microscopic images; Before using the training set to train the cell image semantic segmentation model and using the test set to test and screen the trained cell image semantic segmentation model to obtain the optimal cell image semantic segmentation model, it is necessary to first obtain a standard data set.

[0043] The standard data set should contain a large number of cell phase contrast microscopy images, which need to come from cell samples of different batches and different culture conditions to ensure the diversity and representativeness of the data. For example, you can collect mesenchymal stem cell images of different donor sources, different passage numbers, and different confluences. Usually at least hundreds of images are required to meet the training requirements. After obtaining the standard data set, it is necessary to preprocess it. The first is to perform a scale transformation and uniformly scale all images to a fixed size, such as 672*512 pixels. This is to facilitate the training and reasoning of subsequent models. Secondly, the labels of the images need to be unified. Since the images may come from different sources, their label formats may be different. They need to be unified into a form that the model can accept, such as converting them into mask images, where the pixel value of the cell area is 1 and the pixel value of the background area is 0. After scale transformation and label unification, a standardized data set is obtained.

[0044] Specifically, it can be to obtain the public cell phase contrast microscopy image dataset LIVECell as a standard dataset. The LIVECell dataset is a public dataset for cell image analysis and research. It contains thousands of high-quality cell images, covering various cell types, cell states, and experimental conditions. Specifically, the LIVECell dataset is a large-scale dataset containing 5,239 professionally verified images, with a total of 1,686,352 manually annotated information for individual cells, covering eight different cell types: A172, BT-474, BV-2, Huh7, MCF7, SH-SY5Y, SkBr3, and SK-OV-3, covering different growth stages from early cell seeding to fully fused states, ensuring wide applicability and diversity. This dataset uses a real-time cell image analysis platform to capture images every 4 hours over 3 to 5 days, using a 10x objective to reduce halo artifacts and increase high-frequency content. At the same time, measures were taken to avoid deviations caused by the accumulation of annotators' experience. For example, the data set was divided into eight batches, and cell types, time points, and well plate positions were evenly distributed and then annotated in batches, ensuring the high consistency and reliability of the data set.

[0045] The original label of the LIVECell dataset is in JSON format, which records the cell type, box boundary value sequence, and polygon boundary vertex coordinate sequence corresponding to each cell in each picture. In order to adapt to the semantic segmentation task, the polygon boundary vertex coordinate sequence of each cell in each picture in its JSON label is extracted, and the enclosed area is used as the area occupied by the cell, and the other area is used as the background area. The binary mask image obtained in this way is the semantic segmentation label. In order to adapt to the feature scale scaling of subsequent image feature encoding, the width and height of each cell image are transformed to an integer multiple of 32. Since the original width and height of the image in the dataset are 704*520, in order to keep the original pixel size as much as possible, it is cropped to 672*512. After the above processing, it is divided into a training set containing 3727 images and a test set of 1512 images.

[0046] S102, preprocessing the standard data set and dividing it into a training set and a test set.

[0047] Furthermore, Figure 2 Step S102, i.e., dividing the standard data set into a training set and a test set after preprocessing, includes: Performing preprocessing operations including scale transformation and label unification on the standard data set; and, The standard data set that has undergone scale transformation and label unification preprocessing operations is divided into a training set and a test set.

[0048] For example, after obtaining the public cell phase contrast microscopy image dataset LIVECell, the dataset images are preprocessed, including scale transformation and label unification, and divided into training and test sets.

[0049] In order to train and evaluate the model, the data set needs to be divided into two parts: the training set and the test set. Usually, the random division method is adopted to randomly divide the data set according to a certain ratio (such as 8:2). For example, for a data set containing 500 cell images, 400 of them can be randomly selected as the training set and the remaining 100 as the test set. It should be noted that the divided training set and test set should not overlap to avoid overfitting. At the same time, the diversity of samples in the training set and the test set should be guaranteed. For example, samples with different confluence should be evenly distributed in the two sets to avoid affecting the model performance due to uneven sample distribution.

[0050] Preprocessing the standard dataset and dividing the training and test sets are important preparations for training the semantic segmentation model of cell images. Data quality and quantity are the basis for ensuring model performance. Only by establishing a high-quality standard dataset and undergoing reasonable preprocessing and division can reliable data support be provided for subsequent model training and screening. In practical applications, this process can be adjusted and optimized as necessary according to specific circumstances, such as using more advanced data enhancement techniques to enrich training data and improve data utilization efficiency. The choice of data strategy requires a trade-off between effect and cost in order to achieve the best model performance at the lowest cost.

[0051] Based on the above step S102, in specific implementation, the standard data set is subjected to a preprocessing operation including scale transformation and label unification, including: Performing a scale transformation preprocessing operation on the standard data set to transform the width and height of each picture in the standard data set to an integer multiple of 32; and The standard data set is subjected to a unified label preprocessing operation, and is converted into a corresponding binary mask image according to the original format labels of the images in the standard data set.

[0052] In this implementation, the standard data set is subjected to a scale transformation preprocessing operation, and the width and height of each picture in the standard data set are transformed to an integer multiple of 32. The purpose of this step is to adapt to the scale scaling requirements of subsequent image feature encoding. Since deep learning models usually have certain requirements on the size of the input image, in order to facilitate the training and reasoning of the model, the pictures in the data set need to be uniformly adjusted to a suitable size. Considering that the downsampling operation of the pooling layer in the convolutional neural network is usually performed in powers of 2, adjusting the width and height of the picture to an integer multiple of 32 can better adapt to the network structure. For example, if the size of the original picture is 640×480, it can be scaled to 640×512, so that the width and height are 20 times and 16 times of 32 respectively. The adjusted picture size is not only convenient for feature extraction and downsampling, but also can retain the pixel information of the original image as much as possible, reducing the information loss caused by scale transformation.

[0053] And, in this implementation, the standard data set is subjected to a unified label preprocessing operation, and converted into a corresponding binary mask image according to the original format label of the image in the standard data set. The purpose of this step is to convert the annotation information of different formats into the standard format required for the semantic segmentation task, so as to facilitate subsequent model training and evaluation. In the cell image data set, the original annotation information is usually given in the form of a sequence of polygon vertex coordinates, which identifies the outline of each cell instance in the image. In order to convert it into the pixel-level annotation required for the semantic segmentation task, it is necessary to generate the corresponding binary mask image based on the polygon vertex coordinates.

[0054] Specifically, each pixel in the image can be traversed to determine whether it is located in the polygonal area of ​​a cell. If so, the pixel is marked as foreground (that is, the category to which the cell belongs), otherwise it is marked as background. In this way, a binary mask image with the same size as the original image is obtained, where the foreground pixel value is 1 and the background pixel value is 0. In this way, the contour information of the cell instance is converted into a pixel-level category annotation, which matches the requirements of the semantic segmentation task and provides a unified annotation format for subsequent model training and evaluation.

[0055] See also Figure 3 , Figure 3 A flowchart of another method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation provided by an embodiment of the present invention. Figure 3 As shown, Figure 2 Compared with the method for calculating the confluence of mesenchymal stem cells shown in the figure, the difference is that Figure 2 The step S110 shown, i.e., using the training set to train the cell image semantic segmentation model, and using the test set to test and screen the trained cell image semantic segmentation model to obtain the optimal cell image semantic segmentation model, includes the following steps S111, S112 and S113: S111, using the training set, training the cell image semantic segmentation model for a preset number of iterations according to a preset batch processing standard; When using the training set to train the semantic segmentation model of cell images, batch processing is usually used, that is, dividing the training set into several batches, each batch containing a certain number of samples. Batch processing can speed up the training process of the model, because in each batch, the forward propagation and back propagation of multiple samples can be calculated in parallel, thereby reducing the training time. At the same time, batch processing can also smooth the gradient fluctuations during training and improve the generalization ability of the model. When determining the batch size, it is necessary to weigh the computational efficiency and memory usage. The larger the batch size, the higher the computational efficiency, but the larger the memory usage. Therefore, it is necessary to select an appropriate batch size based on hardware conditions and data scale.

[0056] S112, after each iteration of training is completed, the segmentation index loss of the training set corresponding to this iteration is recorded; After each iteration of training, the segmentation index loss of the training set corresponding to that round is recorded, such as Dice loss or cross entropy loss. These loss indicators reflect the degree of fit of the model on the training set. The smaller the loss, the better the model fits the training data. However, focusing only on the loss of the training set cannot fully evaluate the performance of the model, because the model may have overfitting problems, that is, it performs well on the training set but performs poorly on the test set. Therefore, during the training process, it is also necessary to monitor the loss and evaluation indicators of the validation set, such as the Dice coefficient or IoU, to determine whether the model is overfitting. Usually, the loss and evaluation indicators of the validation set will first increase as the training progresses, and then gradually stabilize. At this time, it can be considered that the model has reached the optimal state.

[0057] S113, selecting the parameters of the cell image semantic segmentation model with the smallest segmentation index loss during the training process as the final model parameters, so as to obtain the corresponding optimal cell image semantic segmentation model; After the training is completed, the parameters of the cell image semantic segmentation model with the smallest segmentation index loss during the training process are selected as the final model parameters to obtain the corresponding optimal cell image semantic segmentation model. This approach is based on the assumption that the model with the smallest loss fits the best on the training set, and therefore may also perform best on the test set. However, this assumption is not always true, because the model with the smallest loss may have been overfitted to the training set and perform poorly on the test set. Therefore, a more prudent approach is to monitor the evaluation indicators of the validation set during the training process, and select the model with the best evaluation indicators on the validation set as the final model. This can avoid overfitting to a certain extent and improve the generalization ability of the model. In addition, it can also prevent overfitting algorithms, that is, when the evaluation indicators of the validation set have not improved for several consecutive training processes, stop training and return to the model with the best evaluation indicators of the previous validation set as the final model.

[0058] In this embodiment, Figure 3 The step S111 shown, i.e., using the training set to train the cell image semantic segmentation model for a preset number of iterations according to a preset batch processing standard, comprises the following steps: In each training batch, the images in the training set and the corresponding binary mask images are input into the cell image semantic segmentation model according to a preset batch processing standard; Calculating the output segmentation result of the cell image semantic segmentation model on the training set through forward propagation; Using a loss function, calculating the segmentation index loss between the output segmentation result and the binary mask image input to the cell image semantic segmentation model this time; Calculating the gradient of the parameters of the cell image semantic segmentation model through back propagation; Using an adaptive moment estimation optimizer, the parameters of the cell image semantic segmentation model are updated according to the calculated gradient to gradually minimize the segmentation index loss, and the above steps are repeated until all preset iteration rounds of training are completed.

[0059] Furthermore, the segmentation probability threshold is 0.5.

[0060] In specific implementation, in each training batch, the images in the training set and the corresponding binary mask images are input into the cell image semantic segmentation model. Assume that the preset batch size is 16, that is, each batch contains 16 cell images and corresponding 16 binary mask images. Then, in this step S111, 16 pre-processed grayscale cell images of size 672*512 are sent to the cell image semantic segmentation model each time. These cell images may cover cell states at different growth stages and densities, and binary mask images corresponding to the images are also sent. The binary mask image is an image that has been manually annotated by a professional or preliminarily annotated by an algorithm and then manually corrected, in which the cell part is usually marked as white (pixel value is 255) and the background part is black (pixel value is 0). Doing so can ensure that the model learns how to accurately distinguish between cells and background, and enables the model to learn segmentation capabilities from different images more accurately.

[0061] After the model receives a batch of images and masks, it calculates the output segmentation results of the cell image semantic segmentation model on the training set through forward propagation. This step is the process of the model predicting each input cell image based on the current parameters. At this stage, the model will first go through a series of convolutional layers, pooling layers and other structures for feature extraction. In specific implementation, efficient networks such as residual networks can also be used as the backbone to improve the fitting ability of the network. Then, the size of the feature map is gradually restored by upsampling or deconvolution operations to expand the resolution of the feature map, restore the image features, and fuse the features of different resolution levels extracted previously during the restoration process, which is conducive to achieving more refined feature output. Next, each pixel in the image is classified through a classification layer composed of a fully connected layer or 11 convolutional layers to obtain a preliminary segmentation result.

[0062] For example, the output value of a pixel may be between 0 and 1, such as 0.8, which represents the probability that the model predicts that the pixel belongs to a cell. After forward propagation calculation, for this batch of 16 images, the model will obtain 16 corresponding preliminary segmentation prediction maps.

[0063] The output of each pixel can be regarded as a probability value, indicating the probability that the pixel belongs to a cell. Using the loss function, the segmentation index loss between the output segmentation result and the binary mask image input to the cell image semantic segmentation model is calculated. In specific implementation, the segmentation map predicted by the model can be compared with the real binary mask image, and the difference between the two can be calculated to obtain a loss value. Specifically, the segmentation loss often uses the Dice loss to calculate the similarity of the two binary images as a measure of loss. If the model's predicted segmentation result for a certain image is very close to the corresponding real mask, then the calculated Dice coefficient will be very close to 1, and the loss value at this time will be relatively small. Conversely, if the predicted result is very different from the real mask, the Dice coefficient will be smaller and the loss value will be larger. After this step, a corresponding Dice loss will be calculated for each pair of predicted segmentation maps and real masks in this batch.

[0064] The advantage of the above method is that it directly optimizes the performance of the model on the segmentation task. Through back propagation, the gradient of the parameters of the cell image semantic segmentation model is calculated. After obtaining the loss value, the back propagation algorithm can be used to calculate the contribution of each parameter in the model to the loss value, that is, the gradient of the parameter. This step will calculate the gradient of each parameter layer by layer from the output layer in reverse according to the loss value and the current parameters of the model. The calculated gradient actually indicates the adjustment direction of the model parameters. The gradient of each parameter indicates whether this parameter should be increased or decreased in order to reduce the loss value, and how much the change is. Gradient information is the basis for parameter updating.

[0065] Use the adaptive moment estimation optimizer to update the parameters of the cell image semantic segmentation model according to the calculated gradient to gradually minimize the segmentation index loss, and repeat the above steps until all preset iterations of training are completed. In this step, the adaptive moment estimation (Adam) optimizer will be used. After calculating the gradient of each parameter, the Adam algorithm will be used to update the parameters so that the overall segmentation effect is closer to the real effect. The Adam optimizer combines the gradient of each parameter calculated previously and the information of the gradient updated in the previous rounds to generate the size and direction of each parameter update. The key here is to set a suitable learning rate. Too high a learning rate will cause oscillation, and too small a learning rate will cause the update to be too slow. Generally, debugging is done through continuous experiments, or using some heuristic methods, such as step-by-step attenuation or cosine annealing. After the parameters are updated, the model has undergone a round of training, and then steps 1 to 5 will be repeated until all preset training rounds are completed. For example, if a total of 100 rounds of training are preset, then a total of 100 such loop iterations will be performed, each time using all batches of data, and in each iteration the average parameter gradient generated by all the data of the previous training is used to update the parameters until final convergence, so that the loss of the segmentation index is gradually reduced, so as to ultimately achieve a good segmentation effect.

[0066] See also Figure 4 , Figure 4 A flowchart of another method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation provided by an embodiment of the present invention. Figure 4 As shown, Figure 3 Compared with the method for calculating the confluence of mesenchymal stem cells shown in the figure, the difference is that Figure 3 The step S120 shown, i.e., the preprocessing operation including scale transformation and grayscale normalization is performed on the mesenchymal stem cell phase contrast microscopy image to be calculated, includes the following steps S121, S122 and S123: S121, cropping the phase contrast microscopic image of the mesenchymal stem cells to be calculated to obtain a plurality of partial images that meet a preset standard size; Since the pixel size of the original mesenchymal stem cell phase contrast microscopy image is large, such as 4000*3000, the direct processing calculation is large, and the original image has the situation that the middle area is bright and the edge area is dim when shooting, so it needs to be cropped. Assuming that the cell image size used in the training set is 672*512, this 4000*3000 image can be cropped into multiple local images of size 672*512. First, along the width direction of the image, with 672 pixels as a unit, from left to right, it is cropped without overlap, so that 6 cropping areas can be obtained, and the excess parts are discarded. Then, along the height direction of the image, with 512 pixels as a unit, it is cropped from top to bottom without overlap, so that 7 cropping areas can be obtained, and the excess parts are discarded. In this way, 42 local images that meet the preset standard size are obtained through two cropping operations. This processing reduces the amount of subsequent image processing calculations on the one hand, and also reduces the phenomenon of uneven illumination of the entire image. The cropped image is smaller than the original image, and is processed to meet preset standards, such as unifying the resolution of all images. Therefore, this step includes a scale transformation operation.

[0067] S122, calculating the mean and variance of all pixels of each local image; Calculate the mean and variance of all pixels in each local image. For each 672*512 local image obtained in the above steps, calculate the mean and variance of all pixel values. Because each local image contains only one color channel, that is, a grayscale image, with a total of 344064 pixels, the image can be treated as a one-dimensional array with 344064 values ​​for statistical calculations. For example, the mean value of all pixel values ​​in the first image is 150, and the variance is 45. The mean value of all pixel values ​​in the second image is 160, and the variance is 42. Because the light used to shoot the image is usually not uniform, it will be affected by various environmental conditions, resulting in different brightness levels of the cells shot, and the pixel values ​​will fluctuate, so it is reasonable to first calculate the mean and variance of the pixel values ​​of these images. The mean describes the trend of a set of data, and the variance describes the degree of discreteness of a set of data.

[0068] S123. Normalize the pixel values ​​of the local image according to the mean and the variance of the local image so that the mean is 0 and the variance is 1, so as to obtain a phase contrast microscopy image of mesenchymal stem cells to be calculated after scale transformation and grayscale normalization preprocessing operations.

[0069] The two indicators, the mean and the variance of different local images, are likely to be different. After these indicators are counted, it is convenient to perform numerical calculations on them in the subsequent steps to improve data quality. According to the mean and variance of the local image, the pixel values ​​of the local image are normalized so that the mean is 0 and the variance is 1, so as to obtain the mesenchymal stem cell phase contrast microscopy image to be calculated after the scale transformation and grayscale normalization preprocessing operations. After obtaining the mean and variance of each local image, each pixel value of each local image is transformed as follows: the pixel value is subtracted from the mean of the local image, and then divided by the standard deviation (arithmetic square root of the variance) of the local image to obtain a new pixel value. Because the standard deviation is greater than 0 and the mean is not less than 0, after this calculation, these values ​​will be distributed symmetrically with 0 as the center, and the range will be more compact. For example, a pixel value of the first image is 180, and the value obtained after the above transformation becomes approximately 0.667. The original pixel value is 120, and the pixel value becomes approximately -0.667. Because the order of pixels before and after the transformation does not change, the transformation relationship of this step can be expressed as a piecewise function of the original pixel value, that is, it will not change the size order of the original data, and will not make the originally identical pixels become different after the transformation. For the first picture, after all pixels have undergone this transformation, the mean of all its pixels is 0 and the variance is 1. For the second picture, the same transformation is performed, so that the mean of all pixel values ​​is 0 and the variance is 1. For each picture, the transformation is only related to the picture itself. Processing multiple pictures in sequence is an unrelated operation, that is, the calculation of the previous picture will not affect the subsequent results, and vice versa. And so on, all 42 local pictures are processed. In this way, no matter what the original local picture pixel distribution is, after this normalization operation, the mean of the pixel value of all local pictures becomes 0 and the variance becomes 1. This operation is a transformation of the pixel grayscale, the purpose of which is to make the model pay more attention to the content of the picture rather than the size of the pixel. This normalization operation is a commonly used data processing method, which can make the model more stable and adaptable. These processed images can be used as images to be calculated for subsequent calculation processing, and the results of the subsequent calculations can be applied to the original images.

[0070] See also Figure 5 , Figure 5 A flowchart of another method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation provided by an embodiment of the present invention. Figure 5 As shown, Figure 4 Compared with the method for calculating the confluence of mesenchymal stem cells shown in the figure, the difference is that Figure 4The step S130 shown, i.e., using the optimal cell image semantic segmentation model to segment the image of the mesenchymal stem cell phase contrast microscopy image to be calculated into cells and background, and summarizing and statistically analyzing the pixel ratios of the segmented cells to obtain the mesenchymal stem cell confluence, includes the following steps S131, S132, S133, S134 and S135: S131, inputting all local images of the mesenchymal stem cell phase contrast microscopy image to be calculated that have undergone scale transformation and grayscale normalization preprocessing operations into the optimal cell image semantic segmentation model to obtain an initial segmentation result; S132, normalizing the initial segmentation result by using a Sigmoid function to obtain a normalized segmentation result; S133, marking pixels whose normalized segmentation results are greater than the segmentation probability threshold as cell parts, otherwise marking them as background, so as to obtain a segmentation result with marking; S134, according to the size of the original image of the mesenchymal stem cell phase contrast microscopy image to be calculated that has not been preprocessed, splicing all the images in the segmentation results with the identifier, and processing the overlapping parts by taking the intersection to obtain a global segmentation result; S135 , performing pixel statistics of the cell part of the global segmentation result, and calculating the ratio of the pixels of all cells in the global segmentation result to the original image, as the cell confluence degree.

[0071] According to the optimal cell image semantic segmentation model and the phase contrast microscopy image of mesenchymal stem cells, the phase contrast microscopy image of mesenchymal stem cells to be calculated is cropped by the operation method of cropping to the same width and height as the training set data after downsampling, and all local images with consistent width and height are obtained. According to the requirements of the optimal cell image semantic segmentation model, the channel selection and zero mean normalization operation methods in the above steps are used to select channels and perform zero mean normalization on all local images to obtain all preprocessed local images. According to all preprocessed local images and the optimal cell image semantic segmentation model, all preprocessed local images are input one by one into the optimal cell image semantic segmentation model for inference to obtain the initial segmentation results of all local images. Sigmoid function, also known as S-type growth curve, is a common S-type function in biology. In machine learning and deep learning, Sigmoid function is often used as the activation function of neural network due to its monotonically increasing characteristics, mapping the input value to the (0,1) interval so that it can represent probability or perform binary classification. Obtain the Sigmoid normalization processing method and the segmentation probability threshold. If the data type of the pixel value in the initial segmentation result is not a floating point number, perform data type conversion to obtain a pixel value that is convenient for Sigmoid normalization processing; if the data type is already a floating point number, normalize the initial segmentation result through the Sigmoid function to obtain a normalized segmentation result. According to the normalized result, adopt a method greater than the segmentation probability threshold. If the pixel value in the normalized segmentation result is greater than the segmentation probability threshold, then mark the pixel as a cell part and obtain the marked cell part; otherwise, mark it as the background, and obtain a segmentation result with cell part and background identification. Obtain the original phase contrast microscopy image of the mesenchymal stem cells to be calculated. If the original image has no size information, read the image metadata; if the original image has size information, splice the segmentation results of all images according to the splicing method in the above steps of the original image size to obtain a spliced ​​segmented image. According to the operation method of taking the intersection of the overlapping parts in the above steps, the overlapping parts of the spliced ​​segmented images are judged. If a certain pixel is marked as a cell in the overlapping area, the cell identification of the pixel is retained in the spliced ​​segmented image to obtain the segmentation result after the intersection processing; otherwise, the pixel is marked as the background to obtain the global segmentation result. According to the global segmentation result and the method of counting the pixels of the cell part in the above steps, the category judgment is performed by traversing each pixel in the global segmentation result. If a certain pixel is marked as a cell, the value of the cell pixel counter is incremented to obtain the total number of cell pixels; otherwise, the background pixel counter is incremented to obtain the total number of global pixels. According to the total number of cell pixels and the total number of global pixels, the ratio calculation formula in the above steps is used to calculate the ratio of the total number of cell pixels to the total number of global pixels to obtain the cell confluence of mesenchymal stem cells.

[0072] Exemplarily, first, the original mesenchymal stem cell phase contrast microscopy image is cut into a local image with a width and height of 256 pixels to ensure that it is consistent with the size after downsampling of the training set data. Then, each cropped image is channel selected, the green channel is retained, and zero mean normalization is performed, that is, the pixel value is subtracted from the mean and divided by the standard deviation, so that the pixel value distribution range is [-1, 1]. Then, these pre-processed local images are input one by one into the trained cell image semantic segmentation model. The model is trained using the Dice loss function and the adaptive moment estimation optimizer. The inference result is normalized by the Sigmoid function, and the segmentation probability threshold is set to 0.5. Pixels greater than the threshold are regarded as cell parts, otherwise they are regarded as background. Subsequently, according to the size information of the original image, the segmentation results of all local images are spliced, and the overlapping parts are processed by taking the intersection method, that is, only pixels marked as cells in all overlapping areas are retained as cell parts, and the rest are classified as background. In the spliced ​​global segmentation map, each pixel is traversed and the total number of pixels marked as cells is counted. Assuming that the global image size is 1024x1024 pixels, the number of cell pixels is 300,000 and the number of background pixels is 724,000. Finally, the ratio of cell pixels to total pixels is calculated, that is, 300,000 / (300,000 + 724,000) ≈ 0.292, and the confluence of mesenchymal stem cells is 29.2%. Through this series of steps, a complete information processing flow from the original image to the calculation of cell confluence is realized.

[0073] Preferably, in the overlapping part, only the part in which two rows or two columns of pictures are cells is marked as the cell part, and the other parts are marked as the background part.

[0074] Specifically, if the sixth row and the seventh row, the fifth column and the sixth column overlap, then all the marks corresponding to these overlapping areas are obtained. If the marks of the two rows or columns of images in the overlapping area are all 1, then the area is determined to be a cell part. The marks of other non-overlapping areas are obtained. If the area has a mark 1, then it is determined that the pixel marks of these areas remain 1.

[0075] Exemplarily, 7*6 binary mask images that have been processed by Sigmoid normalization are obtained, where the size of each image is 672*512 pixels, and the segmentation probability threshold is set to 0.5. The probability value of each pixel is compared with the threshold. If it is greater than 0.5, it is marked as a cell part and assigned a value of 1, otherwise it is marked as a background part and assigned a value of 0. For example, when processing the first image, all its pixels are traversed, and the pixel probability value at the coordinate (100,200) is found to be 0.6, then the point is marked as a cell part. Then, all the binary mask images are seamlessly spliced ​​according to their original position information. During the splicing process, special attention is paid to the overlapping areas of rows and columns, such as the junction of the sixth and seventh rows, and the fifth and sixth columns. The logical AND operation of the pixels is used to ensure that only when the same position of the upper and lower or left and right adjacent mask images is marked as a cell, the corresponding position of the splicing result image is marked as a cell. Subsequently, based on the stitched overall binary mask image, the total number of pixels with a value of 1 is counted, recorded as N_cell, and the total number of pixels in the entire image N_total is calculated at the same time. The cell confluence is obtained by the formula Confluence = N_cell / N_total*100%. For example, if N_cell is 150,000 and N_total is 400,000, the confluence is 37.5%. In addition, the processing results of each step are logged, including the processing time of each sub-image, pixel statistics, etc., to facilitate subsequent analysis and optimization. Through this analysis and processing accurate to the pixel level, the accuracy of cell identification and the reliability of confluence calculation are ensured.

[0076] Before step S110, that is, the cell image semantic segmentation model is trained using the training set, and the trained cell image semantic segmentation model is tested and screened using the test set to obtain the optimal cell image semantic segmentation model, the following steps are also included: A cell image semantic segmentation model is constructed, wherein the cell image semantic segmentation model includes an image feature encoder, an image feature decoder, and an image semantic segmentor.

[0077] A cell image semantic segmentation model is constructed. The model structure includes three core components: image feature encoder, image feature decoder, and image semantic segmentor, which are used to extract features, fuse features, and segment images respectively.

[0078] See also Figure 6 , Figure 6 A schematic diagram of the network architecture of a cell image semantic segmentation model provided by an embodiment of the present invention is shown in FIG. Figure 6As shown, the cell image data in the training set is obtained, and the image feature encoder 6A in the cell image semantic segmentation model is input to obtain multi-level feature information in the image. The extracted multi-level feature information is integrated by the image feature decoder 6B to obtain the fused feature information. The image semantic segmenter 6C is used to process the fused feature information to obtain the probability value of each pixel belonging to the cell or the background. If the probability of each pixel belonging to the cell is greater than the preset threshold, then the pixel is classified as a cell, otherwise it is the background, and the semantic segmentation result of the cell image is obtained.

[0079] The method of constructing a cell image semantic segmentation model comprises: The image feature encoder 6A outputs image feature codes at four different depth levels through a 34-layer residual network; The image feature decoder 6B comprises four fully connected convolution modules and four segmentation convolution modules, which sample and accumulate the feature codes of four different depths output by the image feature encoder from the deep level to the shallow level respectively; The image semantic segmenter 6C includes a fully connected convolutional layer and an upsampling operation, and uses a linear activation function to output an image segmentation result.

[0080] like Figure 6 As shown, the image feature encoder module 6A uses a deep residual network (ResidualNetwork) ResNet-34 with 34 layers, which uses residual connections (skip connections) to allow information to skip over some layers. The main advantage of this design is that it alleviates the gradient vanishing problem in deep networks and helps the network train deeper models. The module includes an initial convolutional layer (Conv) and multiple residual blocks (ResNet block or Layer), each of which contains multiple convolutional layers and a shortcut connection. These residual blocks are distributed in multiple stages, and each stage changes the spatial resolution of the feature map by stride. The feature extraction capability of this module gradually increases from low-level to high-level. The initial layers extract simple low-level features such as edges and textures, while deeper networks capture more complex, semantically advanced features. As shown Figure 6As shown in the figure, the shallow feature f1 is extracted by the first layer Layer1, which usually captures basic visual information such as edges, textures, and colors; the middle-level feature f2 comes from the second middle layer Layer2 in the network, which captures more abstract features and may include local shapes, object parts, corners, etc.; the deep features come from the deeper third layer Layer3, which learns more semantic features, such as the overall shape of the object, category, and other high-level information; the final features come from the final stage of the network Layer4, which contains the final high-level semantic understanding of the input image. By acquiring features from multiple levels, the detailed information and high-level semantic information of the image can be obtained at the same time. Through the feature representation of different levels, the network can learn richer and more diverse image representations and enhance the generalization ability of the model.

[0081] And, if Figure 6 The image feature decoder module 6B shown includes a top-down pathway and a lateral connection. The top-down pathway first reduces the number of channels by half through a 1*1 convolution for the features of the four levels f1~f4, and then expands the size through interpolation upsampling so as to pass the high-level feature map to the low-level layer; while the lateral connections combine the low-resolution strong semantic features with the high-resolution weak semantic features through element-wise addition, thereby generating semantically rich feature maps at all scales.

[0082] Again, through the training set Figure 6 The image semantic segmenter 6C shown in the figure is trained and output to obtain the optimal cell image semantic segmentation model. The specific process is that the input image size is set to 672*512, the batch size is 4, the number of training iterations is 30, the initial learning rate of the adaptive moment estimation optimizer is 1e-4, and the other parameters of the optimizer use the default values ​​of the open source machine learning framework (Pytorch framework). During the experiment, the random seed is fixed, and the pre-trained weights are not used in the whole process. The loss function used for back propagation during training is Dice loss, which is defined as:

[0083] Among them, Dice_Coefficient is the Dice coefficient, which is used to evaluate the pixel similarity of two pictures and is defined as:

[0084] Among them, pred is the segmentation result output by the semantic segmentation model, truth is the cell image label mask image, and epsilon is a small constant added to prevent the denominator from being zero, which can be taken as 1e-7 here.

[0085] In this implementation, the GPU used for training the cell image semantic segmentation model is NVIDIA GeForce RTX3060 Ti 8GB, the CUDA version is 12.6, and the framework (Pytorch framework) version is 2.4.1.

[0086] See also Figure 7 and Figure 8 , Figure 7 This is an example diagram comparing the effects of the cell image semantic segmentation model on a standard dataset shown in an embodiment of the present invention. Figure 7 As shown, mask is the artificial segmentation effect, and pred is the segmentation effect of the trained cell image semantic segmentation model provided in this embodiment.

[0087] Figure 8 This is an example of the comparison of the effect of the cell image semantic segmentation model on umbilical cord mesenchymal stem cells photographed under a microscope. Figure 8 As shown, Raw Image is the original image taken, GrayScale Image is the preprocessed image, and Confluence is the effect diagram obtained after calculation and splicing by the cell image semantic segmentation model provided in this embodiment. The confluence of mesenchymal stem cells calculated in this example is 30.1%.

[0088] See also Fig. 9 , Fig. 9 This is a module diagram of another mesenchymal stem cell confluence calculation system based on image semantic segmentation provided by an embodiment of the present invention. Fig. 9 As shown, the mesenchymal stem cell confluence calculation system based on image semantic segmentation provided by an embodiment of the present invention includes: The optimal cell image semantic segmentation model training module 901 is used to train the cell image semantic segmentation model using the training set, and to test and screen the trained cell image semantic segmentation model using the test set to obtain the optimal cell image semantic segmentation model; An image processing module 902 is used to perform preprocessing operations including scale transformation and grayscale normalization on the mesenchymal stem cell phase contrast microscopic image to be calculated; The calculation processing module 903 is used to segment the image of the mesenchymal stem cell phase contrast microscopy image to be calculated into cells and background using the optimal cell image semantic segmentation model, and summarize and count the pixel proportions of the segmented cells to obtain the mesenchymal stem cell confluence.

[0089] Obtain a cell phase contrast microscopy image dataset. If the dataset images have scale and label inconsistencies, then perform scale transformation and label unification preprocessing operations on the dataset images. According to the preprocessing results, the dataset is divided into a training set and a test set. Construct a cell image semantic segmentation model. If the model consists of an image feature encoder, an image feature decoder, and an image semantic segmenter, then determine that the model construction is complete. Use the training set to train the cell image semantic segmentation model. If the loss function uses Dice loss, then before calculating the loss, perform Sigmoid normalization on the output of the cell image semantic segmentation model to obtain all outputs normalized to [0, 1] data as the probability that each pixel belongs to the cell part, calculate the predicted value, and use the real mask image label to calculate the Dice loss to obtain the loss value. Back-propagate gradient optimization model parameters through the adaptive moment estimation optimizer. Use grid search to select the hyperparameters of the segmentation model. If there are multiple hyperparameter combinations, then train each hyperparameter combination. After training, segment each cell image in the test set to obtain a predicted segmented image. The predicted segmentation image is compared with the real mask image to calculate the Dice coefficient. The average value of the Dice coefficient corresponding to the hyperparameter combination is obtained by statistics. According to the obtained Dice coefficient average value data, the highest Dice coefficient average value is determined, and the segmentation model obtained by training this hyperparameter combination is determined as the optimal cell image semantic segmentation model. Obtain the phase contrast microscopy image of mesenchymal stem cells to be calculated. If the image has uneven brightness when taken in a bright field, the image is preprocessed, and the original image is cut into the same size according to the width and height after the training set data is transformed to obtain an image with a width and height of 672*512. For each cropped image, a fixed channel is selected from the RGB three channels to obtain it as a grayscale image. According to the pixel distribution difference of the training data, zero mean normalization is performed to obtain a normalized image. Using the trained optimal cell image semantic segmentation model, the result of segmenting the phase contrast microscopy image of mesenchymal stem cells to be calculated into cells and background is obtained through model reasoning. By counting the proportion of the cell part in the entire image by pixel, the cell confluence data is obtained, and the calculation result of the mesenchymal stem cell confluence is determined based on this data.

[0090] For example, a cell phase contrast microscopy image dataset is obtained, such as 1,000 cell images, each with an original resolution of 1024×768. The images are uniformly cropped to 672×512 through scale transformation to ensure data consistency. A cell image semantic segmentation model is constructed using the U-Net architecture, in which the encoder uses ResNet34 to extract features, the decoder gradually upsamples to restore the image size, and finally outputs the binary classification result through 1×1 convolution. The model is trained using the training set, and the loss function uses Dice loss. The model output is Sigmoid normalized, and the output value is limited to the [0,1] interval, indicating the probability that the pixel belongs to the cell. The Dice loss of the predicted value and the true mask image is calculated, and the Adam optimizer is used for back propagation, the learning rate is set to 0.001, the batch size is 8, and 100 epochs are iterated. Hyperparameters were selected through grid search, such as learning rate [0.001, 0.0001] and batch size [8, 16]. After each combination was trained, the Dice coefficient was calculated on the test set. It was found that when the learning rate was 0.001 and the batch size was 8, the average Dice coefficient was the highest, which was 0.92. This combination was determined to be the optimal model. Phase contrast microscopy images of mesenchymal stem cells to be calculated were obtained, such as an image with a resolution of 1024×768, which was cropped to 672×512. The green channel in the RGB channel was selected as the grayscale image, and zero mean normalization was performed to eliminate the problem of uneven brightness. The preprocessed image was segmented using the optimal model to obtain a binary image of cells and background. The proportion of cells was counted by pixel. For example, the number of cell pixels was 300,000 and the total number of pixels was 512×672=349,568. The confluence was calculated to be 300,000 / 349,568≈0.86, and the cell confluence of the image was determined to be 86%.

[0091] In summary, the present invention discloses an automated solution for calculating the confluence of mesenchymal stem cells. The core business scenario problem is how to automatically calculate the confluence of mesenchymal stem cells from phase contrast microscopic images accurately and efficiently, especially when the cell boundaries are unclear and the image quality is uneven.

[0092] Specifically, the present invention discloses a method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation. This method proposes an automated and accurate solution to the problem of confluence assessment during mesenchymal stem cell culture. The present invention first constructs and trains a cell image semantic segmentation model, which uses a deep residual network to extract multi-level features, performs feature fusion through upsampling and lateral connections, and finally generates a segmentation result of the cell image. On this basis, the present invention preprocesses and segments the phase-contrast microscopy images of mesenchymal stem cells for which the confluence is to be calculated, optimizes the segmentation results through post-processing steps, and finally obtains the confluence by counting the cell pixel ratio. This method not only improves the accuracy and efficiency of mesenchymal stem cell confluence calculation, but also provides strong support for automated monitoring and quality control of the cell culture process, which is of great significance for promoting stem cell research and application.

[0093] The present invention constructs a deep learning model, which extracts multi-level features based on an image feature encoder of a 34-layer residual network, fuses these features through an image feature decoder including four fully connected convolutional modules and four segmentation convolutional modules, and finally generates segmentation results by an image semantic segmentor. The model is trained and verified using a standard data set containing a variety of cell phase contrast microscopy images, and an adaptive moment estimation optimizer is used in each training batch to update the model parameters to minimize the loss of segmentation indicators. The significant technical effect of this solution is that it realizes the automation and high precision of mesenchymal stem cell confluence calculation.

[0094] The description of the above embodiments is only used to help understand the technical solutions and core ideas of the present application. Ordinary technical personnel in this field should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or replace some of the technical features therein with equivalents. However, these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation, characterized in that: The following steps are involved: Using the training set to train the cell image semantic segmentation model, and using the test set to test and screen the trained cell image semantic segmentation model to obtain the optimal cell image semantic segmentation model; Performing preprocessing operations including scale transformation and grayscale normalization on the phase contrast microscopic images of mesenchymal stem cells to be calculated; The optimal cell image semantic segmentation model is used to segment the image of the mesenchymal stem cell phase contrast microscopy image to be calculated into cells and background, and the pixel ratios of the segmented cells are summarized and counted to obtain the confluence of the mesenchymal stem cells.

2. The method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation according to claim 1, characterized in that: The method of training the cell image semantic segmentation model using the training set and testing and screening the trained cell image semantic segmentation model using the test set to obtain the optimal cell image semantic segmentation model further includes: Get standard data sets; The standard data set includes several cell phase contrast microscopic images; After preprocessing, the standard data set is divided into the training set and the test set.

3. The method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation according to claim 2, characterized in that: After preprocessing the standard data set, the data set is divided into a training set and a test set, including: Performing preprocessing operations including scale transformation and label unification on the standard data set; And, the standard data set that has undergone the preprocessing operations of scale transformation and label unification is divided into the training set and the test set.

4. The method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation according to claim 3, characterized in that: The standard data set is subjected to preprocessing operations including scale transformation and label unification, including: Performing a scale transformation preprocessing operation on the standard data set, and transforming the width and height of each image in the standard data set to an integer multiple of 32; Furthermore, a unified label preprocessing operation is performed on the standard data set, and the original format labels of the images in the standard data set are converted into corresponding binary mask images.

5. The method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation according to claim 4, characterized in that: The method of training the cell image semantic segmentation model using the training set and testing and screening the trained cell image semantic segmentation model using the test set to obtain the optimal cell image semantic segmentation model includes: Using the training set, the cell image semantic segmentation model is trained for a preset number of iterations according to a preset batch processing standard; After each iteration of training, the segmentation index loss of the training set corresponding to this round is recorded; The parameters of the cell image semantic segmentation model with the smallest segmentation index loss during the training process are selected as the final model parameters to obtain the corresponding optimal cell image semantic segmentation model.

6. The method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation according to claim 5, characterized in that: The method of using the training set to train the cell image semantic segmentation model for a preset number of iterations according to a preset batch processing standard includes: In each training batch, the images in the training set and the corresponding binary mask images are input into the cell image semantic segmentation model according to a preset batch processing standard; Calculating the output segmentation result of the cell image semantic segmentation model on the training set through forward propagation; Using a loss function, calculating the segmentation index loss between the output segmentation result and the binary mask image input to the cell image semantic segmentation model this time; Calculating the gradient of the parameters of the cell image semantic segmentation model through back propagation; Using an adaptive moment estimation optimizer, the parameters of the cell image semantic segmentation model are updated according to the calculated gradient to gradually minimize the segmentation index loss, and the above steps are repeated until all preset iteration rounds of training are completed.

7. The method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation according to claim 1, characterized in that: The preprocessing operation including scale transformation and grayscale normalization is performed on the mesenchymal stem cell phase contrast microscopic image to be calculated, including: Cutting the phase contrast microscopic image of the mesenchymal stem cells to be calculated to obtain a plurality of partial images that meet a preset standard size; Calculating the mean and variance of all pixels of each of the local images; According to the mean and the variance of the local image, the pixel values ​​of the local image are normalized so that the mean is 0 and the variance is 1, so as to obtain the mesenchymal stem cell phase contrast microscopy image to be calculated after scale transformation and grayscale normalization preprocessing operations.

8. The method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation according to claim 7, characterized in that: The method of using the optimal cell image semantic segmentation model to segment the image of the mesenchymal stem cell phase contrast microscopy picture to be calculated into cells and background, and summarizing and counting the pixel proportions of the segmented cells to obtain the mesenchymal stem cell confluence degree includes: Inputting all local images of the mesenchymal stem cell phase contrast microscopy image to be calculated that have undergone scale transformation and grayscale normalization preprocessing operations into the optimal cell image semantic segmentation model to obtain an initial segmentation result; Normalizing the initial segmentation result by using a Sigmoid function to obtain a normalized segmentation result; The pixels whose normalized segmentation results are greater than the segmentation probability threshold are marked as cell parts, otherwise they are marked as background, so as to obtain a segmentation result with a mark; According to the size of the original image of the phase contrast microscopic image of the mesenchymal stem cells to be calculated that has not been preprocessed, all images in the segmentation results with the identifier are spliced, and the overlapping parts are processed by taking the intersection to obtain a global segmentation result; Perform pixel statistics of the cell part of the global segmentation result, and calculate the ratio of the pixels of all cells in the global segmentation result to the original image as the cell confluence; The segmentation probability threshold is 0.5, and the overlapping part in which only two rows or two columns of images are cells is marked as the cell part, and the other parts are marked as the background part.

9. The method for calculating the confluence of mesenchymal stem cells based on image semantic segmentation according to claim 1, characterized in that: The method of training the cell image semantic segmentation model using the training set and testing and screening the trained cell image semantic segmentation model using the test set to obtain the optimal cell image semantic segmentation model further includes: Constructing a cell image semantic segmentation model, wherein the cell image semantic segmentation model includes an image feature encoder, an image feature decoder, and an image semantic segmentor; The method of constructing a cell image semantic segmentation model comprises: The image feature encoder outputs image feature codes at four different depth levels through a 34-layer residual network; The image feature decoder comprises four fully connected convolution modules and four segmentation convolution modules, which sample and accumulate the feature codes of four different depths output by the image feature encoder from the deep level to the shallow level respectively; The image semantic segmentor includes a fully connected convolutional layer and an upsampling operation, and uses a linear activation function to output the image segmentation result.

10. A mesenchymal stem cell confluence calculation system based on image semantic segmentation, characterized in that: include: An optimal cell image semantic segmentation model training module is used to train the cell image semantic segmentation model using a training set, and to test and screen the trained cell image semantic segmentation model using a test set to obtain an optimal cell image semantic segmentation model; An image processing module, used for performing preprocessing operations including scale transformation and grayscale normalization on the mesenchymal stem cell phase contrast microscopic images to be calculated; The calculation processing module is used to segment the image of the mesenchymal stem cell phase contrast microscopy picture to be calculated into cells and background using the optimal cell image semantic segmentation model, and summarize and count the pixel proportions of the segmented cells to obtain the confluence of the mesenchymal stem cells.

Images