Cell growth dynamics prediction method and device based on space-time image sequence

Abstract

The invention provides a cell growth dynamics prediction method and device based on a space-time image sequence, and belongs to the technical field of computer vision and deep learning. The method comprises the following steps: acquiring a multi-focal-plane space-time image sequence containing a cell complete growth process; preprocessing and normalizing images in the image sequence to obtain a final test image sequence; and inputting the test image sequence into a cell growth dynamics prediction model consisting of a multi-scale feature extraction module and a time sequence modeling module which are connected in sequence to obtain a cell growth stage identification result corresponding to each time point image in the sequence so as to realize cell growth dynamics prediction. The method can effectively solve the problems of poor adaptability, insufficient stability, high labeling cost and the like when a traditional model is used for processing the dynamic medical image sequence, and has remarkable advantages and application prospects in the aspect of dynamic prediction of the cell growth space-time image sequence.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and deep learning, and particularly relates to a cell growth dynamics prediction method and device based on a space-time image sequence. BACKGROUND

[0002] In biological and medical research, cell growth dynamics has long been a focus as an important entry point to reveal the mechanisms of cell behavior and the laws of tissue development. Cells exhibit highly complex spatiotemporal evolution characteristics in processes such as proliferation, migration, and apoptosis. These dynamic changes are not only closely related to individual development, but also closely related to the mechanisms of major diseases such as cancer. With the help of advanced microscopes such as inverted confocal microscopes, researchers can obtain cell image data containing multi-focal plane, long-term, and multi-scale information, providing rich data support for quantitative analysis and model reconstruction of cell behavior.

[0003] With the rapid evolution of deep learning and artificial intelligence, image recognition and understanding technologies have made breakthrough progress. Especially in the field of natural image and medical image analysis, models that integrate spatial and temporal dimensions are constantly emerging, such as spatiotemporal attention networks, dynamic convolutional networks, and their variants, which have significantly improved the feature capture capability in time-series images. However, there are still significant challenges in directly applying such methods to high-dimensional image sequence modeling of cell growth processes, which can be summarized in the following three aspects:

[0004] First, poor adaptability. Existing models usually rely on fixed-length, equally spaced, and continuous complete image frames as input, making it difficult to deal with data irregularities in cell imaging processes, such as uneven sampling time, missing image frames, and focal drift. This strong dependence on ideal data severely limits the robustness and adaptability of the model when faced with actual complex and variable medical image data, often missing important biological processes, affecting the accurate identification of dynamic mechanisms, and severely limiting the generalization ability of the model.

[0005] Second, insufficient stability. Traditional models are mostly based on static feature extraction and lack the ability to model the temporal evolution of cells, making it difficult to capture long-term dependencies and nonlinear changes in cell behavior. In addition, some three-dimensional or time-series model structures are complex and computationally intensive, making it difficult to meet the high-throughput and high-precision requirements of cell dynamics research.

[0006] Third, high labeling cost. Existing deep learning methods highly depend on large-scale and accurately labeled datasets, while cell image data often has labeling difficulties and sample scarcity issues. The identification and classification of cell behavior require a wealth of professional knowledge and a large amount of manual input, which restricts the practical deployment and promotion of the model in clinical and research scenarios. SUMMARY

[0007] The present application aims to overcome the shortcomings of the prior art and provide a cell growth dynamics prediction method and device based on a spatiotemporal image sequence. The present application can effectively solve the problems of poor adaptability, insufficient stability, and high labeling cost when traditional models process dynamic medical image sequences, and has significant advantages and application prospects in cell growth spatiotemporal image sequence dynamics prediction.

[0008] The first aspect of the present application provides a cell growth dynamics prediction method based on a spatiotemporal image sequence, comprising:

[0009] acquiring a spatiotemporal image sequence containing a complete cell growth process, wherein each image at a time point in the image sequence is a multi-focal plane image;

[0010] after pre-processing and normalization of the images in the image sequence, obtaining a final test image sequence;

[0011] inputting the test image sequence into a preset cell growth dynamics prediction model to obtain a cell growth stage recognition result corresponding to each image at a time point in the test image sequence, so as to realize cell growth dynamics prediction; wherein the model comprises a multi-scale feature extraction module and a time series modeling module connected in sequence, the multi-scale feature extraction module adopts a ResNet-50 network, and the time series modeling module adopts a gated recurrent unit (GRU).

[0012] In one specific embodiment of the present application, the pre-processing includes:

[0013] denoising the images in the image sequence;

[0014] detecting and removing empty frames from the denoised image sequence; wherein if there is an empty frame in the multi-focal plane image at any time point in the image sequence, all images at the time point are removed from the image sequence;

[0015] performing cell region identification and image cropping on the images in the image sequence after removing the empty frames to obtain an image sequence containing only cell regions.

[0016] In one specific embodiment of the present application, before inputting the test image sequence into the preset cell growth dynamics prediction model, the method further comprises:

[0017] training the cell growth dynamics prediction model;

[0018] The training of the cell growth dynamics prediction model comprises:

[0019] 1) obtain a cell growth spatiotemporal image sequence for original training and a cell growth spatiotemporal image sequence for transfer learning respectively; wherein the two image sequences both contain the complete process of cell growth; the cell types corresponding to the two image sequences are consistent with the cell type corresponding to the test image sequence;

[0020] perform cell growth kinetics annotation on the cell growth spatiotemporal image sequence for original training frame by frame;

[0021] 2) pre-process the images in the two image sequences obtained in step 1);

[0022] 3) group the cell growth spatiotemporal image sequence for original training after pre-processing and the corresponding annotation results into samples, and then randomly divide all the samples into an initial training set and an initial validation set according to a set proportion;

[0023] 4) perform normalization and enhancement processing on the images in the initial training set and the cell growth spatiotemporal image sequence for transfer learning after pre-processing, to obtain a final training set and an optimized cell growth spatiotemporal image sequence for transfer learning, wherein the optimized cell growth spatiotemporal image sequence for transfer learning is used as a transfer learning data set;

[0024] perform normalization processing on the images in the initial validation set to obtain a final validation set;

[0025] 5) construct a cell growth kinetics prediction model;

[0026] 6) complete the training process of the cell growth kinetics prediction model by using the training set, the validation set and the transfer learning data set, to obtain a trained cell growth kinetics prediction model.

[0027] In a specific embodiment of the present application, the training process of the cell growth kinetics prediction model comprises:

[0028] perform original training on the multi-scale feature extraction module by using the training set and the validation set;

[0029] perform transfer training on the multi-scale feature extraction module after original training by using the transfer learning data set;

[0030] after the transfer training of the multi-scale feature extraction module is completed, freeze the model parameters of the multi-scale feature extraction module, and perform distributed training on the time series modeling module.

[0031] In a specific embodiment of the present application, it further comprises:

[0032] Before the original training of the multi-scale feature extraction module using the training set and the validation set, a linear classification head is added behind the ResNet-50 network, and the output dimension of the linear classification head is the dimension of the cell growth kinetics feature.

[0033] In one specific embodiment of the application, it further comprises:

[0034] Before the start of the transfer training, the linear classification head of the ResNet-50 network that has completed the original training is removed, and then a SimSiam-based contrast learning framework is connected behind the ResNet-50 network, the contrast learning framework includes a projection head submodule and a prediction head submodule; the projection head submodule and the prediction head submodule are connected in turn behind the ResNet-50 network.

[0035] In one specific embodiment of the application, it further comprises:

[0036] After the completion of the transfer training, the projection head submodule and the prediction head submodule connected behind the ResNet-50 network are removed, the output of the ResNet-50 network is connected to the GRU, and the parameters of the ResNet-50 network are frozen;

[0037] Then set three different tasks to complete the training of the time series modeling, the tasks include: future frame prediction task, masking frame reconstruction task and time series smoothing constraint task; the joint loss function of the time series modeling module training is the weighted sum of the loss functions of the three tasks.

[0038] The second aspect embodiment of the application proposes a cell growth kinetics prediction device based on a spatiotemporal image sequence, comprising:

[0039] An image sequence acquisition module is configured to acquire a spatiotemporal image sequence containing a complete cell growth process, and each image at a time point in the image sequence is a multi-focal plane image.

[0040] An image sequence processing module is configured to obtain a final test image sequence after pre-processing and normalizing the images in the image sequence.

[0041] A cell growth kinetics prediction module is configured to input the test image sequence into a preset cell growth kinetics prediction model to obtain a cell growth stage recognition result corresponding to each image at a time point in the test image sequence, so as to realize cell growth kinetics prediction; wherein the model comprises a multi-scale feature extraction module and a time series modeling module connected in turn, the multi-scale feature extraction module adopts a ResNet-50 network, and the time series modeling module adopts a gated recurrent unit (GRU).

[0042] The third aspect of the present application provides an electronic device, comprising:

[0043] at least one processor; and a memory connected with the at least one processor in communication;

[0044] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the cell growth dynamics prediction method based on the spatiotemporal image sequence.

[0045] The fourth aspect of the present application provides a computer readable storage medium storing computer instructions for causing the computer to execute the cell growth dynamics prediction method based on the spatiotemporal image sequence.

[0046] The present application has the characteristics and advantages of:

[0047] The present application adopts a two-level modeling strategy of "spatial reference-time sequence modeling", which can realize the extraction of the dynamics of the whole process of cell growth. Through the use of self-supervised contrast learning in the training stage, the model can effectively generalize on unannotated or less annotated data. The present application integrates a pre-trained two-dimensional convolutional neural network and a gated recurrent unit in the overall architecture, corresponding to the two stages of spatial feature extraction and temporal dependence modeling, respectively. In the transfer learning training of the model, self-supervised contrast learning based on a twin network architecture is used to make the model have strong robustness and good generalization performance. The model can integrate spatiotemporal image information and automatically capture the feature extraction method of the evolution pattern of cell behavior, which will provide a more efficient and more generalized technical path for cell growth dynamics modeling under low annotation conditions and provide a more objective evaluation index for cell culture.

[0048] Overall, the present application not only considers the diversity of changes in cell growth images in the time and space dimensions, but also has good generalization for image data from different sources, which can effectively solve the problems of poor adaptability, insufficient stability, and weak generalization faced by traditional models when processing dynamic medical image sequences, and has significant advantages and application prospects in cell growth spatiotemporal image sequence dynamics prediction. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The present application provides a cell growth dynamics prediction method based on a spatiotemporal image sequence. DETAILED DESCRIPTION

[0050] The present application provides a cell growth dynamics prediction method based on a spatiotemporal image sequence.

[0051] An embodiment of the first aspect of the present application provides a cell growth dynamics prediction method based on a spatiotemporal image sequence, the overall process of which is shown in Figure 1 as follows:

[0052] An image sequence containing a complete cell growth process is acquired, and each image at a time point in the image sequence is a multi-focal plane image.

[0053] After the images in the image sequence are preprocessed and normalized, a final test image sequence is obtained.

[0054] The test image sequence is input into a preset cell growth dynamics prediction model to obtain a cell growth stage recognition result corresponding to each image at a time point in the test image sequence, so as to realize cell growth dynamics prediction. The model includes a multi-scale feature extraction module and a time series modeling module connected in sequence. The multi-scale feature extraction module adopts a ResNet-50 network, and the time series modeling module adopts a gated recurrent unit (GRU).

[0055] In one specific embodiment of the present application, the cell growth dynamics prediction method based on the spatiotemporal image sequence is used to determine the stage state of cell growth and division, and includes the following steps:

[0056] 1) Training stage.

[0057] 1-1) Acquire a cell growth spatiotemporal image sequence for original training and a cell growth spatiotemporal image sequence for transfer learning, respectively, and the specific steps are as follows:

[0058] 1-1-1) Acquire a cell growth spatiotemporal image sequence for original training and label it. In this embodiment, multi-focal plane images containing a complete cell growth process are collected to form a cell growth spatiotemporal image sequence for original training. The cells corresponding to each sequence belong to the same type.

[0059] In this embodiment, the acquisition of each cell sample of the collected images follows strict biological experimental norms, ensuring the activity and integrity of the cells. At each time point, multiple focal plane cell images are acquired by a specific imaging device (such as a laser confocal microscope, etc.), which clearly presents the morphology, distribution, and interaction of cells at different depth levels. Each focal plane image is related to each other, and together they depict the growth state of cells in three-dimensional space. During imaging, the key parameters of the imaging device are strictly set. For example, the resolution of the microscope needs to reach the level of clearly distinguishing the subcellular structures of individual cells, such as optical resolution which can be set between tens of nanometers and hundreds of nanometers; the frame rate of imaging is adjusted according to the growth rate of cells, for cells that grow rapidly, the frame rate can be set to several frames per minute, while for slow-growing cells, the frame rate can be appropriately reduced, but it needs to ensure that key changes in cell growth can be captured; the focal plane spacing needs to be reasonable to ensure that the spatial information of the cells between adjacent focal planes is completely covered, and important details are not missed due to excessive spacing, generally the focal plane spacing can be controlled at the micron level. To ensure the sufficiency and effectiveness of the original training and transfer learning, a sufficient number of cell growth spatiotemporal image sequences need to be acquired. In this embodiment, the number of sequences used for original training is not less than 500.

[0060] On this basis, the present embodiment records the cell growth kinetics frame by frame for the collected image sequences in reference to the recognized cell growth cycle classification standard, clearly defines the biological state of the cells at each time point, and establishes a one-to-one correspondence between the state and the image acquisition time point in the sequence.

[0061] In a specific embodiment of the present application, only the typical development stages of embryos are annotated, and according to the international embryonic morphology evaluation system, the development process of embryos is divided into multiple clear stages, including the fertilized egg stage, the 2-cell stage, the 4-cell stage, the 8-cell stage, the morula stage, the early blastocyst stage, the expanded blastocyst stage, etc. a total of 12 cell growth stages. The annotation results are saved in the form of information structure with time sequence, forming a standardized time sequence label set. The annotation data of each cell is independently stored in the corresponding.csv file, which should include time stamp, image number, growth stage category and other key fields to ensure the integrity and traceability of data organization, facilitating the fine management of subsequent model training and evaluation.

[0062] 1-1-2) Obtain cell growth spatiotemporal image sequences for transfer learning.

[0063] In this example, by setting different cell culture conditions or using different collection equipment than step 1-1-1), a multi-focal plane image sequence containing the entire cell growth process is obtained for transfer learning, wherein each cell growth spatio-temporal image sequence for transfer learning corresponds to the same type of cell as the cell growth spatio-temporal image sequence for original training, and the number of focal planes in the two sequences is consistent.

[0064] In this embodiment, the number of sequences for transfer learning is not less than 50, and the cells for transfer learning and original training are of the same type, but the data collection equipment or cell culture conditions should be different, and no kinetics recording is required.

[0065] In one specific embodiment of the present application, the cell growth spatio-temporal image sequence set for original training is derived from a publicly available embryonic development time-lapse photography image database. This database is one of the time-lapse image data resources currently available in the public domain, covering a relatively complete development stage and detailed annotation information, containing a total of 704 complete embryonic development videos. The annotation information provided is based on a unified morphokinetic event definition standard, covering 16 key development stages of embryos from fertilization to blastocyst hatching, containing a total of 704 complete embryonic development videos. All annotations are completed by a team of embryologists with professional experience. The single-frame image resolution is 500x500 pixels, and the format is.jpeg. In this embodiment, all image frames of the video on different focal planes in the database are extracted to form image sequences. The images in any image sequence of this embodiment at the same time point are collected on 7 different focal planes, and the total number of images in the image sequence set is about 2.4 million. This spatio-temporal image sequence data has high spatial resolution and temporal continuity.

[0066] In one specific embodiment of the present application, the cell growth spatio-temporal image sequence for transfer learning is collected using time-lapse microscopy technology. Specifically, a multi-focal plane time sequence image sequence of a single embryonic cell in vitro culture is continuously collected in the same culture environment, ensuring that the image sequence completely covers the growth, division, and evolution of the cell, and the number of focal planes is consistent with that of the sequence for original training. To ensure training quality, the collected data must meet the following requirements. The cell source should have high consistency, preferably of the same cell type, to exclude interference caused by too large differences in growth rate, morphological evolution, and metabolic behavior between different cell types. To support the model's ability to transfer learning, it is recommended to collect complete spatio-temporal sequence images of at least 50 groups of cells, with each cell as a unit, stored in separate folders, and the collection time point of each frame of image, the corresponding focal plane position, and other metadata are recorded simultaneously. The single-frame image resolution requirement is not less than 500x500 pixels, and the format is.jpeg.

[0067] In this embodiment, the collected are human in vitro fertilization early embryo cells, all from the same standard operating system under the embryo culture process, with similar morphological structure, consistent growth conditions and other characteristics. This embodiment collects 100 independent embryo cell image sequences.

[0068] 1-2) Preprocess the images in the cell growth spatiotemporal image sequence obtained in step 1-1), the specific steps are as follows:

[0069] 1-2-1) Denoise the images in the cell growth spatiotemporal image sequence for original training and the cell growth spatiotemporal image sequence for transfer learning, respectively, to obtain the corresponding denoised cell growth spatiotemporal image sequence.

[0070] In this embodiment, Gaussian filtering is used as the main linear denoising method in the preprocessing stage. By performing weighted convolution operation on the original image, the high-frequency noise components in the image are effectively suppressed, and the background disturbance introduced by the imaging device or environmental interference is reduced. The denoised image is smoothed and optimized in overall brightness and structural level, suitable for subsequent feature extraction operation.

[0071] To further improve the retention ability of cell edges and internal structures in complex background conditions, nonlinear smoothing algorithms such as bilateral filtering and median filtering can be selectively introduced according to the specific characteristics of the image. The above nonlinear method has the ability to consider both image spatial information and gray scale similarity, and can effectively protect the texture details and boundary contours of the cells while removing noise.

[0072] The whole image processing flow aims to maximize the retention of key biological structural features of cells, focusing on maintaining the morphological profile, edge structure and internal texture information in the image, and avoiding important discriminant features from being weakened or lost in the denoising process. The image sequence processed by the above multi-level filtering has high signal-to-noise ratio and structural integrity, and can be used as effective input data for subsequent deep learning models, significantly improving the robustness and accuracy of the model in spatial structure analysis and growth state discrimination tasks.

[0073] 1-2-2) Detect the empty frames of the two cell growth spatiotemporal image sequences denoised in step 1-2-1), and remove the empty frames to obtain two preliminary optimized cell growth time sequence image sequences.

[0074] In this embodiment, during the cell growth time sequence image acquisition process, cells may be removed, transferred or growth cycle terminated at certain stages, often resulting in "empty frames" in the image sequence, which only contain background culture dishes and no longer contain valid cell structures. To ensure the biological effectiveness and temporal consistency of the input images in subsequent modeling, empty frame detection and removal operations are required, and the timestamp information is updated simultaneously.

[0075] In the embodiment, the traditional visual method based on image brightness, contrast and edge intensity is used to identify the empty frame for each frame image. First, the denoised each frame image I t The global average brightness μ t , the contrast σ t , and the edge pixel density E t are calculated, wherein the edge information is extracted by Canny edge detection operator and the proportion of non-zero edge pixels is normalized:

[0076]

[0077]

[0078] Wherein, the subscript t represents the time point corresponding to the image, H and W are the height and width of the single frame image, τ is the edge intensity threshold, which is 0.5 in the embodiment, is the Canny edge detection operator, 1[ ] is the indicator function, I t (i,j) represents the value of the pixel point with horizontal coordinate i and vertical coordinate j.

[0079] The contrast threshold σ min and the edge pixel density threshold E min are set, and the following conditions are determined:

[0080] If the frame image satisfies:

[0081] σ t < σ min

[0082] E t < E min

[0083] The frame image is considered as an empty frame with low contrast and low edge density, and it can be determined as an invalid image without cells. In the embodiment, the value of the contrast threshold σ min is 10, and the value of the edge pixel density threshold E min is 0.01.

[0084] After calculation, the detected empty frame is removed, and the time stamp information corresponding to the empty frame is also removed. The time index of each frame image in the optimized cell growth time sequence image sequence accurately corresponds to the actual shooting time in the original sequence, and the strict alignment relationship between the image and the time is maintained. It should be noted that if an empty frame is detected in the image at a certain time point in the denoised image sequence, all images at the time point are removed from the image sequence.

[0085] After the treatment of this step, a continuous, non-empty frame interference, time sequence consistent and biological structure intact cell growth spatio-temporal image sequence can be obtained, which provides a high-quality, clean data basis for subsequent deep model feature learning in spatial structure and time dynamics.

[0086] 1-2-3) The images in the two preliminary optimized cell growth spatio-temporal image sequences obtained in step 1-2-2) are subjected to cell region recognition and image cropping, respectively, to obtain a pre-processed cell growth spatio-temporal image sequence for original training and a cell growth spatio-temporal image sequence for transfer learning.

[0087] In this embodiment, a deep learning-based target detection algorithm is used to accurately locate the cell regions in the images of the two preliminary optimized cell growth spatio-temporal image sequences, so as to obtain a cell growth spatio-temporal image sequence containing only cell regions. Such an algorithm directly predicts the position and category information of the target in the image through an end-to-end regression mechanism, generating a bounding box containing the target region.

[0088] In one specific embodiment of the present application, the YOLOv8n algorithm is used to extract the spatial boundaries of cells in each image in the image sequence, and based on this, the original image is cropped to retain only the effective region image containing the cells and eliminate the background region and other irrelevant interference information. It should be noted that for the size and pixel loss caused by cropping, necessary image size normalization and edge information repair processing are required. Specifically, to solve the problem of inconsistent image sizes caused by cropping, all cropped images are uniformly adjusted to a fixed resolution of 512x512 through Bilinear Interpolation to ensure the consistency of the dimensions of the subsequent model input. At the same time, to preserve as much structural information of the cell edges as possible, the boundaries are appropriately expanded based on the cropped region, with a buffer zone of 5% image height added above and below to avoid truncating the cell edges or key information of the surrounding microenvironment due to the compactness of the detection box.

[0089] This processing method effectively improves the discriminative feature density of the subsequent training samples, reduces the influence of redundant information on the model learning process, and helps to enhance the model's perception of cell morphology, structure, and state changes, thereby improving the stability and accuracy of subsequent feature extraction and classification tasks.

[0090] 1-3) Based on the results of step 1-2), the pre-processed cell growth spatio-temporal image sequence for original training and its corresponding annotation results are combined into samples, and then all samples are divided into an initial training set and an initial validation set.

[0091] In this embodiment, the pre-processed cell growth spatio-temporal image sequence for original training is taken as the basic unit to ensure strict alignment of the image frame and its corresponding growth stage label in the time dimension. Each group of image sequences contains a unique identifier, a timestamp index, an image path, and a corresponding stage label sequence, forming a structured sample entity.

[0092] On this basis, all samples are randomly divided into an initial training set (80% in one specific embodiment of the present application) and an initial validation set (20% in one specific embodiment of the present application) in proportion, following the principles of time continuity and sample independence. The division process is strictly operated in units of files to ensure that the images and labels of the same cell are not split into different subsets, fundamentally avoiding label information leakage and data redundancy, and ensuring the fairness of the training process and the scientific nature of the evaluation results. The division results record the sample paths and their belonging subsets in a standard format, which are used for subsequent data loading, batch processing, and model input configuration.

[0093] 1-4) Normalizing and enhancing the images in the initial training set obtained in step 1-3) and the pre-processed cell growth spatio-temporal image sequence for transfer learning obtained in step 1-2) respectively to obtain a final training set and an optimized cell growth spatio-temporal image sequence for transfer learning as a transfer learning data set;

[0094] Normalizing the images in the initial validation set obtained in step 1-3) to obtain a final validation set.

[0095] In this embodiment, to improve the generalization ability and robustness of the model in the cell growth stage recognition task, the images in the initial training set or the pre-processed cell growth spatio-temporal image sequence for transfer learning are normalized and enhanced to simulate the possible morphological, positional, and illumination variations of cells under different imaging conditions, thereby improving the adaptability of the model to real scenarios.

[0096] In the normalization stage, the linear intensity of the image pixels is uniformly adjusted. The specific operation in this embodiment is to map the pixel value p of each image from the interval [0, 255] to the interval [0, 1], or perform Z-score standardization processing:

[0097]

[0098] where μ and σ are the mean and standard deviation of the image pixels in the training set or the pre-processed cell growth spatio-temporal image sequence for transfer learning, respectively. This processing method can effectively reduce the interference of brightness deviation between images on feature extraction.

[0099] In the data augmentation stage, a series of spatial and geometric transformations are introduced to the images in the initial training set or the preprocessed spatiotemporal image sequence for transfer learning, including but not limited to:

[0100] Random Rotation: Randomly rotate the image within a range of ±15° with a certain probability, to enhance the robustness of the model to changes in cell orientation;

[0101] Random Crop: Extract a sub-region image from the center or a random region of the image, to simulate scenarios such as shooting offset and local magnification of cells;

[0102] Horizontal / Vertical Flip: Simulate the randomness of the natural direction of cells in the culture dish; Brightness / Contrast Adjustment: Adjust the brightness or contrast of the image to adapt to the differences in different imaging batches;

[0103] Gaussian Noise Injection: Simulate experimental imaging device noise to improve the model's ability to recognize low signal-to-noise ratio images.

[0104] All augmentation operations in this embodiment are efficiently implemented in batch processing mode through image processing tools (torchvision.transforms in PyTorch), and the augmentation parameters are dynamically sampled in each training iteration to ensure the diversity and stability of the augmented samples.

[0105] In comparison, the initial validation set does not introduce random augmentation operations, only uniform normalization is performed to ensure the repeatability of the evaluation process and the objectivity of the results, avoiding the non-consistent interference with the accuracy of model performance evaluation due to data augmentation.

[0106] 1-5) Construct a cell growth dynamics prediction model.

[0107] In this embodiment, the model includes a multi-scale feature extraction module and a time series modeling module connected in sequence.

[0108] The multi-scale feature extraction module adopts a pre-trained deep convolutional neural network ResNet-50 network, which is used for feature coding and extraction in the spatial dimension of the input single-frame cell image. In the input part of the ResNet-50 network, different scale convolution kernels are used to realize multi-scale feature expression and comprehensive perception of the input image. Meanwhile, the original fully connected classification layer (output dimension is 1000) for the ImageNet task is replaced by a linear classification head, and the output dimension is adjusted to the dimension of the kinetic feature to be recognized, which is used to complete the prediction task of cell growth kinetics.

[0109] In this embodiment, in order to enhance the perception ability of the model to different scale features in the image, the multi-scale feature extraction module constructed integrates three different size convolution kernels, which are 7x7, 5x5 and 3x3. The above convolution kernels can extract large scale structure information, medium scale texture features and local details in the image respectively, and improve the integrity and discriminability of the overall feature extraction.

[0110] In terms of channel dimension design, the ResNet-50 network adopts a step-by-step expanding feature channel growth strategy. The initial input channel number is 1 (corresponding to the original gray image), and then it is expanded to 32, 64 and 128 channels in turn, and finally it is shrunk to 64 channels. This channel design method helps to introduce more rich expression ability layer by layer in the feature extraction process, enhances the representation ability of the network to complex image features. At the same time, the moderate contraction of the terminal channel number can effectively reduce the computational load of the subsequent processing stage, balance the expression ability and resource consumption of the model, and improve the overall computing efficiency.

[0111] In terms of spatial size processing, the ResNet-50 network adopts a phased down-sampling strategy, which gradually reduces the resolution of the input image from the original size (such as 512x512) to 250x250, and then further to 125x125. This method realizes effective feature compression and calculation resource control on the basis of maintaining spatial hierarchical information, avoids the information loss problem caused by direct large-scale scaling, thereby ensuring the recognition accuracy of the model to the key spatial structure and improving the running efficiency.

[0112] In terms of spatial information integration, for different focal planes, the processed feature layers are added according to certain weights to integrate the whole spatial information. The weight is a normalized Gaussian distribution, which is used to weight N focal planes, wherein the weight of the middle focal plane is the largest, and gradually decreases towards both sides, and the overall presents a narrow peak-shaped Gaussian curve, and the sum of the final N weights is 1. In this embodiment, the N focal planes are numbered as follows: That is, the middle is 0, and the two sides are symmetrical. The Gaussian function is defined as:

[0113]

[0114] where w(x) is the value of the Gaussian function with mean 0 and variance σ at x.

[0115] Then normalize all w(x) so that the sum is 1:

[0116]

[0117] In this embodiment, N = 7, take σ = 0.5, and the focal plane weight is 1.20e-08, 2.64e-04, 1.06e-01, 7.87e-01, 1.06e-01, 2.64e-04, 1.20e-08, respectively. Through the synergistic effect of the above convolution kernel scale design, channel expansion and spatial compression and fusion strategy, the multi-scale feature extraction module can efficiently and stably extract multi-scale and structured image features.

[0118] Through the above structural optimization, the model can perform higher precision and stability in actual tasks, and is suitable for scenarios with high requirements for feature extraction accuracy and generalization ability of cell growth dynamics.

[0119] To enhance the modeling ability of the model for the time series dynamic features in the cell growth process, the time series modeling module uses a Gated Recurrent Unit (GRU) as the main network to process and dynamically model the spatial feature sequence extracted by the multi-scale feature extraction module. In this embodiment, the output layer of the GRU is replaced with a classification head with a dimension equal to the number of kinetic features to adapt to the kinetic prediction task.

[0120] 1-6) Train the model established in step 1-5).

[0121] In this embodiment, the model established in step 1-5) is trained using the training set and the validation set, wherein the multi-scale feature extraction module is further trained using the transfer learning dataset after the original training; the specific steps are as follows:

[0122] 1-6-1) Perform original training on the multi-scale feature extraction module.

[0123] In this embodiment, before the original training, a linear classification head is added at the back of the ResNet-50 network, and the output dimension of the linear classification head is the dimension of the kinetic features to be recognized, which is used to complete the prediction task of cell growth kinetics.

[0124] Further, in view of the characteristics of the processed medical image dataset in terms of structural features, image complexity, and sample size, the classification head structure of the ResNet-50 network is optimized to improve the classification performance and overall robustness of the model. Specifically, a 256-dimensional intermediate fully connected layer is added before the original dimension classification head to enhance feature integration and expression ability, thereby improving the modeling effect of high-level abstract features. To suppress overfitting during training and enhance the generalization ability of the model on non-training samples, the Dropout mechanism is introduced in the intermediate fully connected layer. During the training phase, this mechanism randomly discards some neuron connections, effectively reducing the model's excessive dependence on local features and improving its stability and adaptability under different data distributions.

[0125] The original training process of the multi-scale feature extraction module uses supervised learning. The training set obtained in steps 1-4) is used to train the multi-scale feature extraction module, with the goal of minimizing the cross-entropy loss function between the model output and the true label.

[0126] To improve training efficiency and scalability, the multi-GPU distributed training mechanism based on the PyTorch DistributedDataParallel (DDP) framework is used to realize parallel parameter updating across nodes or devices.

[0127] During training, the key hyperparameters are set as follows: the optimizer uses Adam (Adaptive Moment Estimation) to achieve efficient optimization of complex loss spaces through adaptive estimation of first and second moments; the initial learning rate is set to 1e-4, and the CosineAnnealing strategy is used for smooth decay to adapt to the learning needs of different training stages of the model; the weight decay coefficient is set to 1e-5 to suppress overfitting and enhance the generalization ability of the model; the batch size is set to 64 to adapt to multi-GPU parallel training and improve computational efficiency and gradient stability; the total number of training rounds is 150 to ensure that the model learns various features adequately; the loss function uses categorical cross-entropy loss to measure the difference between the predicted probability distribution and the true label; the activation function uses ReLU to enhance the non-linear expression ability of the model; in terms of parameter initialization, the backbone network in the multi-scale feature extraction module inherits the pre-trained weights of ResNet-50 on ImageNet, while the newly constructed classification head uses the He initialization strategy to optimize the initial gradient distribution and promote fast convergence during training.

[0128] To prevent overfitting and improve generalization performance, the early stopping strategy is introduced in this embodiment to monitor the loss value or accuracy change on the validation set. If the validation set loss of the last 10 epochs does not improve significantly (less than 1e-4), the training is terminated in advance. At the same time, the ModelCheckpoint mechanism is used to automatically save the model parameters with the best validation set performance for subsequent time series modeling module training stage.

[0129] All training processes are completed in a multi-card GPU server environment supporting CUDA acceleration to ensure efficient training speed and scalability. Training logs include loss value, accuracy, learning rate change, model weight change, etc., and are visualized in real time through TensorBoard.

[0130] 1-3-2) Transfer training of multi-scale feature extraction module.

[0131] This embodiment performs transfer training on the multi-scale feature extraction module trained in step 1-3-1), which is based on the SimSiam contrastive learning framework, enabling the module to generalize to unlabeled transfer learning datasets.

[0132] In this embodiment, to achieve asymmetric mapping of input features and contrastive feature learning, a feature projection and prediction structure based on the SimSiam framework is used, which includes two sub-modules: projection head (Projection MLP) and prediction head (Prediction MLP).

[0133] The projection head (Projection MLP) submodule is used to map the input 2048-dimensional feature vector to a 128-dimensional low-dimensional latent space, mainly including two fully connected layers and an output layer. The input dimension of the first fully connected layer is 2048, and the output dimension is 2048, containing batch normalization (BN) and ReLU activation. The input dimension of the second fully connected layer is 2048, and the output dimension is 2048, containing BN and ReLU. The input dimension of the output layer is 2048, and the output dimension is 128, without BN and activation function. This structure design helps to enhance the separability of features and reduce dimension redundancy, facilitating subsequent feature similarity modeling.

[0134] The prediction head (Prediction MLP) submodule is used to build an asymmetric mapping structure to realize the prediction alignment of the feature space, mainly including a full connection layer and an output layer. The input dimension of the full connection layer is 128, and the output dimension is 512, which contains BN and ReLU. The input dimension of the output layer is 512, and the output dimension is 128, without BN and ReLU. Through the design of the bottleneck structure, the prediction head can re-encode the features of the previous layer and participate in the similarity measurement process in an asymmetric manner, thereby improving the model's adaptability to semantic consistency and structural transformation.

[0135] In this embodiment, the training in this stage is performed using a transfer learning dataset without labels. The linear classification head of the ResNet-50 network in the multi-scale feature extraction module after the original training in step 1-3-1) is removed, and the convolutional layer and global average pooling layer are retained. The output dimension of the multi-scale feature extraction module is 2048, denoted as f. Then, a SimSiam-based contrastive learning framework is connected after the ResNet-50 network. The projection head and the prediction head are connected in sequence after the ResNet-50 network, denoted as h. In the transfer training of this stage, the parameters of the ResNet-50 network do not need to be frozen.

[0136] The training process of the contrastive learning in this embodiment is as follows. For an input image x, two sets of augmented images v1 and v2 are generated through step 1-4). The features obtained by f are z1 and z2, respectively. The prediction results obtained by h are p1 and p2.

[0137] The loss function of the contrastive learning training is:

[0138]

[0139] where is the negative cosine similarity, and stopgrad() is the stop gradient backpropagation. The goal is to maximize the consistency of image representations under different views, relying only on positive sample pairs during training.

[0140] During training, a progressive fine-tuning strategy is adopted. The key configurations of this embodiment are consistent with those in step 1-6-1). Different learning rate configurations are adopted for the ResNet50 network and the SimSiam contrastive learning architecture. The learning rate of the ResNet50 network is 0.1 times that of SimSiam, which ensures that the parameter update of the multi-scale feature extraction module after the original training in step 1-5-1) is not too fast, and the representation ability for the training set can be preserved.

[0141] 1-6-3) Distributed training of the time series modeling module.

[0142] In this embodiment, after completing the migration training of step 1-6-2), the projection head (Projection MLP) submodule and the prediction head (Prediction MLP) submodule connected after the ResNet-50 network in the multi-scale feature extraction module are removed, only the basic feature extraction network ResNet-50 trained and migrated is reserved as the backbone structure, and the parameters of the ResNet-50 network are frozen. In this embodiment, the training process of the time series modeling module is set to three different tasks to better characterize the cell growth process, including future frame prediction, masking frame reconstruction, and time series smoothing constraint.

[0143] For the future frame prediction task, in this embodiment, given an image sequence where T is the number of frames, i.e., the length of the time series, after extracting the time series features by GRU, the future prediction task is constructed to enable the model to learn the dynamics before and after the development process. The loss function is cross-entropy loss.

[0144] For the masking reconstruction task, in this embodiment, a feature reconstruction task based on random mask is designed, and the features of the time series are randomly masked with a probability of 30%. The loss function of the reconstruction task adopts cosine similarity, and this task enables the model to learn the feature correlation across time, and enhances the robustness and stability of the model to local anomalies. The loss function adopts cosine similarity.

[0145] For the time series smoothing constraint task, in this embodiment, considering that embryonic development has a connection, a feature smoothing constraint is applied between adjacent frames to ensure that the development trajectory of the time series feature sequence is continuous and smooth, and sudden changes in the prediction results are avoided. The loss function is:

[0146]

[0147] where subscript t represents the time point corresponding to the image sequence, h t represents the value of the hidden layer of GRU at time point t, and ε is a hyperparameter to prevent gradient disappearance. In one specific embodiment of the present application, ε = 1e-5.

[0148] The joint loss function of the time series modeling module training is:

[0149] L = aL predict + βL masked + γL smooth

[0150] where L predict represents the cross-entropy loss function of the future prediction task, L masked represents the cosine similarity loss function of the masking reconstruction task, and L smoothLoss function representing the temporal smoothing constraint. a, b, g represent the weights of the three loss functions respectively.

[0151] In this embodiment, a = 1, b = 0.2, and g = 0.1.

[0152] In this embodiment, the training of the temporal modeling module adopts a distributed data parallel strategy (Distributed Data Parallel, DDP), and the GRU and its classification head are jointly trained in a multi-GPU environment to ensure efficient processing and gradient synchronization of large-scale image sequences. The optimizer is Adam, the initial learning rate is set to 1e-4, and the StepLR learning rate scheduler is used, which decays the learning rate to 0.5 of the original every 10 rounds; the weight decay is set to 1e-5; the batch size is set to 16 (divided by sequence); the total number of training rounds is 120 rounds. Since the data used for transfer learning does not provide explicit label information, after the training of the temporal modeling module is completed, a multi-level clustering method is used to analyze the prediction results of the model to indirectly verify the effect of unsupervised temporal modeling. Specifically, the model trained by the GRU network and the classification head is applied to the image sequence to be tested to obtain the high-dimensional feature representation of the final output, and a similarity matrix between samples is constructed accordingly. To enhance the stability and representativeness of the clustering process, the standardized Euclidean distance is used as the basis for clustering measurement, and a bottom-up agglomerative clustering strategy is used to generate a visual clustering tree. By observing the branch structure of the clustering tree, the separation degree between clusters, and the temporal feature consistency of different sub-clusters in the clustering hierarchy, the discriminability and temporal consistency of the features learned by the model in the unsupervised scenario can be indirectly evaluated. After the training of the temporal modeling module is completed, the cell growth dynamics prediction model is trained.

[0153] 2) Application stage.

[0154] 2-1) Obtain test samples

[0155] In this embodiment, the test samples are a set of complete cell growth spatio-temporal image sequences, wherein each time point in the sequence uses a multi-focal plane image, the number of focal planes is consistent with the training stage, and the cell type in the sequence image is consistent with the cell type of the collected images in the training stage.

[0156] 2-2) The cell growth spatio-temporal image sequence obtained in step 2-1) is sequentially preprocessed to obtain an optimized cell growth spatio-temporal image sequence.

[0157] The preprocessing includes:

[0158] Denoising the images in the image sequence;

[0159] performing empty frame detection on the denoised image sequence; wherein if there is an empty frame in the multi-focal plane image at any time point in the image sequence, all images at the time point are removed from the image sequence;

[0160] performing cell region identification and image cropping on the images in the image sequence after the empty frame detection is completed, to obtain an image sequence containing only cell regions, which is the optimized cell growth spatiotemporal image sequence.

[0161] In one specific embodiment of the present application, all images in the optimized image sequence are uniformly adjusted to 512x512 resolution.

[0162] In one specific embodiment of the present application, a timelaps time difference incubator is used for data acquisition, a set of spatiotemporal image sequences of embryo development is acquired, which contains 7 focal planes in space and 500 time points in the whole cycle of cell growth in time, and the cell growth dynamics label is manually annotated by a doctor. After denoising and empty frame detection and removal, the image sequence is subjected to cell identification and cropping, and finally normalized to 512x512 resolution. The test image does not need to be subjected to data enhancement.

[0163] 2-3) performing normalization operation on the images in the optimized cell growth spatiotemporal image sequence obtained in step 2-2), linearly mapping the pixel values to the interval [0, 1] uniformly, to obtain the final test image sequence.

[0164] 2-4) inputting the test image sequence of step 2-3) into the cell growth dynamics prediction model trained in step 1) to obtain the cell growth dynamics label (i.e. the identification result of the cell growth stage) corresponding to each time point image in the sequence, so as to realize cell growth dynamics prediction.

[0165] To realize the above-mentioned embodiments, the second aspect embodiment of the present application proposes a cell growth dynamics prediction device based on spatiotemporal image sequence, comprising:

[0166] an image sequence acquisition module, configured to acquire a spatiotemporal image sequence containing the complete growth process of cells, wherein the image at each time point in the image sequence is a multi-focal plane image;

[0167] an image sequence processing module, configured to obtain a final test image sequence after pre-processing and normalization of the images in the image sequence;

[0168] a cell growth dynamics prediction module, configured to input the test image sequence into a preset cell growth dynamics prediction model to obtain a cell growth stage recognition result corresponding to each time point image in the test image sequence, so as to realize cell growth dynamics prediction; wherein the model comprises a multi-scale feature extraction module and a time sequence modeling module connected in sequence, the multi-scale feature extraction module adopts a ResNet-50 network, and the time sequence modeling module adopts a gated recurrent unit (GRU).

[0169] It should be noted that the foregoing embodiment of the cell growth dynamics prediction method based on a time-space image sequence is also applicable to the cell growth dynamics prediction device based on a time-space image sequence of the present embodiment, and will not be described herein again. According to the cell growth dynamics prediction device based on a time-space image sequence proposed in the embodiment of the present application, a time-space image sequence containing a complete cell growth process is obtained, and each time point image in the image sequence is a multi-focal plane image. After the images in the image sequence are preprocessed and normalized, a final test image sequence is obtained. The test image sequence is input into a preset cell growth dynamics prediction model to obtain a cell growth stage recognition result corresponding to each time point image in the test image sequence, so as to realize cell growth dynamics prediction. The model comprises a multi-scale feature extraction module and a time sequence modeling module connected in sequence, the multi-scale feature extraction module adopts a ResNet-50 network, and the time sequence modeling module adopts a gated recurrent unit (GRU). Thus, the problems of poor adaptability, insufficient stability, and high labeling cost of traditional models in processing dynamic medical image sequences can be effectively solved, and the cell growth time-space image sequence dynamics prediction has significant advantages and application prospects.

[0170] In a specific embodiment of the present application, the preprocessing comprises:

[0171] de-noising the images in the image sequence;

[0172] detecting and removing empty frames from the de-noised image sequence; wherein if there is an empty frame in the multi-focal plane image at any time point in the image sequence, all images at the time point are removed from the image sequence;

[0173] performing cell region recognition and image cropping on the images in the image sequence after removing the empty frames to obtain an image sequence containing only cell regions.

[0174] In a specific embodiment of the present application, before the test image sequence is input into the preset cell growth dynamics prediction model, the method further comprises:

[0175] training the cell growth dynamics prediction model;

[0176] the training of the cell growth kinetics prediction model comprises:

[0177] 1) respectively acquiring a cell growth spatiotemporal image sequence for original training and a cell growth spatiotemporal image sequence for transfer learning; wherein the two image sequences both contain a complete process of cell growth; the cell types corresponding to the two image sequences are consistent with the cell type corresponding to the test image sequence;

[0178] performing cell growth kinetics labeling on the cell growth spatiotemporal image sequence for original training frame by frame;

[0179] 2) pre-processing the images in the two image sequences obtained in step 1);

[0180] 3) grouping the cell growth spatiotemporal image sequence for original training and the corresponding labeling result after pre-processing into samples, and then randomly dividing all the samples into an initial training set and an initial validation set according to a set proportion;

[0181] 4) performing normalization and enhancement processing on the images in the initial training set and the cell growth spatiotemporal image sequence for transfer learning after pre-processing, to obtain a final training set and an optimized cell growth spatiotemporal image sequence for transfer learning, wherein the optimized cell growth spatiotemporal image sequence for transfer learning is taken as a transfer learning data set;

[0182] performing normalization processing on the images in the initial validation set to obtain a final validation set;

[0183] 5) constructing a cell growth kinetics prediction model;

[0184] 6) completing the training process of the cell growth kinetics prediction model by using the training set, the validation set and the transfer learning data set, to obtain a trained cell growth kinetics prediction model.

[0185] In a specific embodiment of the present application, the training process of the cell growth kinetics prediction model comprises:

[0186] performing original training on the multi-scale feature extraction module by using the training set and the validation set;

[0187] performing transfer training on the multi-scale feature extraction module after original training by using the transfer learning data set;

[0188] after the transfer training of the multi-scale feature extraction module is completed, freezing the model parameters of the multi-scale feature extraction module and performing distributed training on the time series modeling module.

[0189] In a specific embodiment of the present application, it further comprises:

[0190] Before the original training of the multi-scale feature extraction module using the training set and the validation set, a linear classification head is added behind the ResNet-50 network, and the output dimension of the linear classification head is the dimension of the cell growth kinetics feature.

[0191] In one specific embodiment of the present application, further comprising:

[0192] Before the start of the transfer training, the linear classification head of the ResNet-50 network that has completed the original training is removed, and then a SimSiam-based contrast learning framework is connected behind the ResNet-50 network, the contrast learning framework comprising a projection head submodule and a prediction head submodule; the projection head submodule and the prediction head submodule are connected in turn behind the ResNet-50 network.

[0193] In one specific embodiment of the present application, further comprising:

[0194] After the transfer training is completed, the projection head submodule and the prediction head submodule connected behind the ResNet-50 network are removed, the output of the ResNet-50 network is connected to the GRU, and the parameters of the ResNet-50 network are frozen;

[0195] Then set three different tasks to complete the training of the time series modeling, the tasks including: future frame prediction task, masking frame reconstruction task and time series smoothing constraint task; the joint loss function of the time series modeling module training is the weighted sum of the loss functions of the three tasks.

[0196] To achieve the above-mentioned embodiments, the third aspect of the present application proposes an electronic device, comprising:

[0197] At least one processor; and a memory communicatively connected with the at least one processor;

[0198] The memory stores instructions executable by the at least one processor, and the instructions are configured to execute the above-mentioned cell growth kinetics prediction method based on a spatiotemporal image sequence.

[0199] To achieve the above-mentioned embodiments, the fourth aspect of the present application proposes a computer readable storage medium, the computer readable storage medium stores computer instructions, and the computer instructions are used to make the computer execute the above-mentioned cell growth kinetics prediction method based on a spatiotemporal image sequence.

[0200] It should be noted that the computer-readable medium in the above disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the present disclosure, the computer-readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer-readable program code. Such a propagated data signal can take many forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium other than the computer-readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to a wire, a cable, an RF (radio frequency) or the like, or any suitable combination of the above.

[0201] The computer-readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device. The computer-readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the cell growth kinetics prediction method based on the spatio-temporal image sequence described above.

[0202] Computer program code for carrying out operations of the present disclosure can be written in any one or more programming languages or combinations of languages including object or visual programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0203] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the specification and the features of the different embodiments or examples, without contradiction.

[0204] In addition, the terms "first", "second", etc. are used only for the purpose of description and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise explicitly specified.

[0205] Any process or method descriptions or descriptions of the flow diagrams in the specification or otherwise described herein can be understood as representing code modules, segments, or portions of code which include one or more executable instructions for implementing specific logic functions (or steps) in the process, and the various embodiments of the application can include additional or fewer functions. The various embodiments of the application can also modify other processes or methods described or otherwise suggested to perform the functions of the various embodiments of the application, and these modifications should be understood as falling within the scope of the various embodiments of the application. The various embodiments of the application should be understood to support a variety of additional modifications to otherwise result in performing the functions of the various embodiments of the application.

[0206] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, processor- based system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For purposes of this specification, a "computer-readable medium" can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium. The computer- readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer-readable storage medium include the following: an electrical connection having one or more wires (electrical connections), a portable computer diskette (a magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and stored in a computer memory.

[0207] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the above embodiments, the various steps or methods can be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following technologies, known in the art, or their combinations can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and the like.

[0208] Those skilled in the art can understand that all or part of the steps carried out by the above-mentioned embodiments can be completed by programs instructing related hardware, and the programs can be stored in a computer-readable storage medium. When the programs are executed, they include one or a combination of the steps of the method embodiments.

[0209] In addition, each of the functional units in the various embodiments of the present application can be integrated in one processing module, or each of the units can be physically present separately, or two or more units can be integrated in one module. The integrated module can be implemented in the form of hardware or in the form of a software functional module. When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer readable storage medium.

[0210] The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it should be understood that the above embodiments are exemplary and should not be construed as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for predicting cell growth dynamics based on spatiotemporal image sequences, characterized in that, include: Acquire a spatiotemporal image sequence containing the complete cell growth process, wherein the image at each time point in the image sequence is a multifocal image; After preprocessing and normalizing the images in the image sequence, the final test image sequence is obtained; The test image sequence is input into a preset cell growth kinetics prediction model to obtain the cell growth stage identification result corresponding to each time point image in the test image sequence, so as to realize cell growth kinetics prediction; wherein, the model includes a multi-scale feature extraction module and a temporal modeling module connected in sequence, the multi-scale feature extraction module adopts a ResNet-50 network, and the temporal modeling module adopts a gated recurrent unit (GRU).

2. The method according to claim 1, characterized in that, The preprocessing includes: Denoise the images in the image sequence; Empty frames are detected and removed from the denoised image sequence; wherein, if an empty frame exists in the multi-focal plane image at any time point in the image sequence, all images at that time point are removed from the image sequence. Cell region identification and image cropping are performed on the images in the image sequence after removing empty frames to obtain an image sequence containing only cell regions.

3. The method according to claim 2, characterized in that, Before inputting the test image sequence into a preset cell growth kinetics prediction model, the method further includes: Train the cell growth kinetics prediction model; The training of the cell growth kinetics prediction model includes: 1) Obtain spatiotemporal image sequences of cell growth for original training and spatiotemporal image sequences of cell growth for transfer learning, respectively; wherein, both image sequences contain the complete process of cell growth; the cell types corresponding to the two image sequences are consistent with the cell types corresponding to the test image sequences; Cell growth dynamics were annotated frame by frame in the spatiotemporal image sequence of cell growth used for the original training. 2) Preprocess the images in the two image sequences obtained in step 1); 3) The preprocessed spatiotemporal image sequence of cell growth used for original training and its corresponding annotation results are combined into a sample. Then, all samples are randomly divided into an initial training set and an initial validation set according to a set ratio. 4) Normalize and enhance the images in the initial training set and the preprocessed spatiotemporal image sequence of cell growth for transfer learning to obtain the final training set and the optimized spatiotemporal image sequence of cell growth for transfer learning. The optimized spatiotemporal image sequence of cell growth for transfer learning is used as the transfer learning dataset. The images in the initial validation set are normalized to obtain the final validation set; 5) Construct a cell growth kinetics prediction model; 6) Use the training set, validation set, and transfer learning dataset to complete the training process of the cell growth dynamics prediction model and obtain the trained cell growth dynamics prediction model.

4. The method according to claim 3, characterized in that, The training process of the cell growth kinetics prediction model includes: The multi-scale feature extraction module was trained using the training and validation sets. The multi-scale feature extraction module, which was originally trained, was transferred and trained using a transfer learning dataset. After the multi-scale feature extraction module has completed transfer training, the model parameters of the multi-scale feature extraction module are frozen, and the time series modeling module is trained in a distributed manner.

5. The method according to claim 4, characterized in that, Also includes: Before performing the original training of the multi-scale feature extraction module using the training and validation sets, a linear classification head is added after the ResNet-50 network. The output dimension of the linear classification head is the dimension of the cell growth dynamics features.

6. The method according to claim 5, characterized in that, Also includes: Before the transfer training begins, the linear classification head is removed from the ResNet-50 network that has completed the original training. Then, a SimSiam-based contrastive learning framework is connected after the ResNet-50 network. The contrastive learning framework includes a projection head submodule and a prediction head submodule. The projection head submodule and the prediction head submodule are connected sequentially after the ResNet-50 network.

7. The method according to claim 5, characterized in that, Also includes: After the transfer training is completed, the projection head submodule and the prediction head submodule connected to the ResNet-50 network are removed, the output of the ResNet-50 network is connected to the GRU, and the parameters of the ResNet-50 network are frozen. Then, three different tasks are set up to complete the training of the temporal modeling fast, including: future frame prediction task, occlusion frame reconstruction task, and temporal smoothing constraint task; The joint loss function trained by the time-series modeling module is a weighted sum of the loss functions of the three tasks.

8. A cell growth dynamics prediction device based on spatiotemporal image sequences, characterized in that, include: The image sequence acquisition module is used to acquire a spatiotemporal image sequence containing the complete cell growth process, wherein the image at each time point in the image sequence is a multifocal image; The image sequence processing module is used to preprocess and normalize the images in the image sequence to obtain the final test image sequence. The cell growth dynamics prediction module is used to input the test image sequence into a preset cell growth dynamics prediction model to obtain the cell growth stage identification result corresponding to each time point image in the test image sequence, so as to realize cell growth dynamics prediction; wherein, the model includes a multi-scale feature extraction module and a temporal modeling module connected in sequence, the multi-scale feature extraction module adopts a ResNet-50 network, and the temporal modeling module adopts a gated recurrent unit (GRU).

9. An electronic device, characterized in that, include: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions executable by the at least one processor, the instructions being configured to perform the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method according to any one of claims 1-7.

Images