A method for IVF-ET embryo selection based on multi-task learning using time-lapse imaging

Abstract

The present invention discloses an IVF-ET embryo selection method based on time-difference imaging multi-task learning, belonging to the field of image data processing and assisted reproduction. The method comprises the following steps: first, using a coding module, effectively extracting spatiotemporal features in embryo time-difference imaging sequences through an expanded three-dimensional convolutional network and a bidirectional long short-term memory network; then, within a multi-task learning framework with shared hard parameters, utilizing shared feature representations, and simultaneously predicting multiple key indicators such as a quality grade based on the Istanbul consensus and a developmental grade based on the Gardner grade through multiple task-specific predictors; during the training process, the AdaTask optimization algorithm and the Frank-Wolfe method are used to optimize model parameters, and a virtual predictor mechanism is introduced to enhance the model's versatility; the present invention can more accurately, objectively, and efficiently assess embryo quality and developmental potential, reduce the burden on embryologists, and improve the success rate of IVF and ET, and can be applied to fields such as animal husbandry and endangered species protection.

Description

Technical Field

[0001] The present invention relates to the fields of image data processing and assisted reproductive technology, and in particular to an IVF-ET embryo selection method based on time-difference imaging multi-task learning. Background Art

[0002] In vitro fertilization (IVF) has become an important tool for addressing infertility. IVF success rates largely depend on embryo selection and transfer. Selecting the embryo most likely to successfully implant and develop from among the multiple available embryos is a major challenge for embryologists.

[0003] In in vitro fertilization-embryo transfer (IVF-ET) technology, embryo quality grading and blastocyst formation prediction are key factors in determining embryo selection.

[0004] Currently, the internationally accepted embryo scoring standard is the Istanbul Consensus, while blastocyst-stage embryo quality grading primarily utilizes the Gardner blastocyst grading mechanism. However, traditional embryo assessment methods rely primarily on embryologists observing and assessing embryo morphology and developmental rates at different time points using an optical microscope. Due to subjective and empirical differences between observers, or within the same observer at different times, embryo grading results are often subjective and inconsistent. This subjective assessment method is not only inefficient but also has limited accuracy, resulting in a clinical pregnancy rate of only approximately 35% after implantation.

[0005] With the development of artificial intelligence (AI), deep learning techniques have been widely applied in embryo assessment. For embryo quality grading, existing technologies primarily include methods based on convolutional neural networks (CNNs), such as the pronuclear embryo classification method proposed by Dimitriadis et al. and the automatic embryo grading system proposed by Cher et al. These methods analyze embryo images to automatically assess morphological features such as the cytoplasm, pronucleus, and zona pellucida. For blastocyst formation prediction, existing technologies primarily utilize time series analysis methods, such as the time flow model combining DenseNet201 and LSTM proposed by Liao et al., and the multi-focal feature aggregation network proposed by Xie et al. These methods predict an embryo's potential to form a blastocyst by analyzing temporal features during embryonic development.

[0006] However, the existing technology has the following problems:

[0007] First, existing methods primarily utilize single-modality data to assess embryo quality and developmental potential, primarily focusing on analyzing embryo images using CNNs. This results in much of the data relevant to embryo assessment not being effectively utilized. Second, existing methods employ only a single evaluation standard, such as the Istanbul Consensus or Gardner grading, to assess embryos. However, no single evaluation standard currently guarantees an objective, standardized assessment of embryos. This results in biased embryo assessment results, making it difficult to objectively describe embryo quality and developmental potential. Furthermore, existing methods make insufficient use of time-lapse imaging data, failing to effectively capture the temporal characteristics and dynamic changes during embryonic development.

[0008] Therefore, how to further improve the accuracy of embryo evaluation and screening, reduce the bias of prediction results, and improve the efficiency of embryo evaluation and screening is a technical problem that needs to be urgently solved in this field. Summary of the Invention

[0009] The purpose of the present invention is to provide an IVF-ET embryo selection method based on time-lapse imaging multi-task learning to solve the above problems.

[0010] To achieve the above objectives, the present invention adopts the following technical solution: a method for selecting IVF-ET embryos based on time-lapse imaging multi-task learning, comprising the following steps:

[0011] S1: Data preparation and preprocessing:

[0012] S11. Data Acquisition: Collect time-lapse video or image sequences of embryonic development during IVF.

[0013] S12. Data preprocessing: preprocessing the time-lapse imaging video or image sequence;

[0014] S2: Extract spatiotemporal features through the encoding module:

[0015] S21. Input encoding module: input the image sequence preprocessed in S12 into the encoding module;

[0016] S22. ResNet3D Feature Extraction: Utilizes an improved ResNet3D network to process the input sequence and extract spatiotemporal features during embryonic development. Specifically, the improved ResNet3D network reduces the number of parameters in the standard 3D convolution by using a 3D depthwise separable convolution with an improved convolution kernel size of 13×3×3.

[0017] S23. Regularization: Apply a spatial Dropout layer after the ResNet3D layer to preserve dimensionality.

[0018] S24. Feature pooling:

[0019] Applying spatial maximum pooling, we obtain features with a dimension of 8×256, which captures significant spatial structural information; applying spatial average pooling, we obtain features with a dimension of 8×256, which preserves the overall spatial information;

[0020] S25. Time series feature fusion: The results of maximum pooling and average pooling are fed into a bidirectional long short-term memory network layer;

[0021] S3: Multi-task learning and optimization:

[0022] S31. Shared representation: The feature vector output from step S2 is used as the shared representation , which is input to the subsequent task-specific predictor, where is the input sample, For shared encoders, is the encoded feature vector;

[0023] S32. Task-specific predictors: Build independent predictors for each evaluation task ;

[0024] S33. Multi-task learning framework application:

[0025] Adopt hard parameter sharing structure and shared encoder parameters; the Frank-Wolfe-based method is used to treat the problem as a multi-objective optimization problem, and the shared parameters are iteratively updated by solving subproblems to find the Pareto optimal solution;

[0026] S34. Model training and optimization:

[0027] Loss function: For classification tasks, the cross entropy loss function is used as ;Total loss is the sum of losses of each task;

[0028] ,

[0029] in: is the cross entropy loss of the kth task; is the true label of category c, that is, the value in one-hot encoding, 0 or 1; log is the natural logarithm function; The probability of category c predicted by the model;

[0030] Optimizer: Use AdaTask optimization method to update model parameters ;

[0031] Training process: Use the labeled embryo dataset to train the model, and iteratively update the network parameters through backpropagation and optimizer until the model converges or reaches the preset number of training rounds;

[0032] The present invention preferably searches for the Pareto optimal solution based on Frank-Wolfe:

[0033] Multi-objective optimization perspective: Multi-task learning can be viewed as a multi-objective optimization problem, that is, minimizing the loss function of all tasks at the same time The goal is to find a Pareto optimal solution, i.e. one where the performance of any task cannot be improved without sacrificing the performance of at least one task.

[0034] Frank-Wolfe (Conditional Gradient) method: This method can be used to find Pareto stationary points. Its core idea is to find an effective descent direction by solving a linear programming subproblem in each iteration. Specifically, it calculates the gradient of all tasks. , and find a direction (usually the negative gradient direction of a task or a convex combination thereof) so that the losses of all tasks in this direction have the potential to be improved or reach a certain balance.

[0035] Solve the subproblem: in the shared parameters In the update of , the FW algorithm needs to solve the following sub-problems: find smallest , where G is a matrix whose columns are the gradients of each task with respect to the shared parameters , is a weight vector (of a simplex). This usually means choosing the single task gradient direction that is most opposite to the current combined gradient direction.

[0036] Parameter update: Then update the shared parameters along the found direction This approach helps to make trade-offs between the objectives of different tasks and find a set of solutions that can better balance the performance of all tasks, rather than just optimizing the weighted sum.

[0037] S4: Classification prediction and embryo selection:

[0038] S41. Feature input: The shared feature vector output by the encoding module is input into each task-specific predictor of the classification module;

[0039] S42. Multilayer Perceptron Processing:

[0040] First hidden layer: 128 neurons, ReLU activation function, followed by Dropout (0.5),

[0041] Second hidden layer: 64 neurons, ReLU activation function, followed by Dropout (0.5),

[0042] S43. Output prediction:

[0043] Istanbul consensus classification task: The output layer has 5 neurons, applies the Softmax activation function, and outputs a probability distribution corresponding to 5 quality levels;

[0044] Gardner grading task: The output layer has 6 neurons, and the Softmax activation function is applied to output the probability distribution corresponding to the 6 developmental levels;

[0045] S44. Embryo selection decision: Develop an embryo selection strategy based on multiple predictions output by the model, combined with clinical experience and specific needs.

[0046] As a preferred technical solution,

[0047] In step S11, each embryo sample contains continuous image frames covering key developmental stages; each sample contains 32 frames of grayscale images, and each frame has a resolution of 256×256 pixels;

[0048] In step S12, the preprocessing includes image alignment, brightness normalization, and denoising operations.

[0049] As a preferred technical solution,

[0050] In S22, the ResNet3D network includes three 3D residual blocks, each residual block includes two 3D convolutional layers, and the convolution kernel size is 13×3×3; after processing by multiple residual blocks, the dimension of the feature map gradually decreases, and the final output feature map dimension is: 8×32×32×256.

[0051] In S23, the dimension 8×32×32×256 is kept unchanged to prevent overfitting;

[0052] In S24, spatial maximum pooling and spatial average pooling are applied, and both obtain features with a dimension of 8×256;

[0053] In S25, the bidirectional long short-term memory network layer contains 128 units in each direction, processes information in the time dimension, and captures the temporal dependency of embryonic development; the bidirectional long short-term memory network layer ultimately outputs a 256-dimensional feature vector that integrates spatiotemporal information.

[0054] About the Bidirectional Long Short-Term Memory network layer, or Bi-LSTM:

[0055] Function: Bi-LSTM is good at processing time series data and can learn dependencies in the time dimension. The calculation of the forget gate, input gate, and output gate of Bi-LSTM can be expressed as:

[0056] ,

[0057] ,

[0058] ,

[0059] in, 、 and are the activation values ​​of the forget gate, input gate, and output gate, respectively. and b are learnable parameters, is the sigmoid function. The update of cell state and hidden state can be expressed as:

[0060] ,

[0061] ,

[0062] ,

[0063] Where: t represents the time step, For the Gate of Forgetfulness, is the input gate, is the output gate, 、 、 、 is the weight matrix, 、 、 、 is the bias vector, is the hidden state at the previous moment, is the input at the current moment, is the sigmoid activation function, is the candidate cell state, is the cell state at the current moment, is the cell state at the previous moment, is the hidden state at the current moment, is an element-by-element multiplication operation, and tanh is a hyperbolic tangent activation function.

[0064] As a preferred technical solution,

[0065] In S32, the predictor adopts a multi-layer perceptron structure, including:

[0066] Input layer: receives feature representations from the shared encoder;

[0067] Hidden layer: contains multiple fully connected layers, each followed by a ReLU activation function and Dropout regularization for nonlinear transformation of features and prevention of overfitting;

[0068] Output layer: Use the Softmax activation function according to the specific task and output the probability distribution of the corresponding level:

[0069] ,

[0070] in: is the probability that the input x belongs to category c, is the raw score (logits) of category c, is the total number of categories of task k, exp is the natural exponential function;

[0071] The tasks include the Istanbul Consensus Classification and the Gardner Classification.

[0072] The classification module of the present invention adopts a multi-layer perceptron (MLP) structure to predict and evaluate embryo quality.

[0073] As a preferred technical solution,

[0074] In S33, the hard parameter sharing structure includes a shared encoder and Task-specific predictors , the objective function is the sum of all task losses:

[0075] ,

[0076] in: is the set of all model parameters, are the parameters of the shared encoder, are the parameters of the k-th task-specific predictor, For shared encoders, is the kth task-specific predictor, is the i-th input sample, is the true label of the i-th sample on the k-th task, is the loss function of the kth task, is the number of samples, K is the number of tasks, Represents function composition.

[0077] As a preferred technical solution,

[0078] In S33, a virtual predictor is introduced , whose structure is the same as the real predictor but the parameters are randomly initialized and fixed during training; by constraining the gradient norm of the virtual predictor, the representation generated by the shared encoder is forced to be friendly to any predictor rather than just adapting to the optimal predictor;

[0079] Add a penalty term for the virtual predictor gradient norm to the training objective , to improve the versatility of shared representations,

[0080] Among them, the loss function of a single task is to add the penalty term of the virtual predictor gradient norm to the original loss function:

[0081] ,

[0082] The specific form is:

[0083] ,

[0084] in: is the predicted value of the k-th task for the i-th sample, is the predicted value of the virtual predictor for the i-th sample, is the shared representation of the i-th sample, is the penalty term coefficient, which is used to balance the task loss and the virtual gradient norm;

[0085] The total loss function of multi-task is the sum of all task losses:

[0086] ,

[0087] Among them, the loss of each task consists of two parts:

[0088] True predictor loss : By the true predictor Generate predictions and calculate the traditional mission losses;

[0089] Virtual predictor gradient norm penalty: through virtual predictor Generate predictions , calculates the gradient norm of its loss with respect to its own parameters, which is used to improve the versatility of shared representation.

[0090] in: is the total loss of the k-th task, is the sample size, is the true predictor loss function for the k-th task, is the true label of the i-th sample of the k-th task, is the predicted value of the true predictor for the i-th sample of the k-th task, is the virtual predictor loss function for the k-th task, is the predicted value of the virtual predictor for the i-th sample of the k-th task, is the virtual predictor parameter for the kth task, is the gradient of the virtual predictor parameters, is the Frobenius norm, is the penalty coefficient.

[0091] Universal Quantification:

[0092] Definition: Commonality of shared encoders It is defined as the inverse of the difference between the optimal predictor loss and the arbitrary predictor loss. If any randomly initialized "virtual predictor" If low loss can be achieved based on shared representation, then the versatility is high;

[0093] Theorem: Under the assumption of convexity of the loss function, the generalizability is inversely proportional to the gradient norm of the virtual predictor:

[0094] ,

[0095] Note: is the universality measure of the shared encoder, is the virtual predictor parameter for the kth task, represents the gradient with respect to the parameters of the dummy predictor, is the Frobenius norm, and α represents a proportional relationship.

[0096] As a further preferred technical solution,

[0097] In step S34, the classification task includes Istanbul consensus and Gardner classification; the total loss includes a penalty term of the virtual predictor gradient norm;

[0098] The AdaTask optimization method is:

[0099] Initialization: Set the exponential decay factor v and the initial learning rate , smoothing factor ε, initialize the cumulative gradient variable for each task k ;

[0100] Gradient calculation: At training step t, the gradient of the parameter θ is calculated for each task ;

[0101] Cumulative gradient update: For each task, the cumulative gradient is updated by exponential decay average:

[0102] ,

[0103] Parameter update amount calculation: Calculate the parameter update amount for each task:

[0104] ,

[0105] Model parameter update: Update the model parameters according to the parameter update amount of all tasks:

[0106] ,

[0107] Where: t is the training step, k is the task index, i is the parameter index, is the cumulative gradient of task k with respect to parameter i at step t, is the gradient of task k with respect to parameter i at step, v is the exponential decay factor, is the initial learning rate, ε is the smoothing factor, is the update amount of task k to parameter i, is the value of parameter i at step t.

[0108] Using AdaTask optimization method:

[0109] Task-based separation of accumulated gradients: When traditional adaptive learning rate optimizers are used to optimize shared parameters in MTL models, the statistics aggregate the gradients of all tasks, which can easily lead to certain tasks dominating the accumulated gradients and learning rates. AdaTask maintains separate accumulated gradient variables for each shared parameter and task, avoiding task conflicts and improving optimization efficiency. This is preferably achieved by combining RMSProp with AdaTask.

[0110] Optimization effect: By maintaining independent cumulative gradient statistics for each task, AdaTask can avoid gradient interference between tasks, enabling each task to obtain an appropriate learning rate, thereby improving the overall performance of multi-task learning.

[0111] As a preferred technical solution,

[0112] In S44, the final prediction result is obtained by taking the category with the highest probability:

[0113] ,

[0114] in: is the predicted category of the kth task, argmax means taking the category c with the maximum probability, is the predicted probability that the input x belongs to category c.

[0115] This probability-based prediction method not only provides an assessment of the embryo's grade but also gives the confidence level of the prediction, which helps in clinical decision-making.

[0116] The main innovation of the method of the present invention lies in the adoption of multi-task learning and the use of two optimization methods to solve various problems in multi-task learning to improve the training effect.

[0117] The innovation of the multi-task learning framework of this invention lies in:

[0118] 1. This invention is the first in the art to use this framework to solve the problem of embryo selection;

[0119] 2. Optimize multi-task learning using adaptive learning rates and generality enhancement methods:

[0120] 2.1 The generality enhancement strategy mainly addresses the bias of the feature extraction module in extracting single-task features;

[0121] 2.2 Adaptive Learning Rate solves the problem that excessive adjustment of the learning rate of a single task affects other tasks, and realizes the separation of the learning rate of each task.

[0122] The present invention can improve the accuracy of embryo assessment and screening and reduce the bias of prediction results. Specifically:

[0123] In the direction of improving accuracy:

[0124] 1. Adaptive Learning Rate

[0125] The adaptive learning rate mechanism employed in this paper dynamically adjusts the learning rate based on the convergence speed and gradient information of different tasks. This avoids the problem of traditional fixed learning rates in multi-task scenarios, which can lead to insufficient learning or premature cessation of certain tasks. This allows the model to more fully learn the deep features of each task, thereby improving the accuracy of overall predictions. For example, the dynamic characteristics and indicative weights of the final developmental potential may differ between the early embryonic development stage and the blastocyst formation stage. The adaptive learning rate can better capture the contribution of these dynamic changes to different prediction tasks.

[0126] 2. Virtual prediction mechanism

[0127] By introducing a virtual prediction mechanism, the present invention can reduce the overfitting of the model to a specific training data distribution and improve the generalization ability of the model on unseen new data or data from different clinics.

[0128] In terms of reducing bias:

[0129] 1. Adaptive Learning Rate

[0130] By dynamically adjusting the learning pace for different tasks or data subsets, the model can avoid premature convergence to suboptimal solutions due to certain dominant features or majority group data, thereby giving minority groups or less significant features more opportunities to learn. This allows the model to exhibit more consistent performance for embryo data from different sources or types, reducing prediction bias caused by data imbalance or feature differences.

[0131] 2. Virtual prediction mechanism

[0132] The virtual prediction mechanism forces the model to learn features that are robust to different data distributions, rather than relying solely on accidental correlations or biased features that may exist in the training data. This allows the model to make fairer and less biased predictions when faced with data with different potential biases.

[0133] Compared with existing AI-based evaluation methods, for example, the multi-task deep learning and dynamic programming (MTDL-DP) method proposed by Zihan Liu et al.:

[0134] 1. Different evaluation criteria

[0135] Unlike the MTDL-DP method by Zihan Liu et al., which primarily focuses on morphological classification of embryonic developmental stages, this method, through an innovative multi-task learning framework, not only integrates the Istanbul consensus's macroscopic developmental potential judgment and the Gardner classification's fine-grained morphological indicators, but also focuses on directly predicting assessment indicators that are highly correlated with clinical pregnancy outcomes. This makes the leap from "stage identification" to "potential prediction," and the assessment results are more clinically meaningful.

[0136] 2. Different learning rate adjustment strategies

[0137] The adaptive learning rate mechanism introduced in this paper can dynamically adjust learning weights based on the complexity and convergence of each subtask in multi-task learning. This avoids the problem of insufficient learning or overfitting of certain key tasks caused by improper learning rate settings in traditional MTDL methods, ensures sufficient learning of the features of each evaluation dimension, and thus improves the accuracy of the evaluation overall. In contrast, existing methods such as MTDL-DP, if they use a fixed learning rate, may find it difficult to optimally balance the learning of classification tasks at different developmental stages;

[0138] 3. Different optimizations for data universality

[0139] To address the performance degradation and prediction bias that can occur in existing AI methods (including MTDL-DP and most CNN-based methods) when applied to diverse data sources (e.g., different reproductive centers, different culture systems, and different imaging devices), the present invention employs a virtual prediction mechanism. This strategy forces the model to learn robust, cross-domain, intrinsic characteristics of embryos rather than biased information from specific datasets. As a result, the present invention demonstrates stronger generalization and lower prediction bias when processing data from diverse sources, resulting in more objective and fair evaluation results.

[0140] The method of this invention, which employs an improved ResNet3D (3D depthwise separable convolution with optimized convolution kernels) and a Bi-LSTM network, can automatically and precisely capture the three-dimensional spatial morphological structure and its dynamic evolution throughout embryonic development, from early development to blastocyst formation, as well as the long-term temporal dependencies between these features, from time-lapse imaging sequences. This means the model can identify subtle but critical dynamic patterns and morphological cues that are imperceptible or easily overlooked by the human eye and are associated with the embryo's developmental potential (ultimately, whether it can successfully implant and develop into a fetus). For example, specific cell division timing, subtle changes in fragmentation, and the rate and uniformity of blastocoel expansion may all be related to the embryo's chromosomal status and intrinsic vitality. More precise identification of these "high-quality signals" allows for more accurate screening of embryos with high implantation potential. By sharing hard parameters, embryo grades based on the Istanbul Consensus (macroscopic developmental potential) and the Gardner grading (fine-grained morphological indicators) are simultaneously learned and predicted. This allows the assessment results to integrate the strengths of different evaluation systems, providing a more comprehensive and multidimensional understanding of embryo quality. For example, an embryo may perform averagely on a specific Gardner grading component but excel on the Istanbul Consensus Assessment, which indicates overall developmental potential (or vice versa). Multi-task learning can balance this information, avoiding the bias that can arise from using a single criterion. This allows for a more accurate assessment of an embryo's true developmental potential and the selection of embryos of superior overall quality. An AI model may perform well on a specific dataset, but its performance may deteriorate when applied to data from different centers, different equipment, or different culture systems, indicating a lack of generalization. By introducing a "virtual predictor" with fixed parameters and random initialization, and including its gradient norm as a penalty in the loss function, the shared encoder is forced to learn feature representations that are not only adapted to the current real-world predictor but also adapt to potential prediction tasks or data distribution changes that were not explicitly defined during training. This means that the model learns more fundamental and universal patterns of embryonic development, rather than the "biases" or noise of a specific dataset. Consequently, the model maintains high assessment accuracy when applied to new embryo data from different sources (e.g., different reproductive centers, different imaging equipment, or different culture conditions). This enhanced robustness ensures that truly high-quality embryos can be screened in a wider range of clinical practices, thereby steadily improving pregnancy rates. Using the method of the present invention, it is reasonably expected that clinical pregnancy rates can be increased by 10%-20%.

[0141] Compared with the prior art, the advantages of the present invention are:

[0142] This invention achieves more objective and accurate embryo assessment by constructing a new embryo evaluation model that takes into account both the Istanbul consensus and the Gardner grading. It also improves the accuracy of embryo assessment and screening and reduces the bias in prediction results through an innovative multi-task learning framework and optimization strategy. This invention can reduce the workload of embryologists, improve the efficiency of embryo assessment and screening, avoid observation omissions and misjudgments due to fatigue, and provide a technical solution that can be used as a reference for fields such as animal husbandry, endangered species protection, and embryonic stem cell research, helping to achieve ecological balance and sustainable development. Specifically:

[0143] (1) Improved the accuracy of embryo assessment:

[0144] The deep spatiotemporal feature extraction and multi-dimensional collaborative evaluation of this invention bring about improved accuracy and objectivity:

[0145] The ResNet3D and bidirectional LSTM (Bi-LSTM) networks employed in the encoding module of this method are able to automatically and precisely capture the spatiotemporal dynamic features of embryonic development from fertilization to the blastocyst stage. This is because ResNet3D can extract deep information about the temporal evolution of spatial morphology from volumetric data, while Bi-LSTM further captures the long-term dependencies and dynamic patterns between these spatiotemporal features. This organic combination overcomes the limitations of traditional manual assessment, which relies on the subjective visual judgment of embryologists. Because the machine can identify and quantify subtle changes that are difficult or easily overlooked by the human eye, the depth and consistency of feature extraction far exceed those of traditional manual observation, laying a solid foundation for accurate assessment.

[0146] (2) Improved the objectivity of embryo assessment:

[0147] More comprehensive and objective evaluation: By combining a multi-task learning framework with hard parameter sharing, this method can simultaneously learn and predict embryo grades based on multiple industry-recognized evaluation standards, such as the Istanbul Consensus and Gardner grading. The logic behind this is that the shared encoder is optimized to learn universal feature representations that are beneficial for all tasks. This means that the model is forced to understand embryo quality from multiple complementary perspectives (i.e., different evaluation criteria). This allows the evaluation results to integrate the advantages of different evaluation systems, overcoming the bias or incomplete information that may arise from a single evaluation standard, resulting in a more comprehensive and objective evaluation conclusion.

[0148] (3) Improved robustness and reliability of embryo assessment:

[0149] Improved robustness and reliability brought about by the dual optimization mechanism of learning quality and efficiency: Through the precise coordination of the virtual predictor mechanism and the AdaTask optimization algorithm, this invention fundamentally addresses the core challenges of feature universality and task specificity, as well as optimization efficiency and task balance in multi-task learning. This significantly improves learning quality and efficiency, enhances the robustness of the model and the reliability of the evaluation results:

[0150] The virtual predictor mechanism enhances the generality and robustness of the model: This mechanism introduces a penalty on the gradient norm of a "virtual predictor" with fixed parameters and random initialization in the optimization objective. The principle is that this forces the feature representation learned by the shared encoder to adapt not only to the current real task predictor, but also to potential prediction tasks or data distribution changes that are not explicitly defined in training. Therefore, the model's adaptability to data variations, noise interference, and different clinical scenarios that may exist in the real world is enhanced, showing better generalization and robustness;

[0151] The AdaTask optimization algorithm ensures the efficiency and balance of multi-task learning: When optimizing shared parameters in multi-task learning, there may be conflicts or imbalances in the gradients of each task. The AdaTask algorithm solves this problem by independently maintaining the cumulative gradient statistics for the gradient components that contribute to the shared parameters for each task and adaptively adjusting their learning rates. This ensures that even while pursuing a universal feature representation (guided by the virtual predictor mechanism), each specific evaluation task (such as lstanbu classification and Gardner classification) can be fully and specifically optimized, avoiding the situation where some tasks are insufficiently learned during the optimization process due to gradient dominance or overwhelmed, thereby improving the overall optimization efficiency and the balance of performance of each task;

[0152] Synergistic effects improve overall performance: The virtual predictor mechanism ensures the breadth of learning directions and the robustness of features, while AdaTask ensures the efficiency and fairness of the learning process across multiple tasks within this optimization direction. The logical advantage of combining these two approaches is that they jointly enable the model to stably and efficiently learn embryo features that are both broadly applicable (insensitive to unknown data variations and less prone to overfitting to a specific task subset) and excel across specific evaluation metrics (well-trained across all evaluation dimensions). This thoughtful combination enables this method to produce more reliable, accurate, and clinically valuable embryo assessment results in complex multi-criteria evaluation scenarios, thereby comprehensively improving the scientific nature and success rate of embryo selection.

[0153] (4) Significantly improve evaluation efficiency, lower application barriers, and expand the scope of technology application:

[0154] Automation and Efficiency: The fully automated evaluation process, based on fast, deterministic calculations based on pre-trained deep learning models, significantly reduces the workload of embryologists, freeing them from tedious, repetitive, and time-consuming film reading tasks, allowing them to focus more on clinical decision-making and difficult cases. This shortened evaluation cycle stems directly from the efficiency of the calculations, but also from minimizing human intervention, reducing the risk of evaluation errors due to subjective judgment, visual fatigue, and individual operational differences, and helping to reduce related operating costs.

[0155] (5) Wide applicability and promotion value:

[0156] The core technical solution of this invention, namely combining deep spatiotemporal feature extraction with an optimized multi-task learning framework for dynamic evaluation of complex biological processes, is not only of great significance for innovating embryo selection in the field of human assisted reproduction, but also has promotional value.

[0157] The logical basis for its widespread application lies in the inherent advantages of this methodology in terms of accuracy, objectivity, robustness, and efficiency demonstrated through the aforementioned mechanisms, which makes it highly universal. Therefore, it can be readily extended to other fields requiring detailed evaluation of dynamic sequence data, such as the selection of superior livestock breeds, the development of breeding and conservation strategies for endangered species, and the study of embryonic stem cell differentiation potential and developmental trajectories. This is expected to not only bring significant socioeconomic benefits but also strongly promote technological advancement in related interdisciplinary fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0158] Figure 1 This is a brief flow chart of the method;

[0159] Figure 2 This is a schematic diagram of the structure of the residual 3D convolution module in this method;

[0160] Figure 3 This is a schematic diagram of the structure of the bidirectional LSTM module in this method;

[0161] Figure 4 It is a schematic diagram of the specific algorithm flow of the virtual prediction mechanism in this method;

[0162] Figure 5 This is a schematic diagram of the model framework used in this method. DETAILED DESCRIPTION

[0163] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0164] A method for selecting embryos for IVF-ET based on multi-task learning using time-lapse imaging. A simplified flowchart of the entire method is available in [1]. Figure 1 , the specific process includes the following steps:

[0165] S1: Data preparation and preprocessing

[0166] S11. Data Acquisition: Collect time-lapse video or image sequences of embryonic development during IVF. Each embryo sample contains consecutive image frames covering key developmental stages (in this example, from fertilization to the blastocyst stage). In this example, each sample contains 32 grayscale images, each with a resolution of 256×256 pixels.

[0167] S12. Data preprocessing: Preprocess the original image sequence, including image alignment, brightness normalization, denoising and other operations. These operations all use existing well-known methods to improve the stability and accuracy of subsequent model processing.

[0168] S2: Extract spatiotemporal features through the encoding module:

[0169] S21. Input encoding module: Input the preprocessed image sequence (dimensions: 32×256×256×1) to the encoding module;

[0170] S22. ResNet3D Feature Extraction: We use an improved ResNet3D network to process the input sequence and extract spatiotemporal features during embryonic development. The ResNet3D network consists of three 3D residual blocks. Figure 2 The schematic diagram of the residual 3D convolution module in this method is shown. Each residual block contains two 3D convolution layers with a convolution kernel size of 13×3×3. After processing by multiple residual blocks, the feature map dimension is gradually reduced, and the final output feature map dimension is: 8×32×32×256;

[0171] S23. Regularization: Apply a SpatialDropout layer after the ResNet3D layer, maintaining the dimensions 8×32×32×256 to prevent overfitting.

[0172] S24. Feature pooling:

[0173] Apply spatial max pooling (Spatialmax.pool) to obtain features of dimension 8×256, capturing significant spatial structural information; apply spatial average pooling (Spatialavg.pool) to obtain features of dimension 8×256, preserving the overall spatial information;

[0174] S25. Temporal feature fusion: The results of max pooling and average pooling (both with dimensions of 8×256) are input into a bidirectional long short-term memory (Bi-LSTM) layer. Figure 3This figure shows the structure of the Bi-LSTM module used in this method. This Bi-LSTM layer contains 128 units in each direction, processing information in the time dimension (8 time steps) to capture the temporal dependencies of embryonic development. The Bi-LSTM layer ultimately outputs a 256-dimensional feature vector that incorporates spatiotemporal information.

[0175] S3: Multi-task learning and optimization

[0176] S31. Shared representation: The 256-dimensional feature vector output from step S2 is used as the shared representation , which is fed into the subsequent task-specific predictor;

[0177] S32. Task-specific predictors: Build independent predictors for each evaluation task (e.g., Istanbul consensus rating and Gardner rating) ; In this embodiment, the predictor adopts a multi-layer perceptron (MLP) structure.

[0178] S33. Multi-task learning framework application:

[0179] Adopt hard parameter sharing structure and shared encoder Parameters of , introduce virtual predictor , which has the same structure as the true predictor but the parameters are randomly initialized and fixed. Figure 4 The specific algorithm flow diagram of the virtual prediction mechanism in this method is shown in detail. The penalty term of the gradient norm of the virtual predictor is added to the training target. , to improve the versatility of shared representation; adopt the Frank-Wolfe-based method, regard the problem as a multi-objective optimization, and iteratively update the shared parameters by solving sub-problems to find the Pareto optimal solution;

[0180] S34. Model training and optimization:

[0181] Loss function: For classification tasks (Istanbul consensus and Gardner classification), the cross entropy loss function (Cross EntropyLoss) is used as The total loss is the sum of the losses of each task (which may include a virtual gradient penalty term);

[0182] Optimizer: Use AdaTask optimization method (LAdaTask-RMSProp) to update model parameters AdaTask maintains independent cumulative gradient statistics for each task and shared parameters (or layers) to avoid task conflicts and improve optimization efficiency. Set an appropriate initial learning rate , momentum parameters (such as Adam's ), regularization coefficient etc. hyperparameters;

[0183] Training process: Use the labeled embryo dataset to train the model, and iteratively update the network parameters through backpropagation and optimizer until the model converges or reaches the preset number of training rounds.

[0184] S4: Classification prediction and embryo selection:

[0185] S41. Feature Input: The 256-dimensional shared feature vector output by the encoding module is input into each task-specific predictor (MLP) of the classification module.

[0186] S42.MLP processing:

[0187] The first hidden layer has 128 neurons, ReLU activation function, followed by Dropout (0.5),

[0188] Second hidden layer: 64 neurons, ReLU activation function, followed by Dropout (0.5),

[0189] S43. Output prediction:

[0190] Istanbul consensus classification task: The output layer has 5 neurons, applies the Softmax activation function, and outputs a probability distribution corresponding to 5 quality levels;

[0191] Gardner grading task: The output layer has 6 neurons, and the Softmax activation function is applied to output the probability distribution corresponding to the 6 developmental levels;

[0192] S44. Embryo selection decision:

[0193] Based on the multiple prediction results output by the model, combined with clinical experience and specific needs, this embodiment proposes a weighted scoring system to formulate an embryo selection strategy. This system assigns weights to the two prediction results and calculates a comprehensive score for the embryo, thereby achieving quantitative selection. Specifically, the comprehensive score (Score) of each embryo is calculated using the following formula:

[0194] ,

[0195] in, and Represent the weights of the Istanbul grade probability and the Gardner grade probability, and the sum of the two is 1. According to the final score, the most suitable embryo for transplantation is selected, such as Figure 5 As shown in the figure, a schematic diagram of the complete model framework used in this method is shown.

[0196] The above embodiments are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A method for selecting IVF-ET embryos based on multi-task learning using time-lapse imaging, characterized in that: The steps include: S1: Data preparation and preprocessing: S11. Data Acquisition: Collect time-lapse video or image sequences of embryonic development during IVF. S12. Data preprocessing: preprocessing the time-lapse imaging video or image sequence; S2: Extract spatiotemporal features through the encoding module: S21. Input encoding module: input the image sequence preprocessed in S12 into the encoding module; S22. ResNet3D Feature Extraction: Use an improved ResNet3D network to process the input sequence and extract spatiotemporal features during embryonic development. S23. Regularization: Apply a spatial Dropout layer after the ResNet3D layer to preserve dimensionality. S24. Feature pooling: Applying spatial maximum pooling, we obtain features with a dimension of 8×256, which captures significant spatial structural information; applying spatial average pooling, we obtain features with a dimension of 8×256, which preserves the overall spatial information; S25. Temporal feature fusion: Feed the results of max pooling and average pooling into a bidirectional long short-term memory network layer; S3: Multi-task learning and optimization: S31. Shared representation: The feature vector output from step S2 is used as the shared representation , which is input to the subsequent task-specific predictor, where is the input sample, For shared encoders, is the encoded feature vector; S32. Task-specific predictors: Build independent predictors for each evaluation task ; S33. Multi-task learning framework application: Adopt hard parameter sharing structure and shared encoder parameters; the Frank-Wolfe-based method is used to treat the problem as a multi-objective optimization problem, and the shared parameters are iteratively updated by solving subproblems to find the Pareto optimal solution; S34. Model training and optimization: Loss function: For classification tasks, the cross entropy loss function is used as ;Total loss is the sum of losses of each task; ， in: is the cross entropy loss of the kth task; is the true label of category c, that is, the value in one-hot encoding, 0 or 1; log is the natural logarithm function; The probability of category c predicted by the model; Optimizer: Use AdaTask optimization method to update model parameters ; Training process: Use the labeled embryo dataset to train the model, and iteratively update the network parameters through backpropagation and optimizer until the model converges or reaches the preset number of training rounds; S4: Classification prediction and embryo selection: S41. Feature input: The shared feature vector output by the encoding module is input into each task-specific predictor of the classification module; S42. Multilayer Perceptron Processing: First hidden layer: 128 neurons, ReLU activation function, followed by Dropout (0.5), Second hidden layer: 64 neurons, ReLU activation function, followed by Dropout (0.5), S43. Output prediction: Istanbul consensus classification task: The output layer has 5 neurons, applies the Softmax activation function, and outputs a probability distribution corresponding to 5 quality levels; Gardner grading task: The output layer has 6 neurons, and the Softmax activation function is applied to output the probability distribution corresponding to the 6 developmental levels; S44. Embryo selection decision: Develop an embryo selection strategy based on multiple predictions output by the model, combined with clinical experience and specific needs.

2. The method according to claim 1, characterized in that In step S11, each embryo sample contains continuous image frames covering key developmental stages; each sample contains 32 frames of grayscale images, and each frame has a resolution of 256×256 pixels; In step S12, the preprocessing includes image alignment, brightness normalization, and denoising operations.

3. The method according to claim 1, characterized in that In S22, the ResNet3D network includes three 3D residual blocks, each residual block includes two 3D convolutional layers, and the convolution kernel size is 13×3×3; after processing by multiple residual blocks, the dimension of the feature map is gradually reduced, and the final output feature map dimension is: 8×32×32×256; In S23, the dimension 8×32×32×256 is kept unchanged to prevent overfitting; In S24, spatial maximum pooling and spatial average pooling are applied, and both obtain features with a dimension of 8×256; In S25, the bidirectional long short-term memory network layer contains 128 units in each direction, processes information in the time dimension, and captures the temporal dependency of embryonic development; the bidirectional long short-term memory network layer ultimately outputs a 256-dimensional feature vector that integrates spatiotemporal information.

4. The method according to claim 1, wherein In S32, the predictor adopts a multi-layer perceptron structure, including: Input layer: receives feature representations from the shared encoder; Hidden layer: contains multiple fully connected layers, each followed by a ReLU activation function and Dropout regularization for nonlinear transformation of features and prevention of overfitting; Output layer: Use the Softmax activation function according to the specific task and output the probability distribution of the corresponding level: ， in: is the probability that the input x belongs to category c, is the raw score (logits) of category c, is the total number of categories of task k, exp is the natural exponential function; The tasks include the Istanbul Consensus Classification and the Gardner Classification.

5. The method according to claim 1, wherein In S33, the hard parameter sharing structure includes a shared encoder and Task-specific predictors , the objective function is the sum of all task losses: ， in: is the set of all model parameters, are the parameters of the shared encoder, are the parameters of the k-th task-specific predictor, For shared encoders, is the kth task-specific predictor, is the i-th input sample, is the true label of the i-th sample on the k-th task, is the loss function of the kth task, is the number of samples, K is the number of tasks, Represents function composition.

6. The method according to claim 1, wherein In S33, a virtual predictor is introduced , which has the same structure as the real predictor but with randomly initialized and fixed parameters; a penalty term of the gradient norm of the virtual predictor is added to the training objective , to improve the versatility of shared representations, Among them, the loss function of a single task is to add the penalty term of the virtual predictor gradient norm to the original loss function: ， The specific form is: ， in: is the predicted value of the k-th task for the i-th sample, is the predicted value of the virtual predictor for the i-th sample, is the shared representation of the i-th sample, is the penalty term coefficient, which is used to balance the task loss and the virtual gradient norm; The total loss function of multi-task is the sum of all task losses: ， Among them, the loss of each task consists of two parts: True predictor loss : By the true predictor Generate predictions and calculate the traditional mission losses; Virtual predictor gradient norm penalty: through virtual predictor Generate predictions , calculate the gradient norm of its loss with respect to its own parameters to improve the versatility of shared representation; in: is the total loss of the k-th task, is the sample size, is the true predictor loss function for the k-th task, is the true label of the i-th sample of the k-th task, is the predicted value of the true predictor for the i-th sample of the k-th task, is the virtual predictor loss function for the k-th task, is the predicted value of the virtual predictor for the i-th sample of the k-th task, is the virtual predictor parameter for the kth task, is the gradient of the virtual predictor parameters, is the Frobenius norm, is the penalty coefficient.

7. The method according to claim 6, characterized in that In step S34, the classification task includes Istanbul consensus and Gardner classification; the total loss includes a penalty term of the virtual predictor gradient norm; The AdaTask optimization method is: Initialization: Set the exponential decay factor v and the initial learning rate , smoothing factor ε, initialize the cumulative gradient variable for each task k ; Gradient calculation: At training step t, the gradient of the parameter θ is calculated for each task ; Cumulative gradient update: For each task, the cumulative gradient is updated by exponential decay average: ， Parameter update amount calculation: Calculate the parameter update amount for each task: ， Model parameter update: Update the model parameters according to the parameter update amount of all tasks: ， Where: t is the training step, k is the task index, i is the parameter index, is the cumulative gradient of task k with respect to parameter i at step t, is the gradient of task k with respect to parameter i at step, v is the exponential decay factor, is the initial learning rate, ε is the smoothing factor, is the update amount of task k to parameter i, is the value of parameter i at step t.

8. The method according to claim 1, characterized in that In S44, the final prediction result is obtained by taking the category with the highest probability: ， in: is the predicted category of the kth task, argmax means taking the category c with the maximum probability, is the predicted probability that the input x belongs to category c.

Images