Intelligent embryo quality evaluation system and evaluation method based on deep learning

Abstract

The application discloses an embryo quality intelligent evaluation method, comprising the following steps: establishing a pronucleus stage instance segmentation model; establishing a pronucleus stage target tracking model based on DeepSort; establishing a cleavage stage cell detection model; establishing a cleavage stage cell fragmentation degree rating model; establishing a semantic segmentation model based on U-Net; establishing a quality rating model based on Inception V3; and based on the six models established above, establishing an embryo quality intelligent evaluation model based on a graph convolutional neural network, so as to evaluate the embryo quality. The application models different stages in the embryo development process respectively, fuses the embryo characteristics of different stages, trains the graph convolutional neural network, effectively combines the characteristic parameters with physical significance in different periods, establishes the embryo quality intelligent evaluation model, and by using the intelligent evaluation method instead of manual evaluation, the measuring efficiency is greatly improved while the high precision is maintained.

Description

Technical Field

[0001] This invention relates to a deep learning-based intelligent embryo quality assessment system and method. Background Technology

[0002] Selecting embryos with higher developmental potential for implantation is a key research issue in assisted reproductive technology. Currently, the most widely used method in clinical practice is morphological scoring, where embryologists observe the morphology of embryos at certain fixed time points during embryonic development, comprehensively assess embryo quality, and select embryos with good developmental potential for transfer. However, this approach has several drawbacks. First, embryonic development is dynamic, and observing embryonic morphology at fixed time points can overlook important characteristics during development. Second, manual assessment is not only time-consuming and labor-intensive, but also requires strong professional knowledge and is subject to a degree of subjectivity.

[0003] In recent years, Time-Lapse Imaging (TLI) has emerged as a non-invasive technique for assessing embryo quality. It involves installing a built-in camera within an incubator to automatically capture images of each embryo at regular intervals. This technology combines the incubator with an imaging system, allowing for complete and detailed recording of embryonic development without disturbing the embryo culture environment.

[0004] Time-lapse imaging (TLI) helps improve the success rate of embryo selection and transfer. Therefore, how to utilize artificial intelligence technology to analyze massive amounts of time-lapse embryo images has become a key research issue in the intersection of medicine and artificial intelligence. Based on TLI-recorded embryonic development information, previous assessments of embryo quality mostly relied on doctors' clinical experience or traditional machine learning methods to predict embryonic development quality. In recent years, a series of deep learning-based intelligent assessment methods for TLI embryonic development have emerged, achieving numerous technological breakthroughs. However, these achievements mostly address two aspects: firstly, directly constructing embryo quality assessment models using embryonic development sequence images, which lacks interpretability and has weak generalization performance; secondly, constructing embryo quality assessment models based on non-sequence images of embryonic development. These methods mostly focus on a specific stage of embryonic development, directly rating or predicting the morphological characteristics of that stage, thus potentially losing some important feature parameters and causing uncertainty in the assessment results. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a simple and low-cost intelligent embryo quality assessment system based on deep learning, and a simple algorithm with high assessment accuracy for intelligent embryo quality assessment.

[0006] The technical solution of the present invention to solve the above-mentioned technical problems is: a deep learning-based intelligent embryo quality assessment system, including a pronuclear stage instance segmentation module, a pronuclear stage target tracking module, a cleavage stage target detection module, a cleavage stage cell fragmentation degree rating module, a blastocyst stage semantic segmentation module, a blastocyst stage quality rating module, and an intelligent embryo quality assessment module.

[0007] The prokaryotic phase instance segmentation module is used to segment three different target regions: cell, cytoplasm, and prokaryote; the prokaryotic phase instance segmentation module is connected to the prokaryotic phase target tracking module.

[0008] The pronuclear stage target tracking module is used to track the trajectories of female and male pronuclei among pronuclear targets. It further distinguishes between male and female pronuclei based on the tracking trajectories. The pronuclear stage target tracking module is connected to the embryo quality intelligent assessment module.

[0009] The pronuclear stage instance segmentation module and the pronuclear stage target tracking module calculate the spatial morphological parameters of the embryonic pronuclear stage based on the segmentation results of four different target regions: cell, cytoplasm, female pronucleus, and male pronucleus.

[0010] The cleavage phase target detection module is used to detect the number, location and uniformity of cells during cell division. The cleavage phase target detection module is connected to the cleavage phase cell fragmentation degree rating module.

[0011] The cleavage-stage cell fragmentation rating module is used to rate the degree of cell fragmentation, and the cleavage-stage cell fragmentation rating module is connected to the embryo quality intelligent assessment module.

[0012] The blastocyst semantic segmentation module is used to accurately segment the two different target regions, the trophoblast and the inner cell mass. The blastocyst semantic segmentation module is connected to the blastocyst quality rating module.

[0013] The blastocyst quality rating module is used to rate the quality of the trophoblast and inner cell mass. The blastocyst quality rating module is connected to the embryo quality intelligent assessment module.

[0014] The intelligent embryo quality assessment module constructs a feature vector of the embryo image using the aforementioned spatial morphological parameters, cell location, cell number, cell uniformity, cell fragmentation degree, trophoblast and inner cell mass quality, and trains an intelligent embryo quality assessment model based on graph convolutional neural network to predict the embryo quality assessment result.

[0015] An intelligent method for assessing embryo quality includes the following steps:

[0016] S1: Establish a prokaryotic instance segmentation model based on the Mask R-CNN network through the prokaryotic instance segmentation module, and train and optimize the prokaryotic instance segmentation model;

[0017] S2: Establish a prokaryotic target tracking model based on DeepSort through the prokaryotic target tracking module, and train and optimize the prokaryotic target tracking model;

[0018] S3: Establish a cleavage stage cell detection model through the cleavage stage target detection module, and train and optimize the cleavage stage cell detection model;

[0019] S4: Establish a cell fragmentation rating model for the cleavage stage using the cell fragmentation rating module, and train and optimize the cell fragmentation rating model for the cleavage stage.

[0020] S5: Establish a U-Net-based semantic segmentation model using the blastocyst stage semantic segmentation module, and train and optimize the semantic segmentation model.

[0021] S6: Establish a quality rating model based on Inception V3 through the blastocyst stage quality rating module, and train and optimize the quality rating model;

[0022] S7: Based on the six models trained in steps S1 to S6, establish an intelligent embryo quality assessment model based on graph convolutional neural networks, and train and optimize the intelligent embryo quality assessment model to assess embryo quality.

[0023] The specific steps of step S1 in the above-mentioned intelligent embryo quality assessment method are as follows:

[0024] S11: Create a dataset;

[0025] First, using Photoshop, the outlines of four targets—cells, cytoplasm, female pronucleus, and male pronucleus—are delineated on the pronuclear image of embryonic development using different colors, creating mask images for cells, cytoplasm, female pronucleus, and male pronucleus. Then, COCO format annotation files are generated using the mask images of each target. Based on the original images and annotation files, the dataset required for the pronuclear stage instance segmentation model with annotated images is created, and the dataset is divided into training, validation, and test sets.

[0026] S12: Construct a prokaryotic phase instance segmentation model;

[0027] A prokaryotic instance segmentation model based on Mask R-CNN network is constructed. This prokaryotic instance segmentation model uses ResNet101 network as the backbone network and Feature Pyramid Network (FPN) network as the neck to fuse features of different scales. ROIAlign is used instead of ROIPooling operation. The detection head includes three different branches: classification, regression, and masking, to predict classification probability, target location box, and segmentation mask.

[0028] S13: Train and optimize the prokaryotic instance segmentation model;

[0029] The original images of pronuclear stage embryos were used for model training and optimization;

[0030] The output of the pronuclear phase instance segmentation model includes the following parts: the classification branch output is the classification probability of different targets, the regression branch output is the coordinate value of the predicted bounding box of different targets, and the mask branch output is the pixel-level mask of different targets.

[0031] The output results are compared with the true labels, the loss of the pronuclear phase instance segmentation model is calculated, and the network weights are updated using stochastic gradient descent. The training process is iterated continuously to optimize the network.

[0032] S14: Obtain the prokaryotic instance segmentation prediction results using the trained prokaryotic instance segmentation model.

[0033] The specific steps of step S2 in the above-mentioned intelligent embryo quality assessment method are as follows:

[0034] S21: Create a dataset;

[0035] Using the dataset conversion method provided on the MOT official website, the COCO format dataset already created in S11 was converted into a target tracking dataset in MOT format, where each frame image only includes two different targets: female pronucleus and male pronucleus.

[0036] S22: Construct a target tracking model based on DeepSort, and use the target tracking dataset to train and optimize the target tracking model;

[0037] The overall processing flow of the DeepSort-based target tracking model is as follows: First, a ResNet50-based appearance extraction model is trained. The input of this model is the detection box region, and the output is the feature vector in the embedding space. The loss function used in the appearance model training process is triple loss, which is a loss directly constructed from the feature vectors, making the feature vectors of different targets more distinguishable and the feature vectors of the same targets more similar. Kalman filtering is used to obtain the motion states of the two types of targets. A similarity matrix is ​​calculated by combining the target recognition model and the motion states. The Hungarian algorithm is used for cascaded matching. Finally, Kalman filtering is used to update the trajectory and motion states of the two types of targets, thereby improving the accuracy of target recognition.

[0038] S23: Obtain the target tracking model prediction results through the trained target tracking model;

[0039] S24: Calculate the spatial morphological parameters of the pronuclear stage;

[0040] By combining the prokaryotic phase instance segmentation model and the target tracking model, regions of four different targets—cell, cytoplasm, male pronucleus, and female pronucleus—are obtained. Based on the area of ​​different regions, spatial morphological parameters of the prokaryotic phase are calculated, including the maximum cell contraction amplitude, the maximum proportion of the cytoplasmic halo, whether the male and female pronuclei are side by side, the distance from the side-by-side point of the male and female pronuclei to the cell center, the maximum diameter difference between the male and female pronuclei during the entire side-by-side period, and the distance from the initial position of the male pronucleus to the cell center.

[0041] The above-mentioned intelligent embryo quality assessment method, specifically step S3, includes the following steps:

[0042] S31: Create a dataset;

[0043] The LabelImg software was used to label the cell locations in embryo images, and a dataset conforming to the VOC format was created. The dataset was divided into three parts: training set, test set, and validation set in a 7:2:1 ratio.

[0044] S32: Construct and train a cleavage-stage cell detection model;

[0045] First, data augmentation is performed on the images to construct a cell detection model based on YOLOv5. The cell detection model is trained and optimized using images from the cleavage stage to detect the number of cells during cell division, accurately predict the bounding box of each cell, and judge the uniformity of cells. The Adam algorithm is used to optimize the weight parameters of the convolutional neural network.

[0046] S33: Determination of cell uniformity during cleavage;

[0047] The trained cleavage-stage cell detection model is used to predict the number of cleavage-stage cells in the test image and the location of each cell. The area of ​​the location box is approximated to the area of ​​each cell, and the standard deviation of the area of ​​all cells is calculated to determine the uniformity of cells. If the standard deviation is less than the set threshold, the cells in the cleavage-stage embryo image are considered to be relatively uniform; otherwise, they are considered to be non-uniform.

[0048] The above-mentioned intelligent embryo quality assessment method, specifically step S4, includes the following steps:

[0049] S41: Create a dataset;

[0050] Based on the cell location predicted by the cell target detection model in step S2, cells in the embryo image are extracted as ROI regions, and the ROI regions are used as training data for the cleavage stage cell fragmentation degree rating model.

[0051] Based on the ratio of cell fragment area to blastomere area, cell images are divided into four different fragmentation levels: 0%-10%, 10%-20%, 20%-50%, and 50%-100%; and the cell images are divided into training set, validation set, and test set in an 8:1:1 ratio.

[0052] S42: Construct and train a grading model for the degree of cell fragmentation during cleavage;

[0053] A cell fragmentation rating model based on the ResNet50 network was constructed, and the model was trained using cross-entropy as the loss function, and a label smoothing function was introduced.

[0054] S43: Obtain cell fragmentation rating prediction results through a trained cell fragmentation rating model.

[0055] The above-mentioned intelligent embryo quality assessment method, specifically step S5, includes the following steps:

[0056] S51: Create a dataset;

[0057] Using Photoshop, the outlines of trophoblast cells and inner cell mass in blastocyst images were delineated with lines of different colors. The outlines were generated pixel by pixel, binarized, and then erosion and dilation operations were performed to generate a fine mask. The images were divided into training, validation, and test sets in an approximately 8:1:1 ratio.

[0058] S52: Construct and train a semantic segmentation model for the blastocyst stage;

[0059] A blastocyst-stage semantic segmentation model based on the U-Net network was constructed. The U-Net network was trained using the S51 training set, and the model loss was calculated. The network weights were updated using stochastic gradient descent. This process was iterated until the model converged.

[0060] S53: Using a trained semantic segmentation model for the blastocyst stage, the segmented trophoblast cells and inner cell mass regions are obtained.

[0061] The above-mentioned intelligent embryo quality assessment method, specifically step S6, includes the following steps:

[0062] S61: Create a dataset;

[0063] Based on the S5 segmentation of trophoblast cells and inner cell mass regions, the images of trophoblast cells and inner cells are labeled. Inner cell mass is graded as follows: Grade A, cell number greater than a preset value 'a', densely packed; Grade B, cell number between preset values ​​'a' and 'b', loosely packed, 'a' > 'b'; Grade C, cell number less than the preset value 'b'. Trophoblast cells are graded as follows: Grade X, epithelial cell number greater than a preset value 'c', dense structure; Grade Y, epithelial cell number between preset values ​​'c' and 'd', loose structure, 'c' > 'd'; Grade Z, epithelial cell number less than the preset value 'd'. After arranging the two letters, the order from highest to lowest is: AX, AY, BX, BY > AZ, BZ > CX, CY, CZ. This creates a dataset for trophoblast cell and inner cell mass quality rating.

[0064] S62: Build and train a quality rating model;

[0065] A quality rating model based on Inception V3 is constructed. The S61 dataset is used to train the quality rating model. The prediction results are compared with the labels, the model loss is calculated, and the network weights are updated using stochastic gradient descent. This process is iterated until the model converges.

[0066] S63: Obtain the quality rating results of the trophoblast and inner cell mass by using a trained quality rating model.

[0067] The above-mentioned intelligent embryo quality assessment method, specifically step S7, includes the following steps:

[0068] S71: Create a dataset;

[0069] Combining the prediction results of steps S1, S2, S3, S4, S5 and S6, all prediction results are used as feature vectors of the embryo image and normalized to form the dataset for the graph convolutional neural network.

[0070] S72: Construct and train an intelligent embryo quality assessment model;

[0071] 1) Feature extraction: First, the n image features obtained from the six steps S1, S2, S3, S4, S5 and S6 at different stages of embryonic development are input into the vertex feature extraction network. The specific process is as follows: First, the features are processed through a VGG-16 CNN network to obtain n 4096-dimensional vectors. Then, principal component analysis (PCA) is used to obtain n 128-dimensional vectors. These n 128-dimensional vectors constitute the vertices of the graph structure.

[0072] 2) Construct a graph structure. Based on the relationships between the features of each vertex, establish an adjacency matrix of edges between vertices to form a topological structure represented by the graph.

[0073] 3) Train a deep graph convolutional neural network, which is composed of a series of graph convolutional modules connected in series and introduces an attention module. The output of the network is a Bx1 dimensional vector. Use the Adam optimization algorithm to calculate the gradient of the network error for each weight parameter in backpropagation, and obtain the new weights through the parameter update process. Iteratively calculate the model weights until the predetermined small loss is reached and the best prediction value is obtained.

[0074] 4) Prediction: For the features represented by the graph structure, a regression prediction model is constructed to give three levels of embryo quality, namely the best, medium and worst quality.

[0075] The beneficial effects of this invention are as follows:

[0076] 1. This invention utilizes deep learning methods to model different stages of embryonic development and integrates the characteristics of embryos at different stages to train a graph convolutional neural network. This effectively combines physically meaningful feature parameters from different periods to establish an intelligent embryo quality assessment model. Essentially, it simulates the standards and methods used by embryologists to evaluate embryo quality, thereby achieving the goal of intelligent embryo quality assessment. By using intelligent assessment methods to replace manual assessment, it greatly improves measurement efficiency while maintaining high accuracy, and has high practicality.

[0077] 2. The design and selection of the deep neural network in this invention (such as data augmentation strategies, loss function design, image resolution adjustment, etc.) are all based on the principle of improving accuracy. It has high reliability in terms of accuracy of various indicators, thus ensuring its practicality. Attached Figure Description

[0078] Figure 1 This is an overall flowchart of the evaluation method of the present invention.

[0079] Figure 2 Flowchart for instance segmentation and target tracking during the pronuclear phase.

[0080] Figure 3 This is a graph showing the prediction results of the instance segmentation model during the pronuclear phase.

[0081] Figure 4 This is a graph showing the prediction results of the target tracking model during the pronuclear phase.

[0082] Figure 5 This is a diagram showing the spatial morphological characteristics of the embryo.

[0083] Figure 6 This is a flowchart of a cell detection model for the cleavage stage.

[0084] Figure 7 This is a graph showing the training results of the cell detection model during the cleavage stage.

[0085] Figure 8 This is a graph showing the predicted results of a cell detection model during the cleavage stage.

[0086] Figure 9 Flowchart for the grading model of cell fragmentation degree during cleavage.

[0087] Figure 10 This is a graph showing the prediction results of the cell fragmentation level rating model during the cleavage stage.

[0088] Figure 11 Flowchart of semantic segmentation and quality rating during the blastocyst stage.

[0089] Figure 12 This is a graph showing the prediction results of the semantic segmentation model during the blastocyst stage.

[0090] Figure 13 This is a graph showing the prediction results of the rating module for trophoblast cells and inner cell mass during the blastocyst stage.

[0091] Figure 14 This is a graph showing the results of intelligent embryo quality assessment based on graph convolutional neural networks. Detailed Implementation

[0092] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0093] like Figure 1 As shown, a deep learning-based intelligent embryo quality assessment system includes a pronuclear stage instance segmentation module, a pronuclear stage target tracking module, a cleavage stage target detection module, a cleavage stage cell fragmentation degree rating module, a blastocyst stage semantic segmentation module, a blastocyst stage quality rating module, and an intelligent embryo quality assessment module.

[0094] The prokaryotic phase instance segmentation module is used to segment three different target regions: cell, cytoplasm, and prokaryote; the prokaryotic phase instance segmentation module is connected to the prokaryotic phase target tracking module.

[0095] The pronuclear stage target tracking module is used to track the trajectories of female and male pronuclei among pronuclear targets. It further distinguishes between male and female pronuclei based on the tracking trajectories. The pronuclear stage target tracking module is connected to the embryo quality intelligent assessment module.

[0096] The pronuclear stage instance segmentation module and the pronuclear stage target tracking module calculate the spatial morphological parameters of the embryonic pronuclear stage based on the segmentation results of four different target regions: cell, cytoplasm, female pronucleus, and male pronucleus.

[0097] The cleavage phase target detection module is used to detect the number, location and uniformity of cells during cell division. The cleavage phase target detection module is connected to the cleavage phase cell fragmentation degree rating module.

[0098] The cleavage-stage cell fragmentation rating module is used to rate the degree of cell fragmentation, and the cleavage-stage cell fragmentation rating module is connected to the embryo quality intelligent assessment module.

[0099] The blastocyst semantic segmentation module is used to accurately segment the two different target regions, the trophoblast and the inner cell mass. The blastocyst semantic segmentation module is connected to the blastocyst quality rating module.

[0100] The blastocyst quality rating module is used to rate the quality of the trophoblast and inner cell mass. The blastocyst quality rating module is connected to the embryo quality intelligent assessment module.

[0101] The intelligent embryo quality assessment module constructs a feature vector of the embryo image using the aforementioned spatial morphological parameters, cell location, cell number, cell uniformity, cell fragmentation degree, trophoblast and inner cell mass quality, and trains an intelligent embryo quality assessment model based on graph convolutional neural network to predict the embryo quality assessment result.

[0102] An intelligent method for assessing embryo quality includes the following steps:

[0103] S1: Establish a prokaryotic instance segmentation model based on the Mask R-CNN network through the prokaryotic instance segmentation module, and train and optimize the prokaryotic instance segmentation model.

[0104] like Figure 2 As shown, the specific steps of step S1 are as follows:

[0105] S11: Create a dataset;

[0106] First, using Photoshop, the outlines of four targets—cells, cytoplasm, female pronucleus, and male pronucleus—were delineated on the pronuclear image of embryonic development using different colors, creating mask images for cells, cytoplasm, female pronucleus, and male pronucleus. Then, COCO format annotation files were generated using the mask images for each target. Based on the original images and annotation files, a dataset for the pronuclear stage instance segmentation model with annotated images was created. This dataset contains 3202 images, which were divided into training, validation, and test sets in an 8:1:1 ratio.

[0107] S12: Construct a prokaryotic phase instance segmentation model;

[0108] A prokaryotic instance segmentation model based on Mask R-CNN network is constructed. This model uses ResNet101 network as the backbone network and Feature Pyramid Network (FPN) network as the neck to fuse features of different scales. ROIAlign is used instead of ROIPooling to effectively reduce coordinate bias. The detection head includes three different branches: classification, regression, and masking, to predict classification probability, target location box, and segmentation mask.

[0109] S13: Train and optimize the prokaryotic instance segmentation model;

[0110] The original images of pronuclear stage embryos were used for model training and optimization. All images were adjusted to a size of 512*512 to minimize the computational load on the network.

[0111] Based on the characteristic that the aspect ratio of the target in the embryo image is about 1:1, setting the predetermined anchor box scale to 1 in the Region Proposal Network (RPN) can greatly reduce the complexity of the model.

[0112] By combining cross-entropy loss and Dice coefficient loss as the loss function of the mask branch, detection and segmentation performance can be improved.

[0113] The output of the pronuclear phase instance segmentation model includes the following parts: the classification branch output is the classification probability of different targets, the regression branch output is the coordinate value of the predicted bounding box of different targets, and the mask branch output is the pixel-level mask of different targets.

[0114] The output results are compared with the true labels, the loss of the pronuclear phase instance segmentation model is calculated, and the network weights are updated using stochastic gradient descent. The training process is iterated continuously to optimize the network.

[0115] S14: Using the trained prokaryotic instance segmentation model, the prokaryotic instance segmentation prediction results are obtained as follows: Figure 3 As shown.

[0116] S2: Establish a prokaryotic target tracking model based on DeepSort through the prokaryotic target tracking module, and train and optimize the prokaryotic target tracking model.

[0117] The specific steps of step S2 are as follows:

[0118] S21: Create a dataset;

[0119] Using the dataset conversion method provided on the MOT official website, the COCO format dataset already created in S11 was converted into a target tracking dataset in MOT format, where each frame image only includes two different targets: female pronucleus and male pronucleus.

[0120] S22: Construct a target tracking model based on DeepSort, and use the target tracking dataset to train and optimize the target tracking model;

[0121] First, the overall processing flow of the DeepSort-based target tracking model is as follows: First, a ResNet50-based appearance extraction model is trained. The input of this model is the detection box region, and the output is the feature vector of the embedding space. The loss function used in the appearance model training process is triple loss, which is a loss directly constructed from the feature vectors. This improves the discriminative power between feature vectors of different targets and makes feature vectors of the same target more similar. The Adam algorithm is used to optimize the weight parameters of the convolutional neural network, with a learning rate of 0.002, 100 iterations, and a batch size of 8. Kalman filtering is used to obtain the motion states of the two types of targets; a similarity matrix is ​​calculated by combining the target recognition model and the motion states; cascaded matching is performed using the Hungarian algorithm; and the motion states of the two types of targets are updated using Kalman filtering, thereby improving the accuracy of target recognition. Based on the experience of clinical embryologists, during embryonic development, the female pronucleus and male pronucleus exhibit different motion characteristics; that is, compared with the male pronucleus, the female pronucleus travels a longer distance and moves faster. The motion trajectories of the male and female pronuclei predicted by the target tracking model can effectively distinguish between the two pronuclei.

[0122] S23: Obtain the target tracking model's prediction results using the trained target tracking model, such as... Figure 4 As shown;

[0123] S24: Calculate the spatial morphological parameters of the pronuclear stage, such as... Figure 5 As shown;

[0124] By combining the prokaryotic phase instance segmentation model and the target tracking model, regions of four different targets—cell, cytoplasm, male pronucleus, and female pronucleus—are obtained. Based on the area of ​​different regions, spatial morphological parameters of the prokaryotic phase are calculated, including the maximum cell contraction amplitude, the maximum proportion of the cytoplasmic halo, whether the male and female pronuclei are side by side, the distance from the side-by-side point of the male and female pronuclei to the cell center, the maximum diameter difference between the male and female pronuclei during the entire side-by-side period, and the distance from the initial position of the male pronucleus to the cell center.

[0125] S3: Establish a cleavage-stage cell detection model through the cleavage-stage target detection module, and train and optimize the cleavage-stage cell detection model.

[0126] Step S3 is as follows:

[0127] S31: Create a dataset;

[0128] The LabelImg software was used to label cell locations in embryo images to create a dataset conforming to the VOC format. The dataset was divided into three parts in a 7:2:1 ratio: training set, test set, and validation set. The training set contains 5600 images, the test set contains 1600 images, and the validation set contains 800 images.

[0129] S32: Construct and train a cleavage-stage cell detection model;

[0130] First, image data augmentation was performed to construct a cell detection model based on YOLOv5. Cleavage-stage images were used for training and optimization of the model to detect the number of cells during cell division, accurately predict the bounding box of each cell, and assess cell uniformity. The Adam algorithm was used to optimize the weight parameters of the convolutional neural network, with a learning rate of 0.001, 300 iterations, and a batch size of 16. After training, the network can predict features such as cell number, area, and location.

[0131] The flowchart of the cleavage stage cell detection model is as follows: Figure 6 As shown, the training results of the cleavage stage cell detection model are as follows: Figure 7 As shown. Figure 7 (a) - (c) are the convergence curves of the model’s localization loss, target loss and classification loss on the training set, respectively; Figure 7 (d) and (e) represent the model's precision and recall on the validation set, respectively; Figure 7 (f)-(h) are the convergence curves of the model’s localization loss, target loss and classification loss on the validation set, respectively; Figure 7 (i) represents the average accuracy of the model across all classes when the IOU is greater than 0.5 on the validation set. Figure 7 (j) represents the model’s average accuracy across all classes on the validation set at different IoU values ​​(from 0.5 to 0.95, with a step size of 0.05).

[0132] S33: Determination of cell uniformity during cleavage;

[0133] The trained cleavage-stage cell detection model is used to predict the number of cleavage-stage cells in the test image and the location of each cell. The area of ​​the location box is approximated to the area of ​​each cell, and the standard deviation of the area of ​​all cells is calculated to determine the uniformity of cells. If the standard deviation is less than the set threshold, the cells in the cleavage-stage embryo image are considered to be relatively uniform; otherwise, they are considered to be non-uniform.

[0134] The prediction results of the cleavage stage cell detection model are as follows: Figure 8 As shown. The cleavage-stage cell detection model can predict cell location and its confidence level, cell number, and area of ​​each prediction box. Figure 8It can be seen that the mAP (average precision across all classes) of the cleavage stage cell detection model on the test set is 0.96.

[0135] S4: Establish a cell fragmentation rating model for the cleavage stage using the cell fragmentation rating module, and train and optimize the cell fragmentation rating model for the cleavage stage.

[0136] Step S4 is as follows:

[0137] S41: Create a dataset;

[0138] Based on the cell location predicted by the cell target detection model in step S2, cells in the embryo image are extracted as ROI regions, and the ROI regions are used as training data for the cleavage stage cell fragmentation degree rating model.

[0139] Based on the ratio of cell fragment area to blastomere area, cell images were divided into four different fragmentation levels: 0%-10%, 10%-20%, 20%-50%, and 50%-100%. The cell images were then divided into a training set, a validation set, and a test set in an 8:1:1 ratio. The training set contained 2700 images, the validation set contained 338 images, and the test set contained 337 images.

[0140] S42: Construct and train a grading model for the degree of cell fragmentation during cleavage;

[0141] A cell fragmentation rating model based on the ResNet50 network was constructed. Cross-entropy was used as the loss function for training the model, and label smoothing was introduced to enhance its generalization performance. The initial learning rate was set to 0.001, the batch size to 16, and the number of iterations to 200. During network optimization iterations, if the model's loss function did not change significantly after every 20 iterations, the learning rate decreased by 10%; if the model's loss function changed significantly, the learning rate remained unchanged. After training, the model can predict the degree of cell fragmentation in the cleavage stage. The processing flow of the cleavage stage cell fragmentation rating model is as follows: Figure 9 As shown.

[0142] S43: Using a trained cell fragmentation rating model, obtain cell fragmentation rating prediction results, such as... Figure 10 As shown.

[0143] S5: Establish a U-Net-based semantic segmentation model using the blastocyst stage semantic segmentation module, and train and optimize the semantic segmentation model.

[0144] Step S5 is as follows:

[0145] S51: Create a dataset;

[0146] Using Photoshop, the outlines of trophoblast cells and inner cell mass in blastocyst images were delineated with lines of different colors. The outlines were drawn pixel by pixel, binarized, and then erosion and dilation operations were applied to generate a fine mask. The images were divided into training, validation, and test sets in an approximately 8:1:1 ratio. The training set contained 3953 images, the validation set contained 564 images, and the test set contained 565 images.

[0147] S52: Construct and train a semantic segmentation model for the blastocyst stage;

[0148] A semantic segmentation model for the blastocyst stage based on the U-Net network was constructed. Using the S51 training set, the U-Net network was trained, and the model loss was calculated. Stochastic gradient descent was used to update the network weights, and this process was iterated until the model converged. The input image size of the U-Net network was 512×512. It mainly consists of an encoder network and a decoder network. The encoder network contains 4 layers of convolution and downsampling operations, and the decoder network contains 4 layers of transposed convolution and upsampling operations. Skip connections were introduced between the encoder and decoder modules, allowing them to share network weights for better segmentation results. After training, the model can accurately segment the trophoblast cells and inner cell mass regions in blastocyst images. The flowchart of the blastocyst stage semantic segmentation model is shown below. Figure 11 As shown.

[0149] S53: Using a trained semantic segmentation model for the blastocyst stage, the segmented trophoblast cells and inner cell mass regions are obtained, such as... Figure 12 As shown.

[0150] S6: Establish a quality rating model based on Inception V3 through the blastocyst stage quality rating module, and train and optimize the quality rating model.

[0151] Step S6 is as follows:

[0152] S61: Create a dataset;

[0153] Based on S5 segmentation of trophoblast cells and inner cell mass regions, and combined with annotations of trophoblast cells and inner cell images by clinical embryologists, the inner cell mass is graded as follows: Grade A, cell number greater than the preset value 'a', tightly packed; Grade B, cell number between the preset values ​​'a' and 'b', loosely packed, 'a' > 'b'; Grade C, cell number less than the preset value 'b'. Trophoblast cells are graded as follows: Grade X, epithelial cell number greater than the preset value 'c', dense structure; Grade Y, epithelial cell number between the preset values ​​'c' and 'd', loose structure, 'c' > 'd'; Grade Z, epithelial cell number less than the preset value 'd'. The two letters are arranged from highest to lowest as follows: AX, AY, BX, BY > AZ, BZ > CX, CY, CZ, thus creating a dataset for trophoblast cell and inner cell mass quality rating.

[0154] S62: Build and train a quality rating model;

[0155] A quality rating model based on Inception V3 was constructed, using the S61 dataset. The model was trained, and the predicted results were compared with the labeled values. The model loss was calculated, and stochastic gradient descent was used to update the network weights. This process was iterated until the model converged. Compared to a typical CNN network, this model reduces the number of parameters, expands the receptive field, and improves network performance. After training, the model can predict the quality grades of trophoblast cells and inner cell mass in the blastocyst stage.

[0156] S63: Using a trained quality rating model, obtain the rating results for the quality of the trophoblast and inner cell mass, such as... Figure 13 As shown.

[0157] S7: Based on the six models trained in steps S1 to S6, establish an intelligent embryo quality assessment model based on graph convolutional neural networks, and train and optimize the intelligent embryo quality assessment model to assess embryo quality.

[0158] Step S7 is as follows:

[0159] S71: Create a dataset;

[0160] Combining the prediction results of steps S1, S2, S3, S4, S5 and S6, all prediction results are used as feature vectors of the embryo image and normalized to form the dataset for the graph convolutional neural network.

[0161] S72: Construct and train an intelligent embryo quality assessment model, such as... Figure 14 As shown;

[0162] 1) Feature extraction: First, the n image features obtained from six steps (S1, S2, S3, S4, S5, and S6) at different stages of embryonic development are input into the vertex feature extraction network. The specific process is as follows: First, the features pass through a VGG-16 CNN network to obtain n 4096-dimensional vectors, and then through PCA (Principal Component Analysis) to obtain n 128-dimensional vectors. These n 128-dimensional vectors constitute the vertices of the graph structure.

[0163] 2) Construct a graph structure. Based on the relationships between the features of each vertex, establish an adjacency matrix of edges between vertices to form a topological structure represented by the graph.

[0164] 3) Train a deep graph convolutional neural network (DCNN). This DCNN consists of a series of concatenated graph convolutional modules and incorporates an attention module. The network output is a Bx1 dimensional vector (B represents the number of classification categories). The attention mechanism provides adaptive vertex selection by learning the "importance" of vertices. The Adam optimization algorithm is used to calculate the gradient of the network error for each weight parameter during backpropagation, and new weights are obtained through the parameter update process. The model weights are iteratively calculated until a predetermined small loss is reached, and the optimal prediction value is obtained.

[0165] 4) Prediction: For the features represented by the graph structure, a regression prediction model is constructed to give three levels of embryo quality, namely the best, medium and worst quality.

[0166] This embodiment implements intelligent quality assessment of embryos using timelapse imaging based on deep learning algorithms, employing the PyTorch 1.4 framework and Python 3.7. The experimental platform is Ubuntu 18.04, equipped with two Intel Xeon CPUs, one NVIDIA GTX 2080ti 11Gb GPU, and 256Gb of memory. The timelapse imaging system is Timelapse, capturing one embryo image every 5 minutes. After monitoring for 3 days from fertilization, all images of each embryo are sequentially integrated into a 750-800 frame embryo video.

[0167] This embodiment collected embryo data from the Reproductive Medicine Center of Xiangtan Central Hospital. After data cleaning, 18 embryos were labeled for use in all model training. Simultaneously, data augmentation was performed on the embryo data images using methods such as random mirroring, horizontal compression, and random cropping.

Claims

1. A deep learning-based intelligent embryo quality assessment system, characterized in that: It includes a pronuclear phase instance segmentation module, a pronuclear phase target tracking module, a cleavage phase target detection module, a cleavage phase cell fragmentation degree rating module, a blastocyst phase semantic segmentation module, a blastocyst phase quality rating module, and an embryo quality intelligent assessment module. The prokaryotic phase instance segmentation module is used to segment three different target regions: cell, cytoplasm, and prokaryote; the prokaryotic phase instance segmentation module is connected to the prokaryotic phase target tracking module. The pronuclear stage target tracking module is used to track the trajectories of female and male pronuclei among pronuclear targets. It further distinguishes between male and female pronuclei based on the tracking trajectories. The pronuclear stage target tracking module is connected to the embryo quality intelligent assessment module. The pronuclear stage instance segmentation module and the pronuclear stage target tracking module calculate the spatial morphological parameters of the embryonic pronuclear stage based on the segmentation results of four different target regions: cell, cytoplasm, female pronucleus, and male pronucleus. The cleavage phase target detection module is used to detect the number, location and uniformity of cells during cell division. The cleavage phase target detection module is connected to the cleavage phase cell fragmentation degree rating module. The cleavage-stage cell fragmentation rating module is used to rate the degree of cell fragmentation, and the cleavage-stage cell fragmentation rating module is connected to the embryo quality intelligent assessment module. The blastocyst semantic segmentation module is used to accurately segment the two different target regions, the trophoblast and the inner cell mass. The blastocyst semantic segmentation module is connected to the blastocyst quality rating module. The blastocyst quality rating module is used to rate the quality of the trophoblast and inner cell mass. The blastocyst quality rating module is connected to the embryo quality intelligent assessment module. The intelligent embryo quality assessment module constructs a feature vector of the embryo image using the aforementioned spatial morphological parameters, cell location, cell number, cell uniformity, cell fragmentation degree, trophoblast and inner cell mass quality, and trains an intelligent embryo quality assessment model based on graph convolutional neural network to predict the embryo quality assessment result.

2. A method for intelligent assessment of embryo quality based on the intelligent embryo quality assessment system of claim 1, characterized in that, Includes the following steps: S1: Establish a prokaryotic instance segmentation model based on the Mask R-CNN network through the prokaryotic instance segmentation module, and train and optimize the prokaryotic instance segmentation model; S2: Establish a prokaryotic target tracking model based on DeepSort through the prokaryotic target tracking module, and train and optimize the prokaryotic target tracking model; S3: Establish a cleavage stage cell detection model through the cleavage stage target detection module, and train and optimize the cleavage stage cell detection model; S4: Establish a cell fragmentation rating model for the cleavage stage using the cell fragmentation rating module, and train and optimize the cell fragmentation rating model for the cleavage stage. S5: Establish a U-Net-based semantic segmentation model using the blastocyst stage semantic segmentation module, and train and optimize the semantic segmentation model. S6: Establish a quality rating model based on Inception V3 through the blastocyst stage quality rating module, and train and optimize the quality rating model; S7: Based on the six models trained in steps S1 to S6, establish an intelligent embryo quality assessment model based on graph convolutional neural networks, and train and optimize the intelligent embryo quality assessment model to assess embryo quality.

3. The intelligent embryo quality assessment method according to claim 2, characterized in that, The specific steps of step S1 are as follows: S11: Create a dataset; First, using Photoshop software, the outlines of four targets—cells, cytoplasm, female pronucleus, and male pronucleus—are delineated on the pronuclear image of embryonic development using different colors, creating mask images for cells, cytoplasm, female pronucleus, and male pronucleus; then, COCO format annotation files are generated using the mask images of each target. Based on the original images and labeled files, we created the dataset required for the prokaryotic instance segmentation model with labeled images, and divided the dataset into training set, validation set and test set respectively. S12: Construct a prokaryotic phase instance segmentation model; A prokaryotic instance segmentation model based on Mask R-CNN network is constructed. This prokaryotic instance segmentation model uses ResNet101 network as the backbone network and Feature Pyramid Network (FPN) network as the neck to fuse features of different scales. ROIAlign is used instead of ROIPooling operation. The detection head includes three different branches: classification, regression, and masking, to predict classification probability, target location box, and segmentation mask. S13: Train and optimize the prokaryotic instance segmentation model; The original images of pronuclear stage embryos were used for model training and optimization; The output of the pronuclear phase instance segmentation model includes the following parts: the classification branch output is the classification probability of different targets, the regression branch output is the coordinate value of the predicted bounding box of different targets, and the mask branch output is the pixel-level mask of different targets. The output results are compared with the true labels, the loss of the pronuclear phase instance segmentation model is calculated, and the network weights are updated using stochastic gradient descent. The training process is iterated continuously to optimize the network. S14: Obtain the prokaryotic instance segmentation prediction results using the trained prokaryotic instance segmentation model.

4. The intelligent embryo quality assessment method according to claim 3, characterized in that, The specific steps of step S2 are as follows: S21: Create a dataset; Using the dataset conversion method provided on the MOT official website, the COCO format dataset already created in S11 was converted into a target tracking dataset in MOT format, where each frame image only includes two different targets: female pronucleus and male pronucleus. S22: Construct a target tracking model based on DeepSort, and use the target tracking dataset to train and optimize the target tracking model; The overall processing flow of the DeepSort-based target tracking model is as follows: First, a ResNet50-based appearance extraction model is trained. The input of this model is the detection box region, and the output is the feature vector of the embedding space. The loss function during the appearance model training process is triple loss, which is a loss directly constructed from the feature vectors, making the feature vectors of different targets more distinguishable and the feature vectors of the same targets more similar. Kalman filtering is used to obtain the motion states of the two types of targets. The similarity matrix is ​​calculated by combining the extracted target appearance and the target motion state. The Hungarian algorithm is used for cascaded matching. Kalman filtering is used to update the trajectory and motion states of the two types of targets, thereby improving the accuracy of target recognition. S23: Obtain the target tracking model prediction results through the trained target tracking model; S24: Calculate the spatial morphological parameters of the pronuclear stage; By combining the prokaryotic phase instance segmentation model and the target tracking model, regions of four different targets—cell, cytoplasm, male pronucleus, and female pronucleus—are obtained. Based on the area of ​​different regions, spatial morphological parameters of the prokaryotic phase are calculated, including the maximum cell contraction amplitude, the maximum proportion of the cytoplasmic halo, whether the male and female pronuclei are side by side, the distance from the side-by-side point of the male and female pronuclei to the cell center, the maximum diameter difference between the male and female pronuclei during the entire side-by-side period, and the distance from the initial position of the male pronucleus to the cell center.

5. The intelligent embryo quality assessment method according to claim 4, characterized in that, The specific steps of step S3 are as follows: S31: Create a dataset; The LabelImg software was used to label the cell locations in embryo images, and a dataset conforming to the VOC format was created. The dataset was divided into three parts: training set, test set, and validation set in a 7:2:1 ratio. S32: Construct and train a cleavage-stage cell detection model; First, data augmentation is performed on the images to construct a cell detection model based on YOLOv5. The cell detection model is trained and optimized using images from the cleavage stage to detect the number of cells during cell division, accurately predict the bounding box of each cell, and judge the uniformity of cells. The Adam algorithm is used to optimize the weight parameters of the convolutional neural network. S33: Determination of cell uniformity during cleavage; The trained cleavage-stage cell detection model is used to predict the number of cleavage-stage cells in the test image and the location of each cell. The area of ​​the location box is approximated to the area of ​​each cell, and the standard deviation of the area of ​​all cells is calculated to determine the uniformity of cells. If the standard deviation is less than the set threshold, the cells in the cleavage-stage embryo image are considered to be relatively uniform; otherwise, they are considered to be non-uniform.

6. The intelligent embryo quality assessment method according to claim 5, characterized in that, The specific steps of step S4 are as follows: S41: Create a dataset; Based on the cell location predicted by the cell target detection model in step S2, cells in the embryo image are extracted as ROI regions, and the ROI regions are used as training data for the cleavage stage cell fragmentation degree rating model. Based on the ratio of cell fragment area to blastomere area, cell images are divided into four different fragmentation levels: 0%-10%, 10%-20%, 20%-50%, and 50%-100%; and the cell images are divided into training set, validation set, and test set in an 8:1:1 ratio. S42: Construct and train a grading model for the degree of cell fragmentation during cleavage; A cell fragmentation rating model based on the ResNet50 network was constructed, and the model was trained using cross-entropy as the loss function, and a label smoothing function was introduced. S43: Obtain cell fragmentation rating prediction results through a trained cell fragmentation rating model.

7. The intelligent embryo quality assessment method according to claim 6, characterized in that, The specific steps of step S5 are as follows: S51: Create a dataset; Using Photoshop, the outlines of trophoblast cells and inner cell mass in blastocyst images were delineated with lines of different colors. The outlines were drawn pixel by pixel, binarized, and then erosion and dilation operations were used to generate a fine mask. The images were divided into training, validation, and test sets in an 8:1:1 ratio. S52: Construct and train a semantic segmentation model for the blastocyst stage; A blastocyst-stage semantic segmentation model based on the U-Net network was constructed. The U-Net network was trained using the S51 training set, and the model loss was calculated. The network weights were updated using stochastic gradient descent. This process was iterated until the model converged. S53: Using a trained semantic segmentation model for the blastocyst stage, the segmented trophoblast cells and inner cell mass regions are obtained.

8. The intelligent embryo quality assessment method according to claim 7, characterized in that, The specific steps of step S6 are as follows: S61: Create a dataset; Based on the S5 segmentation of trophoblast cells and inner cell mass regions, the images of trophoblast cells and inner cells are labeled. Inner cell mass is graded as follows: Grade A, cell number greater than a preset value 'a', densely packed; Grade B, cell number between preset values ​​'a' and 'b', loosely packed, 'a' > 'b'; Grade C, cell number less than the preset value 'b'. Trophoblast cells are graded as follows: Grade X, epithelial cell number greater than a preset value 'c', dense structure; Grade Y, epithelial cell number between preset values ​​'c' and 'd', loose structure, 'c' > 'd'; Grade Z, epithelial cell number less than the preset value 'd'. After arranging the two letters, the order from highest to lowest is: AX, AY, BX, BY > AZ, BZ > CX, CY, CZ. This creates a dataset for trophoblast cell and inner cell mass quality rating. S62: Build and train a quality rating model; A quality rating model based on Inception V3 is constructed. The S61 dataset is used to train the quality rating model. The prediction results are compared with the labels, the model loss is calculated, and the network weights are updated using stochastic gradient descent. This process is iterated until the model converges. S63: Obtain the quality rating results of the trophoblast and inner cell mass by using a trained quality rating model.

9. The intelligent embryo quality assessment method according to claim 8, characterized in that, The specific steps of step S7 are as follows: S71: Create a dataset; Combining the prediction results of steps S1, S2, S3, S4, S5 and S6, all prediction results are used as feature vectors of the embryo image and normalized to form the dataset for the graph convolutional neural network. S72: Construct and train an intelligent embryo quality assessment model; 1) Feature extraction: First, the n image features obtained from the six steps S1, S2, S3, S4, S5 and S6 at different stages of embryonic development are input into the vertex feature extraction network. The specific process is as follows: First, the features are processed through a VGG-16 CNN network to obtain n 4096-dimensional vectors. Then, principal component analysis (PCA) is used to obtain n 128-dimensional vectors. These n 128-dimensional vectors constitute the vertices of the graph structure. 2) Construct a graph structure. Based on the relationships between the features of each vertex, establish an adjacency matrix of edges between vertices to form a topological structure represented by the graph. 3) Train a deep graph convolutional neural network, which is composed of a series of graph convolutional modules connected in series and introduces an attention module. The output of the network is a Bx1 dimensional vector. Use the Adam optimization algorithm to calculate the gradient of the network error for each weight parameter in backpropagation, and obtain the new weights through the parameter update process. Iteratively calculate the model weights until the predetermined small loss is reached and the best prediction value is obtained. 4) Prediction: For the features represented by the graph structure, a regression prediction model is constructed to give three levels of embryo quality, namely the best, medium and worst quality.

Images