Embryo Development Quality Assessment Method Based on SMMC Network

Abstract

This invention provides a method for assessing embryonic development quality based on a Smart Memory Network (SMMC). The method involves inputting embryonic images at different developmental stages; assigning different weights to the embryonic images according to the developmental stage, and outputting an attention feature map with channel attention; extracting global and local features from the attention feature map; fusing the global and local features and extracting temporal features to obtain a spatiotemporal feature map; and inputting the spatiotemporal feature map into a binary classification fully connected layer for classification to assess embryonic development quality. By extracting features from various aspects, the method of this invention exhibits superior performance in embryo assessment.

Description

Technical Field

[0001] This invention relates to the field of embryo detection technology, specifically to a method for assessing embryo development quality based on SMMC networks. Background Technology

[0002] In the field of assisted reproductive technology, in-vitro fertilization (IVF) is a common assisted reproductive method, and its success rate is affected by a variety of factors. Assessing embryo quality is a crucial part of IVF. Professional medical personnel need to perform morphological assessments of the embryos' development based on obtained images. This assessment process is essential for selecting embryos with the highest probability of successful implantation, and its accuracy and efficiency directly affect the success rate of IVF.

[0003] Currently, traditional embryo quality assessment methods are still widely used in the IVF field. These methods rely heavily on the subjective judgment of professionals, observing morphological characteristics such as the number of pronuclei during the pronuclear stage, blastomere size, fragmentation, and vacuoles during the cleavage stage to evaluate embryo quality and developmental potential. During the blastocyst stage, quality is assessed based on the overall developmental stage, inner cell mass, and trophoblast cells. However, these traditional methods have many limitations. They are highly subjective, and varying levels of experience and skill among professionals limit the consistency and accuracy of assessment results, leading to inefficiency. Manual observation and recording are time-consuming and labor-intensive, limiting the speed and scale of assessments. Most importantly, they underutilize embryonic image data, failing to fully utilize the rich image data generated during embryonic development and missing opportunities for in-depth analysis and exploration of developmental characteristics.

[0004] With the development of deep learning technology, more and more research is exploring its application in biomedical image analysis, including studies that utilize image processing and machine learning algorithms to automatically assess the development of fertilized eggs. Deep learning models, compared to traditional techniques, possess stronger learning capabilities, adaptability, and nonlinear modeling abilities. They can automatically learn features from large amounts of data and reduce interference from human factors, thereby improving the accuracy and efficiency of image analysis. However, most existing deep learning models are designed for medical images of adults or children. When assessing embryonic development, the lack of consideration for developmental stages and other characteristics leads to lower accuracy in embryo assessment. Summary of the Invention

[0005] This invention proposes an embryo development quality assessment method based on SMMC network to solve the technical problem that existing deep learning methods have low accuracy in embryo assessment because they do not consider the characteristics of embryo development stages.

[0006] To address the aforementioned technical problems, this invention provides a method for assessing embryonic development quality based on SMMC networks. The method is characterized by: inputting embryonic images at different developmental stages; assigning different weights to the embryonic images according to the developmental stage, and outputting an attention feature map with channel attention; extracting global and local features from the attention feature map; fusing the global and local features and extracting temporal features to obtain a spatiotemporal feature map; and inputting the spatiotemporal feature map into a binary fully connected layer for classification to assess embryonic development quality.

[0007] Preferably, the SENet network is used when assigning different weights to the embryo images according to their developmental stage.

[0008] Preferably, the local features are extracted using an MSCNN network.

[0009] Preferably, the first layer of the MSCNN network uses three CNNs with different kernel sizes to capture different frequencies, with kernel sizes of 100×100, 50×50 and 20×20 respectively.

[0010] Preferably, the GELU activation function is used to maintain the nonlinearity of MSCNN.

[0011] Preferably, the global features are extracted using a MambaVision network.

[0012] Preferably, the MambaVision network includes four stages. The first two stages use residual convolutional blocks for fast feature extraction. The third and fourth stages use both MambaVision blocks and Transformer blocks. Both the third and fourth stages are set to N layers. The first N / 2 layers use MambaVision blocks and multilayer perceptron blocks, and the last N / 2 layers use Transformer and MLP blocks.

[0013] Preferably, the temporal features are extracted using a ConvLSTM network.

[0014] Preferably, the classification results are trained using the cross-entropy loss function.

[0015] Preferably, the expression for the cross-entropy loss function is:

[0016] ;

[0017] In the formula, y This is a real label, with a value of 0 or 1; p It is the model's prediction of the probability that the fertilized egg will develop well; It is the calculated loss value.

[0018] The beneficial effects of this invention include at least the following: This invention assigns different weights to embryo images at different developmental stages to consider the characteristics of different stages; since embryos have different characteristics at different stages of development, and the scale, frequency information, etc. of these features are not the same, it is necessary to extract local features. However, the correlation between these local features cannot be effectively modeled. Therefore, this invention also extracts global features. After fusing local and global features, temporal features are extracted. These temporal features can also help to evaluate the quality of embryo development. By extracting features from various aspects, the method of this invention has better performance in embryo evaluation. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention;

[0020] Figure 2 This is a schematic diagram of the model structure according to an embodiment of the present invention;

[0021] Figure 3 This is a schematic diagram of the SENet module according to an embodiment of the present invention;

[0022] Figure 4 This is a schematic diagram of the MSCNN module according to an embodiment of the present invention;

[0023] Figure 5 This is a schematic diagram of the MambaVision model according to an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of the MambaVision block structure according to an embodiment of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0026] In this embodiment, nearly 100,000 images of embryonic development stages were successfully collected from multiple reproductive centers. Images were taken during the first five days of development for each embryo, encompassing the pronuclear stage to the blastocyst stage, totaling 500 images. For embryos with fewer or more than 500 images, existing images were copied, added, or deleted until the total number reached 500. The embryonic images were then sorted according to their developmental timeline and underwent standardized preprocessing to ensure consistent image size for easy input into the model.

[0027] like Figure 1 As shown, this embodiment of the invention provides a method for assessing embryonic development quality based on an SMMC network. Different weights are assigned to embryonic images according to developmental stages, and an attention feature map with channel attention is output. Global and local features of the attention feature map are extracted. The global and local features are fused and temporal features are extracted to obtain a spatiotemporal feature map. The spatiotemporal feature map is then input into a binary fully connected layer for classification to assess embryonic development quality.

[0028] Specifically, in this embodiment of the invention, an SMMC network model is first constructed, such as... Figure 2 The SMMC network model shown mainly consists of four modules. The channel attention mechanism module designed in this invention first uses the channel attention SENet module to weight images at different stages. Given images of embryonic development, this module assigns different weights to 500 images of the embryo. Observing the embryonic images, it is easy to see that the embryo has different characteristics at different stages of development, with varying scales and frequencies. This invention constructs a spatial feature extraction module to address this characteristic. Using the MambaVision module and the multi-kernel convolutional MSCNN module, global and local features of the image are extracted in parallel. MSCNN can be configured with convolutional kernels of different sizes to obtain local image features from the perspectives of frequency and scale. The MambaVision network, through its unique structure, combines the advantages of state-space models and Transformers to enhance its ability to model contextual information, thereby extracting global features. The feature information obtained from both methods is integrated to obtain complete spatial feature data of the embryo. Embryonic development images are captured over multiple consecutive days, inherently possessing strong temporal series characteristics. Corresponding relationships exist between adjacent images. This invention leverages this characteristic by constructing a temporal feature extraction module using Convolutional LSTM (ConvLSTM). ConvLSTM can process different spatial locations of the input at each time step while preserving temporal information, thus significantly enhancing temporal feature extraction. Through the previous three modules, spatiotemporal feature maps of the embryonic images are obtained. These feature maps are then passed through an embryonic development status classification module, outputting a classification of the embryonic development status. The four modules learn from each other to obtain more comprehensive feature information for embryonic images at different developmental stages, thereby reducing the risk of overfitting in the model network and improving the model's generalization ability.

[0029] Furthermore, labeling embryonic image data is a tedious and time-consuming task. Therefore, this invention, through a reasonable model network design, effectively reduces the need for labeled embryonic images and professional personnel, lowers the cost of embryonic image labeling, and further improves the model's generalization ability. Finally, by training the network model using cross-entropy loss, the determination of the embryonic development status is achieved.

[0030] The following is the implementation process of this embodiment:

[0031] 1. Data collection and preprocessing stage

[0032] The embryo image dataset used in this invention comes from multiple reproductive medicine centers, which possess extensive experience and advanced equipment in the field of assisted reproductive technology. The embryo images used in the experiment were captured in an incubator using time-lapse photography, a method that allows for continuous recording of embryonic development without interfering with normal embryonic development. Each embryo was photographed four to five times per hour, with a total recording time of approximately five days and a total of approximately 500 images. The single embryonic cell development image dataset comprehensively covers all developmental stages from the cellular stage to the blastocyst stage. To assess the development of each embryo, five professionals in the field of reproductive medicine were invited to participate in the evaluation. Through professional assessment and judgment of each embryonic developmental process image, embryos were categorized into two distinct categories based on their quality. This process ensures the accuracy of the quality assessment for each embryo, is biologically representative, and accurately reflects the embryonic developmental status. Through this rigorous embryo quality assessment and annotation process, a high-quality, highly representative embryo image dataset was constructed. Finally, the collected dataset was divided into a training set, a validation set, and a test set in a 3:1:1 ratio. The above process provides a solid foundation for the subsequent development and training of embryo development quality assessment models, and also provides a reliable standard for the evaluation and verification of algorithms.

[0033] 2. Model Building

[0034] The model built in this embodiment of the invention mainly consists of five modules: SENet, MSCNN, MambaVision, ConvLSTM, and a binary classification fully connected layer. The construction of these five modules and the loss function selected by the model are described below.

[0035] 1) SENet module setup

[0036] Since each embryo in the dataset has five days of developmental images, and based on experience with traditional embryo quality assessment methods, the impact of images at different stages on judging the quality of embryonic development varies, especially images closer to the blastocyst stage. Therefore, before further processing, the SENet channel attention module is used to weight the images at different stages. Inputting images of embryonic development, this module assigns different weights to 500 images of each embryo, outputting feature maps with channel attention.

[0037] The core idea of ​​SENet is to adaptively recalibrate the feature responses of channels by explicitly modeling the interdependencies between them. SENet primarily achieves this by introducing the Squeeze-and-Excitation block (SE).

[0038] The network structure of SENet is as follows: Figure 3 As shown. The Squeeze part, or compression part, is where the original input set of embryo images is processed by one convolution. The resulting feature map has dimensions H×W×C, where H is the height, W is the width, and C is the number of channels.

[0039] In the Squeeze step, the feature map is compressed along the spatial dimension, turning each two-dimensional feature channel into a real number. That is, the feature map of size H×W×C is compressed into 1×1×C, which is usually achieved by global average pooling, thereby obtaining the global receptive field.

[0040] In the excitation part, after obtaining the 1×1×C representation of the Squeeze, a fully connected (FC) layer is added to predict the importance of each channel. Then, a sigmoid activation function is used to constrain the weight values ​​between 0 and 1, obtaining the importance of different channels, which are then applied (activated) to the corresponding channels of the previous feature map before subsequent operations. Through the above operations, the weighting of different images can be completed.

[0041] 2) MSCNN module setup

[0042] like Figure 4 As shown, observing embryonic images reveals that embryos exhibit different characteristics at different stages of development. These characteristics vary in scale, frequency, and other information. Figure 4 In (a) of the embryo, the pronuclear stage is approximately 50×50 pixels in size. Figure 4 In (b) of the embryo, the dividing cells during the cleavage stage are approximately 100 × 100 mm in size; Figure 4In (c), the outer trophoblast cells of the blastocyst stage are approximately 20×20. To extract all these features completely, a single CNN network is insufficient; therefore, MSCNN, which can use different kernel sizes, was chosen to perform local feature extraction.

[0043] MSCNN extracts features of different scales and frequencies from the input data by using convolutional kernels of varying sizes. In the first layer, three CNNs with different kernel sizes (100×100, 50×50, and 20×20) are used to capture different frequencies. Smaller kernels capture high-frequency and small-scale details such as edges and corners, while larger kernels capture low-frequency and large-scale information such as shape and texture. Each convolutional path in MSCNN consists of four convolutional layers, two max-pooling layers, and one batch normalization layer. Furthermore, the GELU activation function is used to maintain the non-linearity of MSCNN. Each pooling layer downsamples the input. Finally, features from different paths are fused using a concatenation method. The fused features are then used for subsequent classification tasks. Through these steps, the MSCNN network can effectively handle targets of different scales, improving the accuracy and robustness of feature extraction.

[0044] 3) Setting up the MambaVision module

[0045] While MSCNN effectively addresses the problem of extracting feature information at different scales and frequencies, it fails to model the correlations between these local features. Past experience in embryo assessment often involves not just focusing on a single feature but comprehensively evaluating all features. Therefore, strengthening the modeling of global image features is essential. This invention utilizes the cutting-edge MambaVision technique to mine global contextual information within images.

[0046] MambaVision is a novel hybrid Mamba-Transformer backbone network designed specifically for vision applications. The implementation of MambaVision begins by segmenting the input image into small patches and extracting features through a series of CNN layers. In subsequent stages, it redesigns the Mamba formula to enhance its ability to effectively model visual features, employing multiple self-attention blocks in the final stage, which significantly improves its ability to capture global context and long-range spatial dependencies.

[0047] like Figure 5As shown, MambaVision has a hierarchical architecture consisting of four distinct stages. The first two stages consist of CNN-based layers for fast feature extraction at higher input resolutions, while the third and fourth stages include proposed MambaVision and Transformer blocks. Specifically, given an image of size H×W×3, the input is first transformed into an overlapping block of size H / 4 × W / 4 × C, and projected into a C-dimensional embedding space by a stem consisting of two consecutive 3 × 3 CNN layers with a stride of 2. The downsampling between stages consists of batch-normalized 3 × 3 CNN layers with a stride of 2, reducing the image resolution by half. The first two stages use residual convolutional blocks for fast feature extraction. The third and fourth stages use both MambaVision and Transformer blocks. Given N layers in the third and fourth stages, this embodiment uses N / 2 MambaVision and Multilayer Perceptron (MLP) blocks, followed by an additional N / 2 Transformer and MLP blocks. The Transformer block in the last N / 2 layers allows for the recovery of lost global context and the capture of remote spatial dependencies.

[0048] The design of the MambaVision block is as follows: Figure 6 As shown, compared to existing Mamba mixers, it is better suited for vision tasks. First, causal convolutions are replaced with regular convolutions because causal convolutions restrict the influence to one direction, which is unnecessary and restrictive for vision tasks. Furthermore, a symmetric branch without a state-space model (SSM) is added, consisting of additional convolutions and SiLU activations, to compensate for any content lost due to the order constraints of the SSM. The outputs of the two branches are then concatenated and projected through a final linear layer. This combination ensures that the final feature representation incorporates both order and spatial information, thus fully utilizing the advantages of both branches.

[0049] The focus of the Transformer block is the self-attention mechanism. In the MambaVision network, a multi-head self-attention mechanism is chosen. The multi-head self-attention mechanism uses a scaled dot product mechanism to associate elements at different positions in the input sequence, ultimately outputting the output sequence. It consists of H scaled dot product attention modules. First, the input undergoes a linear transformation to obtain the query (Q), key (K), and value (V). Then, these are fed into the H scaled dot product attention modules for processing. Finally, the heads are concatenated and linearly projected to produce the attention output. The above steps can be expressed as follows:

[0050] (1)

[0051] (2)

[0052] (3)

[0053] (4)

[0054] (5)

[0055] in For the input vector, It is a mapping of queries, values, and keys. d It is the dimension of the key vector. It is the j-th attention head. The weight matrix is ​​a learnable matrix. For output, d This is the dimension of the key vector. The feedforward neural network in the encoder is responsible for performing nonlinear transformations and feature extraction on the multi-head attention output. It consists of two fully connected layers (FC) and an activation function (ReLU) between the two fully connected layers.

[0056] This design enables richer feature representations, better generalization, and improved performance for computer vision tasks.

[0057] 4) ConvLSTM module setup

[0058] In this invention, 500 images were taken of an embryo over five days to document its development. These 500 images can be considered a continuous video sequence, exhibiting strong temporal continuity. Adjacent images show strong similarities, while images spaced far apart show significant differences. These temporal features can effectively help assess the quality of embryonic development; therefore, a ConvLSTM network was chosen to extract these temporal features.

[0059] ConvLSTM combines the architecture of CNN and LSTM. Specifically, it adds convolution operations to the internal operations of a standard LSTM unit. This means that each LSTM unit has its own convolutional kernel. This allows ConvLSTM to process different spatial locations of the input at each time step while preserving temporal information. The working principle of ConvLSTM is as follows: First, ConvLSTM processes the input data through convolution operations to capture the spatial features of the input data, which can be represented as:

[0060]

[0061] in The input data is then fed into a standard LSTM unit. The LSTM unit consists of gates with different functions. The forget gate determines which information should be discarded from the memory cells. It is computed by passing the input feature map of the current time step and the memory cell state of the previous time step to the forget convolution kernel and then applying the sigmoid activation function. The input gate determines which new information should be stored in the memory cells. It consists of two parts: a sigmoid layer, which determines which values ​​need to be updated; and a tanh layer, which computes new candidate memory cell values. The output gate determines which information from the memory cells should be output. It is computed by passing the input feature map of the current time step and the memory cell state to the output convolution kernel and then applying the sigmoid activation function. Finally, based on the output and memory cells... Calculate the new hidden state These steps can be represented as follows:

[0062] (7)

[0063] (8)

[0064] (9)

[0065] (10)

[0066] (11)

[0067] (12)

[0068] in , and These are the outputs of the forget gate, input gate, and output gate, respectively; σ and tanh represent the sigmoid activation function and the hyperbolic tangent function, respectively. , , and This represents a trainable weight matrix. Indicates deviation. Indicates the previous hidden state and Relationship This indicates the new candidate cell state.

[0069] The ConvLSTM network constructed above can be used to obtain time-series features in embryonic development images, helping the model to better complete subsequent classification tasks.

[0070] 5) Building a binary fully connected layer

[0071] The binary classification fully connected layer is the output part of the model. After going through the previous four modules, the model extracts the spatiotemporal features of the embryo image. Finally, the model uses a binary classification fully connected layer to classify the extracted features and determine the quality of embryo development.

[0072] 6) Choice of loss function

[0073] In this embodiment of the invention, to evaluate the development of the fertilized egg, the model needs to perform binary classification on the developmental outcome, distinguishing between good and bad. Therefore, the cross-entropy loss function is chosen as the optimization objective. For a binary classification problem, the cross-entropy loss function is defined as:

[0074] (13)

[0075] in, y This is a real label, with a value of 0 or 1; p It is the probability that the fertilized egg will develop into a "good" embryo, as predicted by the model. It is the calculated loss value.

[0076] The cross-entropy loss function guides the model to learn how to accurately predict the development of the fertilized egg by minimizing the difference between the predicted probability distribution and the true distribution. During training, the model's parameters are adjusted using backpropagation and gradient descent to reduce the value of the loss function, thereby obtaining the optimal parameters for the model.

[0077] This invention is a deep learning-based method for assessing embryonic development. By employing multiple collaborative network modules through joint training, it aims to achieve better assessment of embryonic development quality. In the experiments, the Adam optimizer was used for parameter optimization, with a learning rate of 0.005, a batch size of 16, and 200 training epochs. The Adam optimizer combines weight decay and L2 regularization techniques, effectively addressing issues such as slow network convergence and parameter overfitting. Specifically, this invention optimizes all parameters involved in the encoder and decoder using the Adam optimizer to minimize the total loss function. L This optimization method optimizes all network structure parameters, resulting in a superior network model for assessing embryonic development quality. This model can accurately determine the quality of embryonic development based on developmental images, providing more accurate data for subsequent analysis and research.

[0078] To verify the effectiveness of the proposed method, this invention conducted numerous parameter optimization experiments and comparative experiments with different methods. First, an ablation experiment was performed on the model; after removing a specific module, its accuracy in assessing embryo quality was tested, and the results are shown in Table 1.

[0079] Table 1: Ablation Experiment Results

[0080]

[0081] As shown in Table 1, the accuracy of embryo quality assessment is 85.3% after removing the SENet module; 84.1% after removing the MSCNN module; 84.6% after removing the Mambavision module; and [missing information] after removing the ConvLSTM module. The model used in this invention achieves an accuracy of 89.7% for embryo quality assessment. This demonstrates that the four different modules can collaborate effectively to achieve optimal results.

[0082] Table 2 shows the accuracy of different models in assessing embryo quality. The experimental results in Table 2 show that ResNet achieves the best accuracy of 82.5%, which is 7.2% lower than the method proposed in this invention. The best recognition accuracy of this invention is 6.3% higher than the XceptionNet model in the embryo image autofocus task and 5.1% higher than the DenseNet model. This indicates that the method proposed in this invention has a higher recognition performance among currently popular methods.

[0083] Table 2: Recognition accuracy of different models on embryo images

[0084]

[0085] In summary, this invention fully utilizes the relevant information contained in embryo images, employs a combined network model, and extracts various features to study the task of assessing embryonic development quality. Extensive experiments demonstrate that the method proposed in this invention has superior performance.

[0086] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0087] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for evaluating the quality of embryo development based on an SMMC network, characterized in that: Input embryo images at different development stages; assign different weights to the embryo images according to the development stages, output attention feature maps with channel attention; extract global features and local features of the attention feature maps; Fuse the global features and local features and extract time sequence features to obtain a spatio-temporal feature map; input the spatio-temporal feature map into a binary classification fully connected layer for classification to evaluate the embryo development quality; The SENet network is used when assigning different weights to the embryo images according to the development stages; The MSCNN network is used to extract the local features; The first layer of the MSCNN network uses three CNNs with different kernel sizes to capture different frequencies, and the convolution kernel sizes are 100*100, 50*50 and 20*20 respectively; the 100*100 convolution kernel is used to extract the features of the dividing cells in the embryo cleavage stage; the 50*50 convolution kernel is used to extract the features of the pronucleus in the embryo pronucleus stage; and the 20*20 convolution kernel is used to extract the features of the outer trophoblast cells in the blastocyst stage; The ConvLSTM network is used to extract the time sequence features; The MambaVision network is used to extract the global features; The MambaVision network includes four stages, the first two stages use residual convolution blocks for fast feature extraction; the third and fourth stages simultaneously use MambaVision blocks and Transformer blocks, and the third and fourth stages are both set to N layers, the first N / 2 use MambaVision blocks and multi-layer perception blocks, and the last N / 2 use Transformer and MLP blocks.

2. The method of claim 1, wherein the SMMC network is a network of single molecule magnets. The GELU activation function is used to maintain the nonlinearity of the MSCNN. 3.The method of claim 1, wherein the method further comprises: determining the quality of the embryo development based on the SMMC network. The cross-entropy loss function is used to train the classification results.

4. The method according to claim 3, wherein the method is characterized by: The expression of the cross-entropy loss function is: ； In the formula, y is a true label, taking a value of 0 or 1; p is a probability that the model predicts the zygote to develop well; is a loss value calculated.

Images