Embryo development quality evaluation method based on SMMC network

Abstract

The invention provides an SMMC network-based embryo development quality evaluation method. The method comprises the following steps: inputting embryo images in different development periods; distributing different weights to the embryo image according to the development period, and outputting an attention feature map with channel attention; extracting global features and local features of the attention feature map; fusing the global features and the local features and extracting time sequence features to obtain a spatial-temporal feature map; and inputting the spatial-temporal feature map into a dichotomy full-connection layer for classification, and evaluating the embryonic development quality. By extracting the features in various aspects, the method has more excellent performance in embryo evaluation.

Description

Technical Field

[0001] The present invention relates to the technical field of embryo detection, and in particular to an embryo development quality assessment method based on a SMMC network. Background Art

[0002] In the field of assisted reproductive technology, in vitro fertilization (IVF) is a common reproductive assistance method, and its success rate is affected by many factors. The assessment of embryo quality is an important part of IVF. Professional medical personnel need to perform morphological assessment of the embryo's development status based on the obtained embryo images. This assessment process is crucial for selecting the embryo with the highest probability of successful implantation. Its accuracy and efficiency directly affect the success rate of IVF.

[0003] At present, the traditional embryo quality method is still widely used in the field of IVF. The traditional embryo quality assessment method mainly relies on the subjective judgment of relevant professionals. It evaluates the embryo quality and developmental potential by observing the morphological characteristics of the embryo, such as the number of pronuclei, the size of blastomeres, fragments and vacuoles in the cleavage stage, etc. in the pronuclear stage of the embryo; in the blastocyst stage of the embryo, the quality of the blastocyst is assessed according to the comprehensive situation of the blastocyst development stage, inner cell mass and trophoblast cells. However, these traditional embryo quality assessment methods have many limitations, such as strong subjectivity, different experience and skill levels of professionals, resulting in limited consistency and accuracy of the assessment results and low efficiency. The manual observation and recording process is time-consuming and labor-intensive, which limits the speed and scale of the assessment; the most important thing is the insufficient use of embryo image data. The traditional assessment method fails to make full use of the rich image data in the embryonic development process, and misses the opportunity to deeply analyze and explore the characteristics of embryonic development.

[0004] With the development of deep learning technology, more and more studies have begun to explore its application in biomedical image analysis, including studies that use image processing and machine learning algorithms to automatically evaluate the development of fertilized eggs. Compared with traditional technologies, deep learning models have stronger learning ability, adaptability, and nonlinear modeling capabilities. They can automatically learn features from large amounts of data and reduce interference from human factors, thereby improving the accuracy and efficiency of image analysis. However, most of the existing deep learning models are designed for medical images of adults or children. When evaluating embryonic development, the accuracy of embryo evaluation is low because the characteristics such as the embryonic development stage are not taken into account. Summary of the invention

[0005] The present invention proposes an embryo development quality assessment method based on the SMMC network to solve the technical problem that the existing deep learning method has low embryo assessment accuracy due to failure to consider characteristics such as the embryo's developmental stage when facing embryo detection.

[0006] To solve the above technical problems, the present invention provides an embryo development quality assessment method based on an SMMC network, which is special in that: embryo images of different developmental stages are input; different weights are assigned to the embryo images according to the developmental stages, and an attention feature map with channel attention is output; global features and local features of the attention feature map are extracted; the global features and local features are fused and the timing features are extracted to obtain a spatiotemporal feature map; the spatiotemporal feature map is input into a binary fully connected layer for classification, and the embryo development quality is evaluated.

[0007] Preferably, a SENet network is used when assigning different weights to the embryo images according to developmental stages.

[0008] Preferably, the local features are extracted using a MSCNN network.

[0009] Preferably, the first layer of the MSCNN network uses three CNNs with different kernel sizes to capture different frequencies, and the convolution kernel sizes are 100×100, 50×50 and 20×20 respectively.

[0010] Preferably, the GELU activation function is used to maintain the nonlinearity of MSCNN.

[0011] Preferably, the global features are extracted using a MambaVision network.

[0012] Preferably, the MambaVision network includes four stages, the first two stages use residual convolution blocks for fast feature extraction; the third and fourth stages use MambaVision blocks and Transformer blocks at the same time, the third and fourth stages are both set to N layers, the first N / 2 use MambaVision blocks and multi-layer perception blocks, and the last N / 2 use Transformer and MLP blocks.

[0013] Preferably, a ConvLSTM network is used to extract the time series features.

[0014] Preferably, the classification results are trained using a cross entropy loss function.

[0015] Preferably, the expression of the cross entropy loss function is: ; In the formula, y is the true label, with a value of 0 or 1; p is the probability that the model predicts that the fertilized egg will develop well; is the calculated loss value.

[0016] The beneficial effects of the present invention include at least: the present invention assigns different weights to embryo images according to the developmental stages to take into account the features of different stages; since the embryo has different features at different stages during development, the scales, frequency information, etc. of these features are not the same, so it is necessary to extract local features, but the correlation between these local features cannot be effectively modeled, so the present invention extracts global features, then fuses the local features with the global features, and then extracts the time series features. Such time series features can also help evaluate the quality of embryo development. By extracting features from various aspects, the method of the present invention has more excellent performance in embryo evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a method flow of an embodiment of the present invention; Figure 2 A schematic diagram of a model structure of an embodiment of the present invention; Figure 3 A schematic diagram of a SENet module according to an embodiment of the present invention; Figure 4 Schematic diagram of the MSCNN module of an embodiment of the present invention; Figure 5 A schematic diagram of a MambaVision model according to an embodiment of the present invention; Figure 6 Schematic diagram of the MambaVision block structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work belong to the protection scope of the present invention.

[0019] In this embodiment, nearly 100,000 images of embryonic development stages were successfully collected from multiple reproductive centers, of which the first five days of each embryonic development were photographed, including embryonic development from the pronuclear stage to the blastocyst stage, totaling 500 images. For embryos with less than or more than 500 photos, the existing photos were copied, added or deleted on average until the number reached 500. Secondly, the embryonic images were sorted in order according to the embryonic development time, and standardized preprocessing operations were performed to ensure that the size of the embryonic images was consistent, which was convenient for subsequent input into the model.

[0020] like Figure 1As shown, an embodiment of the present invention provides an embryo development quality assessment method based on an SMMC network, which assigns different weights to embryo images according to the developmental stages, and outputs an attention feature map with channel attention; extracts global features and local features of the attention feature map; fuses the global features and local features and extracts the timing features to obtain a spatiotemporal feature map; inputs the spatiotemporal feature map into a binary classification fully connected layer for classification, and assesses the embryo development quality. Specifically, the embodiment of the present invention first constructs an SMMC network model, such as Figure 2 The SMMC network model shown mainly includes four modules. The channel attention mechanism module designed by the present invention first uses the channel attention SENet module to weight images of different periods, and inputs images of embryonic development. The module will assign different weights to 500 pictures of embryos. By observing the embryonic image, it is not difficult to find that the embryo has different characteristics at different periods in the development process, and the scale size, frequency information, etc. of these characteristics are not quite the same. The present invention constructs a spatial feature extraction module for this feature. Using MambaVision module and multi-core convolution MSCNN module, the global features and local features of the image are extracted in parallel, wherein MSCNN can set convolution kernels of different sizes, and obtain local features of the image from the angles of frequency height, scale size, etc., and the MambaVision network, through its unique structure, combines the advantages of state-space model and Transformer, strengthens the modeling ability of context information, thereby extracting global features. The feature information obtained by both is integrated to obtain complete embryonic spatial feature data. The embryonic development images are taken for several consecutive days and have good time series characteristics. There are connections between adjacent images. In view of this feature, the present invention uses convolutional LSTM (ConVLSTM) to construct a time feature extraction module. ConvLSTM can process different spatial positions of the input at each time step and retain time information, thereby greatly enhancing the extraction of time features. Through the previous three modules, the spatiotemporal feature map of the embryonic image is obtained, and these feature maps are passed through the embryonic development status classification module to output the quality of the embryonic development status. The four modules learn from each other to obtain more comprehensive feature information of embryonic images at different developmental stages, thereby reducing the risk of overfitting of the model network and improving the generalization ability of the model.

[0021] In addition, the labeling of embryo image data is a tedious task, which is very time-consuming and labor-intensive. Therefore, the present invention effectively reduces the need for labeled embryo images and professional participation through reasonable model network design, reduces the labeling cost of embryo images, and can further improve the generalization ability of the model. Finally, the network model is trained through cross entropy loss to realize the judgment of the quality of embryo development.

[0022] The following is the implementation process of this embodiment: 1. Data collection and preprocessing stage The embryo image collection used in the embodiment of the present invention comes from multiple reproductive medical centers, which have rich experience and advanced equipment in the field of assisted reproductive technology. The embryo images used in the experiment are captured in an incubator by time-lapse photography technology. This method can continuously record the development of the embryo without interfering with the normal development of the embryo. Each embryo is photographed four to five times per hour, and the total recording time is about five days, and the total number of photos taken is about 500. The single embryo cell development image data set comprehensively covers the various developmental periods from the embryo from the cell stage to the blastocyst stage. In order to judge the development of each embryo, the experiment invited five people in the field of reproductive medicine to participate in the assessment. By professionally evaluating and judging each embryo development process image, it is divided into two different categories based on the quality of the embryo. This process ensures the accuracy of the quality assessment of each embryo, is also representative in biological terms, and can truly reflect the development of the embryo. Through this rigorous embryo quality assessment and annotation process, the experiment constructs a high-quality, highly representative embryo image data set. Finally, the collected data set is divided into a training set, a validation set, and a test set according to a ratio of 3:1:1. The above process provides a solid foundation for the subsequent development and training of embryo development quality assessment models, and also provides a reliable standard for the evaluation and verification of the algorithm.

[0023] 2. Model building The model constructed in the embodiment of the present invention mainly includes five modules, namely SENet, MSCNN, MambaVision, ConvLSTM and a binary classification fully connected layer. The construction of these five modules and the loss function selected by the model are introduced one by one below.

[0024] 1) SENet module construction Since each embryo in the dataset has five days of development images, combined with the experience of traditional embryo quality assessment methods, images at different stages have different impacts on the judgment of embryo development, especially images close to the blastocyst stage are more important. Therefore, before subsequent processing, the channel attention SENet module is used to weight images at different stages. The image of embryo development is input, and the module assigns different weights to each of the 500 images of the embryo, and outputs a feature map with channel attention.

[0025] The core idea of ​​SENet is to adaptively recalibrate the feature response of channels by explicitly modeling the interdependence between feature channels. SENet mainly achieves this function by introducing the SE module (Squeeze-and-Excitation block).

[0026] The network structure of SENet is as follows Figure 3 As shown in the figure, the squeeze part is the compression part. The original input set of embryo images undergoes a convolution. The feature map dimensions obtained after the operation are H×W×C, where H is the height, W is the width, and C is the number of channels.

[0027] The Squeeze part compresses the feature map along the spatial dimension and converts each two-dimensional feature channel into a real number, that is, compresses the feature map of size H×W×C to 1×1×C. This is usually achieved through global average pooling to obtain the global receptive field.

[0028] In the Excitation part, after obtaining the 1×1×C representation of Squeeze, a FC fully connected layer is added to predict the importance of each channel, and then the weight value is limited to between 0 and 1 through the Sigmoid activation function. After obtaining the importance of different channels, it is applied (stimulated) to the corresponding channels of the previous feature map, and then subsequent operations are performed. Through the above operations, the weighting of different images can be completed.

[0029] 2) MSCNN module construction like Figure 4 As shown in Figure 2, by observing the embryo images, it is not difficult to find that the embryo has different characteristics at different stages of development, and the scale and frequency information of these characteristics are different, such as Figure 4 In (a), the pronuclear stage of the embryo is when the pronuclear pixel size is approximately 50×50; Figure 4 In (b), the size of the dividing cells in the cleavage stage of the embryo is about 100×100; Figure 4 In (c), the size of the outer trophoblast in the blastocyst stage is about 20 × 20. It is impossible to extract all these features completely with a single CNN network, so MSCNN, which can use different convolution kernel sizes, is selected to complete local feature extraction.

[0030] MSCNN extracts features of different scales and frequencies of input data by using convolution kernels of different sizes. In the first layer, three CNNs with different kernel sizes are used to capture different frequencies. The convolution kernel sizes are 100×100, 50×50, and 20×20, respectively. Smaller convolution kernels can capture high-frequency and small-scale detail information of images, such as edges and corners, while larger convolution kernels can capture low-frequency and large-scale information of images, such as shape and texture. In MSCNN, each convolution path consists of four convolution layers, two maximum pooling layers, and a batch normalization layer. In addition, the GELU activation function is used to maintain the nonlinearity of MSCNN. Each pooling layer is used to downsample the input. Finally, the features of different paths are fused by splicing. The fused features are used for subsequent classification tasks. Through the above steps, the MSCNN network can effectively handle targets of different scales and improve the accuracy and robustness of feature extraction.

[0031] 3) MambaVision module construction Although MSCNN solves the problem of extracting feature information of different scales and frequencies, it is unable to model the correlation of these local features. In the experience of previous embryo assessment, it is often not to focus on a certain feature information of the embryo for judgment, but to comprehensively evaluate all feature information. Therefore, it is very necessary to strengthen the modeling of the global features of the image. In the present invention, the cutting-edge MambaVision technology is selected to mine the global context information in the image.

[0032] MambaVision is a novel hybrid Mamba-Transformer backbone network designed for vision applications. The implementation of MambaVision first splits the input image into small patches and passes them through a series of CNN layers for feature extraction. In subsequent stages, it redesigns the Mamba formula to enhance its ability to effectively model visual features, and uses multiple self-attention blocks in the final stage, which significantly enhances the ability to capture global context and long-range spatial dependencies.

[0033] like Figure 5As shown in Figure 1, MambaVision has a hierarchical architecture consisting of four different stages. The first two stages consist of CNN-based layers for fast feature extraction at higher input resolutions, while the third and fourth stages include the proposed MambaVision and Transformer blocks. Specifically, given an image of size H×W×3, the input is first converted into overlapping blocks of size H / 4 × W / 4 × C and projected into a C-dimensional embedding space through a stem consisting of two consecutive 3 × 3 CNN layers with a stride of 2. The downsampler between stages consists of a batch-normalized 3 × 3 CNN layer with a stride of 2, which reduces the image resolution by half. The first two stages use residual convolution blocks for fast feature extraction. The third and fourth stages use both MambaVision and Transformer blocks. Given N layers in the third and fourth stages, this embodiment uses N / 2 MambaVision and multilayer perceptron (MLP) blocks, followed by additional N / 2 Transformer and MLP blocks. The Transformer blocks in the last N / 2 layers allow recovering the lost global context and capturing long-range spatial dependencies.

[0034] The design of the MambaVision block is as follows Figure 6 As shown, it is more suitable for visual tasks than the existing Mamba mixer. First, the causal convolution is replaced with a regular convolution, because the causal convolution limits the influence to one direction, which is unnecessary and restrictive for visual tasks. In addition, a symmetric branch without a state-space model (SSM) is added, consisting of additional convolutions and SiLU activations to compensate for anything lost due to the order constraint of the SSM. The outputs of the two branches are then concatenated and projected through a final linear layer. This combination ensures that the final feature representation combines both sequential and spatial information, taking full advantage of the advantages of both branches.

[0035] The focus of the Transformer block is the self-attention mechanism. In the MambaVision network, the multi-head self-attention mechanism is chosen. The multi-head self-attention mechanism uses a scaled dot product mechanism to associate elements at different positions of the input sequence and finally output the output sequence. It consists of H scaled dot product attention modules. First, the input is linearly transformed to obtain the query (Q), key (K) and value (V), and then sent to H scaled dot product attention modules for processing. Finally, the heads are connected and linearly projected to generate attention output. The above steps can be expressed as follows: (1) (2) (3) (4) (5) in is the input vector, are the query, value, and key of the map, d is the dimension of the key vector, is the jth attention head, is the learnable weight matrix, For output, d is the dimension of the key vector. The feedforward neural network in the encoder is responsible for nonlinear transformation and feature extraction of the multi-head attention output. It consists of two fully connected layers (FC) with an activation function (ReLU) between them.

[0036] Such a design enables richer feature representation, better generalization, and improved performance on computer vision tasks.

[0037] 4) ConvLSTM module construction In the present invention, 500 pictures of the development of an embryo were taken for a period of five days. These 500 pictures can be regarded as a continuous video information, which has good continuity in time, strong similarities in adjacent pictures, and strong differences between pictures far apart. This time series feature can also help us evaluate the quality of embryo development, so ConvLSTM network is used to extract this time series feature.

[0038] The structure of ConvLSTM combines CNN and LSTM. Specifically, ConvLSTM adds convolution operations to the internal operations of standard LSTM units. This means that each LSTM unit has its own convolution kernel. This allows ConvLSTM to process different spatial locations of the input at each time step and retain temporal information. The working principle of ConvLSTM is as follows: First, ConvLSTM processes the input data through convolution operations to capture the spatial features of the input data, which can be expressed as:

[0039] in Represents the input data. The processed data is then input into a standard LSTM cell. The LSTM cell consists of gates with different functions. The forget gate determines which information should be discarded from the memory cell. It is calculated by passing the input feature map of the current time step and the memory cell state of the previous time step to the forget convolution kernel, and then applying the sigmoid activation function. The input gate determines which new information should be stored in the memory cell. It consists of two parts: one is a sigmoid layer, which is used to determine which values ​​need to be updated; the other is a tanh layer, which is used to calculate new candidate memory cell values. The output gate determines which information in the memory cell should be output. It is calculated by passing the input feature map of the current time step and the memory cell state to the output convolution kernel, and then applying the sigmoid activation function. Finally, according to the output and memory cell Calculate the new hidden state . These steps can be expressed as follows: (7) (8) (9) (10) (11) (12) in , and are the outputs of the forget gate, input gate, and output gate respectively; σ and tanh represent the sigmoid activation function and the hyperbolic tangent function respectively; , , and Represents a trainable weight matrix. Indicates deviation. Represents the previous hidden state and relationship, Represents a new candidate cell state.

[0040] Through the ConvLSTM network built above, the time series features in the embryonic development images can be obtained, helping the model to better complete subsequent classification tasks.

[0041] 5) Construction of binary classification fully connected layer The binary classification fully connected layer is the output part of the model. After going through the previous four modules, the model extracts the comprehensive spatiotemporal features of the embryo image. Finally, the model classifies the extracted features through a binary classification fully connected layer to judge the quality of embryonic development.

[0042] 6) Choice of loss function In the embodiment of the present invention, in order to evaluate the development of fertilized eggs, the model needs to classify the development results into two categories, that is, good or bad. Therefore, the cross entropy loss function is selected as the optimization target. For the binary classification problem, the cross entropy loss function is defined as: (13) in, y is the true label, with a value of 0 or 1; p is the probability that the model predicts that the fertilized egg will develop as “good”; is the calculated loss value.

[0043] The cross entropy loss function guides the model to learn how to accurately predict the development of fertilized eggs by minimizing the difference between the predicted probability distribution and the true distribution. During the training process, the parameters of the model are adjusted through the back propagation algorithm and the gradient descent method to reduce the value of the loss function and thus obtain the optimal parameters of the model.

[0044] The present invention is an embryo development assessment method based on deep learning. Through the cooperation and joint training of multiple module networks, it aims to achieve better embryo development quality assessment. In the experiment, the Adam optimizer was used for parameter optimization, with a learning rate of 0.005, a batch size of 16, and 200 training rounds. The Adam optimizer combines weight decay and L2 regularization technology to effectively solve problems such as slow network convergence and parameter overfitting. Specifically, the present invention uses the Adam optimizer to optimize all parameters involved in the encoder and decoder to minimize the total loss function L , thereby optimizing all network structure parameters. Through this optimization method, an optimal network model can be obtained for the task of embryo development quality assessment. The model can accurately judge the quality of embryo development through embryo development images, thereby providing more accurate data for subsequent analysis and research.

[0045] In order to verify the effectiveness of the proposed method, the present invention conducted a large number of parameter tuning experiments and comparative experiments of different methods. First, a model ablation experiment was conducted. After removing a certain module, the accuracy of embryo quality assessment was tested. The results are shown in Table 1.

[0046] Table 1: Ablation experiment results

[0047] As shown in Table 1, when the SENet module is deleted, the accuracy of embryo quality assessment is 85.3%; when the MSCNN module is deleted, the accuracy of embryo quality assessment is 84.1%; when the Mambavision module is deleted, the accuracy of embryo quality assessment is 84.6%; when the ConvLSTM module is deleted, the accuracy of embryo quality assessment is; and the accuracy of embryo quality assessment of the model used in the present invention is 89.7%. This shows that the four different modules can cooperate with each other to achieve the best effect.

[0048] Table 2 shows the accuracy of different models in embryo quality assessment. From the experimental results in Table 2, it is found that ResNet can achieve the best accuracy of 82.5%, which is 7.2% lower than the method proposed in this invention; the best recognition accuracy of this invention is 6.3% higher than the result of the XceptionNet model in the embryo image automatic focusing task, and 5.1% higher than the DenseNet model. This shows that the method proposed in this invention has a higher recognition effect than the currently popular methods.

[0049] Table 2: Recognition accuracy of different models on embryo images

[0050] In summary, the present invention makes full use of the relevant information contained in the embryo image, adopts a combined network model, and extracts various features to study the task of embryo development quality assessment. A large number of experiments show that the method proposed in the present invention has more excellent performance.

[0051] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. Only the preferred embodiments of the present invention are expressed. The description is more specific and detailed, but it cannot be understood as limiting the scope of the present invention. As long as there is no contradiction in the combination of these technical features, they should be considered as within the scope of this specification.

[0052] It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these modifications and improvements all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the attached claims.

Claims

1. A method for evaluating embryo development quality based on SMMC network, characterized in that: Inputting embryo images at different developmental stages; assigning different weights to the embryo images according to the developmental stages, and outputting an attention feature map with channel attention; extracting global features and local features of the attention feature map; The global features and local features are fused and the time series features are extracted to obtain a spatiotemporal feature map; the spatiotemporal feature map is input into a binary classification fully connected layer for classification, and the quality of embryo development is evaluated.

2. A method for evaluating embryo development quality based on SMMC network according to claim 1, characterized in that: The SENet network is used to assign different weights to the embryo images according to the developmental stages.

3. A method for evaluating embryo development quality based on SMMC network according to claim 1, characterized in that: The local features are extracted using the MSCNN network.

4. A method for evaluating embryo development quality based on SMMC network according to claim 3, characterized in that: The first layer of the MSCNN network uses three CNNs with different kernel sizes to capture different frequencies, and the convolution kernel sizes are 100×100, 50×50, and 20×20 respectively.

5. A method for evaluating embryo development quality based on SMMC network according to claim 4, characterized in that: The GELU activation function is used to maintain the nonlinearity of MSCNN.

6. A method for evaluating embryo development quality based on SMMC network according to claim 1, characterized in that: The MambaVision network is used to extract the global features.

7. A method for evaluating embryo development quality based on SMMC network according to claim 6, characterized in that: The MambaVision network includes four stages. The first two stages use residual convolution blocks for fast feature extraction. The third and fourth stages use MambaVision blocks and Transformer blocks at the same time. The third and fourth stages are both set to N layers. The first N / 2 use MambaVision blocks and multi-layer perception blocks, and the last N / 2 use Transformer and MLP blocks.

8. A method for evaluating embryo development quality based on SMMC network according to claim 7, characterized in that: The ConvLSTM network is used to extract the time series features.

9. The method for evaluating embryo development quality based on SMMC network according to claim 1, characterized in that: The classification results are trained using the cross entropy loss function.

10. The method for evaluating embryo development quality based on SMMC network according to claim 9, characterized in that: The expression of the cross entropy loss function is: ； In the formula, y is the true label, with a value of 0 or 1; p is the probability that the model predicts that the fertilized egg will develop well; is the calculated loss value.

Images