Freezing embryo transplantation outcome prediction method based on multi-modal data

Abstract

The invention discloses a frozen embryo transplantation outcome prediction method based on multi-modal data, belongs to the field of medical artificial intelligence, and is suitable for single embryo and double embryo transplantation. The method comprises the following steps: firstly, acquiring two high-resolution embryo images and corresponding clinical data of a patient, and copying the images during single embryo transplantation; the image resolution is unified by fixing the aspect ratio of an original image and properly cutting and filling. A standard picture enhancement technology is adopted for model training, and adaptability to variable imaging conditions is improved. During prediction, two preprocessed images share weights in a network with ResNet-50 as a trunk, and feature extraction is enhanced by combining a CBAM attention mechanism and a feature pyramid network. And the extracted features are averagely pooled and then directly connected with the output of a full-connection network for processing clinical data of the patient. And finally, through a two-layer full-connection network and a Sigmoid activation function, obtaining a prediction result of embryo transplantation by using binary threshold mapping.

Description

Technical Field

[0001] The present invention relates to the field of medical artificial intelligence, and in particular to a method for predicting the outcome of frozen embryo transplantation based on multimodal data. Background Art

[0002] Infertility has become one of the important issues that need to be solved in my country's health field. Its main treatment method is human assisted reproductive technology (ART). Human assisted reproductive technology mainly refers to artificial insemination (AI) and in vitro fertilization and embryo transfer (IVF-ET), of which the latter is the mainstream choice. In 40 years, IVF technology has undergone three generations of changes. The third generation of IVF technology can not only solve female infertility and male infertility, but also meet people's needs for eugenics through pre-transplantation screening.

[0003] In vitro fertilization technology uses artificial methods to fertilize eggs and sperm in vitro, and then carry out early embryo development, and then transplant them into the mother's uterus for pregnancy and development, and give birth in a normal way after maturity. Existing in vitro fertilization treatment methods rely more on the doctor's personal clinical experience to select suitable embryos for transplantation, but for inexperienced doctors, it is easy to make mistakes in judgment and cause transplantation failure, and the treatment results can only be determined by passing a pregnancy test two weeks after the transplanted embryo, and the economic cost is also high. Therefore, the failure of in vitro fertilization treatment will not only bring a major mental blow to infertile families, but also bear a huge time and economic burden. If computers can be used to assist in the prediction of in vitro fertilization results, compared with the traditional personal experience of doctors, more accurate and objective predictions of treatment results can be provided, which has important clinical significance for the doctor's subsequent clinical practice, medical advice and psychological adjustment of patients.

[0004] In recent years, deep learning methods have made great achievements in the field of image processing, which also provides the possibility of using deep learning technology to assist doctors in predicting pregnancy outcomes. The convolutional neural network (CNN) based on deep learning has a complex network structure and has more powerful feature learning and feature expression capabilities than traditional machine learning methods. The residual neural network (ResNet) is a classic convolutional neural network model that has achieved good results in the clinical auxiliary diagnosis of major diseases such as lung tumors, breast cancer, skin diseases, and cardiovascular and cerebrovascular diseases. In addition, the combination of residual neural networks and other network models is also widely used in medical image analysis tasks such as medical image segmentation, classification, fusion, reconstruction, synthesis, and super-resolution, which can significantly improve the diagnostic accuracy and efficiency of clinicians, and help evaluate the prognosis of patients. The convolutional block attention module (CBAM) can provide the model with more comprehensive and effective feature extraction capabilities, especially for small sample medical image segmentation. The feature pyramid (FPN) is a method to enhance the hierarchical features of the convolutional network and is widely used in object detection and semantic segmentation.

[0005] At present, there are some in vitro fertilization result prediction methods based on artificial intelligence technology, such as the existing authorized patents: an embryo pregnancy result prediction device based on segmented sampling video features (patent number: CN109214375B), an embryo pregnancy result prediction device based on multimodality (patent number: CN109544512B), an embryo pregnancy state intelligent prediction method and system (patent number: CN111783854B), and a method, processor and device for determining the pregnancy result of embryo transplantation (patent number: CN116433652B) all provide embryo pregnancy prediction technology. However, these existing methods still have the following disadvantages:

[0006] 1. Existing methods: An embryo pregnancy outcome prediction device based on segmented sampling video features (patent number: CN109214375B) and an embryo pregnancy outcome prediction device based on multimodality (patent number: CN109544512B) are based on the blastocyst grading system proposed by embryologists, relying on human subjectivity and traditional experience to select embryo images or embryo videos in the blastocyst stage. Therefore, there is a lack of reliable and scientific image data, especially for inexperienced clinicians, which are prone to judgment errors when selecting, and require a lot of time and energy for labeling and screening. In addition, the above methods select the morphology and structure of a specific period of the embryonic development stage for analysis and feature extraction, but the embryo is in a dynamic development process, so it is impossible to comprehensively and real-time evaluate the embryonic characteristics, and it is impossible to obtain accurate and ideal prediction results.

[0007] 2. Existing methods: An embryo pregnancy outcome prediction device based on segmented sampling video features (patent number: CN109214375B), an embryo pregnancy outcome prediction device based on multimodality (patent number: CN109544512B), and an embryo pregnancy status intelligent prediction method and system (patent number: CN111783854B) only focus on embryo image data, and do not consider other factors that may affect the results of embryo transplantation, such as the influence of clinical data features on embryo pregnancy; the method, processor and device for determining the pregnancy outcome of embryo transplantation (patent number: CN116433652B) considers the patient's clinical medical record text report during the analysis process, but it extracts and analyzes the embryo image features and clinical data features separately, but does not combine them for analysis. Embryonic development and the patient's clinical characteristics are interrelated and influence each other, so the above methods are difficult to apply in actual clinical scenarios to obtain accurate prediction results.

[0008] 3. Existing methods: An embryo pregnancy outcome prediction device based on segmented sampling video features (patent number: CN109214375B). Although the data source is the blastocyst stage embryo image video, in its model, only the image frames are extracted and classified in each time period, and there are still certain limitations in the behavioral judgment of long-term videos.

[0009] 4. The existing method is a multimodal embryo pregnancy outcome prediction device (patent number: CN109544512B). The prediction model selected only uses the ResNet deep learning model for training and analysis. However, the embryo image features are real-time developmental changes and extremely complex, and may require continuous parameter adjustment and retraining. Therefore, improving the model's feature extraction capability and model accuracy is conducive to quickly obtaining accurate prediction results. Summary of the invention

[0010] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a method for predicting the outcome of frozen embryo transfer based on multimodal data, and use a multimodal neural network to predict the success rate of transplant pregnancy for patients before transplantation into the mother. Provide doctors and patients with more accurate in vitro fertilization treatment references, thereby helping to improve the final clinical pregnancy success rate of patients.

[0011] In order to achieve the above-mentioned purpose, the present invention provides a method for predicting the outcome of frozen embryo transplantation based on multimodal data, and the model construction includes: obtaining image data from embryos and clinical data of patients; an image feature extraction module, using a feature extraction network with a ResNet-50 model as the backbone, combined with a CBAM attention mechanism and an FPM feature pyramid network to extract features of image data; a clinical data feature processing module, composed of two layers of fully connected neural networks with the same number of neurons as the input features and a LeakyReLU activation function, to enhance the perception and feature extraction capabilities of the patient's clinical data; a classification module, which predicts the results of in vitro fertilization, is composed of a feature concatenation layer, two layers of fully connected networks and corresponding LeakyReLU activation functions, and a Sigmoid activation function; and finally a binary mapping with a threshold of 0.5 is used to obtain the prediction result. When applied, the computer instructions implement the following steps when they are executed:

[0012] 1) Obtain two high-resolution embryo images and the patient's clinical data. If the patient is a single embryo transfer patient, the images are copied;

[0013] 2) Preprocess the images and resize them to a uniform resolution of 512x512 pixels by scaling and padding to maintain the aspect ratio; apply a standard augmentation pipeline to the training images, including horizontal and vertical flipping, rotation, and color adjustment;

[0014] 3) The clinical data of the patients corresponding to the embryo images were organized into a table, which included three clinical characteristics of each patient: "age", "uterine position" and "endometrial thickness";

[0015] 4) The pre-processed images are passed into the model with shared weights. The artificial intelligence model uses an improved ResNet-50 deep learning network as a feature extractor, which combines the CBAM attention mechanism with the FPN feature pyramid network to extract image features, so that the embryo image is represented with enhanced features;

[0016] 5) The features of the two images are directly stacked after average pooling and reduced to 8 dimensions through two fully connected layers (i.e., classification layers). The activation function after the two fully connected layers is LeakyReLU;

[0017] 6) In another route, a fully connected neural network is used to process the patient clinical data features, including a batch normalization layer and two fully connected layers, and the number of neurons in the fully connected layer is equal to the number of clinical features of the input patient;

[0018] 7) The image features and clinical data features are input into the classification module. After average pooling, the image features and clinical data features are directly integrated through the feature concatenation layer to form a unified feature vector. After two layers of fully connected layers and the corresponding activation function, the Sigmoid activation function is input, and finally the prediction result is obtained through binary mapping with a threshold of 0.5;

[0019] In the above technical solution, high-resolution embryo images based on real clinical cases and clinical reports of patients are used as raw data. The aspect ratio of the embryo images is adjusted to obtain standard resolution, and the preprocessed images and sorted patient clinical data are used as input to the artificial intelligence model. The model is based on ResNet-50, combined with the CBAM attention mechanism and the FPN feature pyramid network, and adds a branch route for processing clinical data features. Finally, the image features and clinical data features are stacked and merged, thereby utilizing the correlation between the two features to finally obtain the prediction results.

[0020] The prediction results combine multiple types of data features to provide a more comprehensive prediction perspective. Deep learning automatically extracts complex features from embryo images and clinical data, making the prediction results more accurate than doctors' experience. After the model training is completed, the prediction results can be obtained by combining two high-resolution images of the same patient with the patient's clinical data. If the patient is a single embryo transplant patient, the image is copied. The prediction results provide important reference for patients and clinicians in their treatment and transplantation decisions.

[0021] In step 4), the ResNet-50 model is used as the backbone of the image feature extraction module, and the CBAM attention mechanism is placed after each layer of downsampling in the feature pyramid network structure. Its kernel function formula is:

[0022]

[0023] Among them, σ is the Sigmoid operation, 7×7 represents the size of the convolution kernel, MLP is a two-layer neural network, F is the input, AvgPool is the average pooling, and MaxPool is the maximum pooling.

[0024] The purpose of using the CBAM attention mechanism is to improve the model's perception, allowing the model to better focus on the important parts of the image; improve network performance: even without significantly increasing the computational cost, it can significantly improve the performance of the network; wide applicability: it can be easily added to any convolutional neural network architecture to enhance the robustness and generalization ability of the model.

[0025] The purpose of using feature pyramid network is: multi-scale feature fusion: combining features of different scales has advantages in processing objects of different sizes; improving feature utilization: the combination of high-level semantic information and low-level detail information improves the efficiency of feature utilization; suitable for a variety of visual tasks: whether it is fine-grained image segmentation or large-scale object detection, feature pyramid network can provide good feature support.

[0026] The final predicted value in step 7) is mapped to a threshold value with 0.5 as the boundary. A value greater than 0.5 indicates pregnancy (1), and a value less than 0.5 indicates non-pregnancy (0).

[0027] Compared with the prior art, the beneficial effects of the present invention are:

[0028] 1. The present invention integrates and analyzes multiple data types, including visual information extracted from embryo images and information from patient clinical data, to provide doctors with more scientific decision support, improve prediction accuracy, reduce uncertainty in subjective judgment, and reduce errors caused by doctors' personal inexperience;

[0029] 2. The present invention realizes the innovation of combining the convolutional neural network, FPN feature pyramid network and CBAM attention mechanism. Deep learning is applied to automatically identify and learn complex patterns in embryo images to reduce human errors. The combination of FPN feature pyramid network and CBAM attention mechanism makes the model more perceptive, allowing the model to pay attention to and obtain embryo feature information more comprehensively and efficiently;

[0030] 3. The present invention realizes the fusion of embryo image features and clinical data features, and uses a unified feature vector for prediction, which improves the clinical practicality and interpretability of the present invention and provides a more comprehensive prediction perspective. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a flow chart of the prediction model in an embodiment of the present invention when predicting the results of in vitro fertilization;

[0032] Figure 2 An image feature extraction module in an embodiment of the present invention;

[0033] Figure 3 is a residual module in the network model used in the embodiment of the present invention;

[0034] Figure 4 CBAM attention mechanism structure diagram in an embodiment of the present invention;

[0035] Figure 5 CBAM channel attention structure diagram in an embodiment of the present invention;

[0036] Figure 6 CBAM spatial attention structure diagram in an embodiment of the present invention;

[0037] Figure 7 Schematic diagram of lateral connection of feature pyramid network in an embodiment of the present invention;

[0038] Figure 8 Schematic diagram of classification layer in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clear, the specific implementation methods of the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0040] Figure 1 The flowchart of the prediction model of the embodiment of the present invention when predicting the results of in vitro fertilization is shown. In the embodiment of the present invention, a method for predicting the outcome of frozen embryo transplantation based on multimodal data is provided, comprising the following steps:

[0041] Step 101, obtaining two high-resolution embryo images and clinical data from the same patient. If the patient is a single embryo transplant patient, the image is copied.

[0042] Step 102, preprocessing the images, uniformly resizing the images to a standard resolution of 512x512 pixels by scaling and padding to maintain the aspect ratio; applying a standard enhancement pipeline to the training images, including horizontal and vertical flipping, rotation, and color adjustment, to enhance the robustness of the model against variable imaging conditions.

[0043] Step 103 organizes the clinical data of the patient corresponding to the embryo image into a table, which includes three clinical characteristics of each patient: "age", "uterine position" and "endometrial thickness".

[0044] Step 104 passes the preprocessed image into the artificial intelligence model with shared weights. The model uses the improved ResNet-50 deep learning network as a feature extractor. The extractor combines the CBAM attention mechanism with the FPN feature pyramid network to extract image features. The CBAM position is located after each layer of FPN downsampling to enhance the feature representation of the embryo image.

[0045] like Figure 2As shown in the figure, the image feature extraction module includes a 7×7 convolutional layer, a maximum pooling layer (MaxPool), 4 blocks, and an average pooling layer connected in sequence; the CBAM attention mechanism is used before upsampling each block, and the feature pyramid network is used to upsample the results of each layer of blocks by integrating multi-scale context information and focusing on the information area therein; block1 includes three residual modules, block2 includes four residual modules, block3 includes five residual modules, and block4 includes three residual modules.

[0046] like Figure 3 As shown in Figure 1, each residual module includes 3 convolution operations and 3 ReLU activation functions.

[0047] Figure 4 This is the structure diagram of the CBAM attention mechanism in the embodiment of the present invention. After the intermediate feature map is input, the attention is inferred independently along the channel and space dimensions in turn, and the attention map and the input feature map are adaptively optimized. The kernel function formulas of the channel attention module and the spatial attention module are:

[0048]

[0049] Among them, σ is the Sigmoid operation, 7×7 represents the size of the convolution kernel, MLP is a two-layer neural network, F is the input, AvgPool is the average pooling, and MaxPool is the maximum pooling.

[0050] Channel attention module such as Figure 5 As shown in the figure, the input intermediate feature map is subjected to global maximum pooling and global average pooling based on width and height, and then to MLP. The features output by MLP are summed up based on element level, and then processed by Sigmoid activation function to generate the final channel attention feature map. The channel attention feature map and the input feature map are multiplied at element level to generate the input features required by the spatial attention module.

[0051] The spatial attention module is Figure 6 As shown in the figure, the feature map output by the channel attention module is used as the input feature map of this module. First, a global maximum pooling and global average pooling based on the channel are performed, and then the two results are spliced ​​based on the channel. Then, a convolution operation is performed to reduce the dimension to 1. The spatial attention feature is then generated through the Sigmoid activation function. Finally, the feature is multiplied with the input feature of the module to obtain the final generated feature.

[0052] Step 105 is used to process the patient's clinical data in another route, which includes a batch normalization layer and two fully connected layers, and the number of neurons in each layer is equal to the number of clinical features of the input patient.

[0053] Step 106: The features of the two images are directly stacked after average pooling, and are reduced to 8 dimensions using two fully connected layers (i.e., classification layers), with the activation function being LeakyReLU.

[0054] Step 107 inputs the image features and clinical data features into the classification module, and the unified feature vector obtained by the feature integration layer is input into the Sigmoid activation function after passing through two fully connected layers, and finally the prediction result is obtained through binary mapping with a threshold of 0.5.

[0055] Step 108 displays the prediction results.

[0056] like Figure 8 As shown in the figure, the classification module passes the unified vector through two fully connected layers and the corresponding LeakyReLU activation function, and then inputs it into the final Sigmoid activation function. The prediction result is obtained by binary mapping with 0.5 as the boundary. If it is greater than 0.5, it means pregnancy (1), and if it is less than 0.5, it means non-pregnancy (0).

[0057] Compared with the personal experience of clinicians, the predicted results provide doctors and patients with accurate and objective reference indicators, thereby helping to develop more reliable follow-up treatment plans and improve the success rate of IVF.

Claims

1. A method for predicting frozen embryo transplantation outcomes based on multimodal data, characterized in that: The construction of the model includes: 1) Obtain image data from embryos and clinical data from patients; 2) Image feature extraction module, which uses an improved ResNet-50 convolutional neural network to extract image features, combined with a feature extraction network using the Feature Pyramid Network (FPN) and Convolutional Block Attention Module (CBAM) attention mechanism; 3) Clinical data feature processing module, which consists of a multi-layer fully connected neural network and a LeakyReLU activation function, to enhance the ability to perceive the patient's clinical data and extract features; 4) Classification module, which predicts the results of in vitro fertilization, consists of a feature concatenation layer, two layers of fully connected networks and a final Sigmoid activation function. The fully connected neural network directly uses the LeakyReLU activation function to enhance the expression of nonlinear features; 5) The final prediction result, i.e., whether the woman is pregnant or not, is obtained through binary mapping with a threshold of 0.

5.

2. The method for predicting frozen embryo transplantation outcomes based on multimodal data according to claim 1, characterized in that: In clinical practice, the computer executes instructions according to the following steps: 1) Obtain two high-resolution embryo images and corresponding clinical data from the same patient. If the patient is a single embryo transfer patient, the image is duplicated; 2) Preprocessing images: The images are automatically resized to a standard resolution of 512×512 pixels before being fed into the model. The aspect ratio is maintained by scaling and padding. The standard enhancement pipeline including horizontal and vertical flipping, rotation, and color adjustment is applied to the images. 3) The clinical data of the patients corresponding to the embryo images were organized into a table, which included three clinical characteristics of each patient: "age", "uterine position", and "endometrial thickness", for the next step of processing; 4) The pre-processed images are passed into the model with shared weights. The artificial intelligence model uses the improved ResNet-50 deep learning network as the feature extractor, with ResNet-50 as the backbone, and combines the CBAM attention mechanism with the FPN feature pyramid network to extract image features, so that the embryo image is represented with enhanced features; 5) The features of the two images are average-pooled and then directly stacked, and reduced to 8 dimensions using two fully connected layers (i.e., classification layers) with LeakyReLU as the activation function; 6) In another route, the patient clinical data features are processed by a fully connected neural network, including a batch normalization layer and two fully connected layers, and the number of neurons in each layer is equal to the number of features in the input patient clinical data; 7) The image features and clinical data features are input into the classification module, and the image features and clinical data features are directly integrated through the feature concatenation layer to form a unified feature vector. After passing through two fully connected layers and the corresponding activation function, the Sigmoid activation function is used, and the binary mapping with a threshold of 0.5 is used to obtain the final prediction result.

3. The method for predicting frozen embryo transplantation outcomes based on multimodal data according to claim 1, characterized in that: In step 1), two embryo images of the patient on any given day are obtained simultaneously, as well as the patient's clinical information, including age, endometrial thickness and uterine position. If the patient is a single embryo transplant patient, the image is copied; In the step 2), an improved ResNet-50 is used as a feature extractor. The extractor is based on the ResNet-50 neural network, combined with the CBAM attention mechanism and the FPN feature pyramid. The CBAM attention mechanism is placed after the downsampling of each layer in the FPN feature pyramid. In the step 3), a batch normalization layer and two fully connected layers are used, and the number of neurons in each layer of the fully connected layer is equal to the number of features in the input patient clinical data; In the step 4), the features obtained from the two routes are directly concatenated after average pooling, and the LeakyReLU activation function is used after the fully connected network.

4. The method for predicting frozen embryo transplantation outcomes based on multimodal data according to claim 2, characterized in that: In the step 2), the preprocessed images are scaled and padded to maintain the aspect ratio, and the embryo images are uniformly adjusted to a resolution of 512x512 pixels. A standard enhancement pipeline is applied to the images, including horizontal and vertical flipping, rotation, and color adjustment, to train the model to improve the robustness of the model and its generalization ability to various imaging conditions.

5. A method for predicting frozen embryo transplantation outcomes based on multimodal data according to claim 2, characterized in that: The network structure of ResNet-50 in the step 4) includes a convolutional layer, a maximum pooling layer, 4 blocks, an average pooling layer, and a fully connected layer connected in sequence; wherein, block1 includes three residual modules, block2 includes four residual modules, block3 includes five residual modules, and block4 includes three residual modules, each residual module includes 3 convolution operations and 3 ReLU activation functions: a 1x1x3 convolution kernel is introduced in the last convolution operation of block2, and a convolution operation is performed on the fourth dimension to change the size of the fourth dimension to 1.

6. A method for predicting frozen embryo transplantation outcomes based on multimodal data according to claim 2, characterized in that: The feature pyramid network in step 4) connects high-level features with low resolution and high semantic information with low-level features with high resolution and low semantic information from top to bottom, so that features at all scales have rich semantic information.

7. A method for predicting frozen embryo transplantation outcomes based on multimodal data according to claim 2, characterized in that: The CBAM attention mechanism in step 4) includes a channel attention module and a spatial attention module, which perform weighted attention on the channel and space, respectively. It is placed after each layer of downsampling of the FPN feature pyramid.

8. The CBAM attention mechanism according to claim 7, characterized in that The channel attention module compresses the channel attention feature map in the spatial dimension to obtain a one-dimensional vector and then performs average pooling or maximum pooling operations. Average pooling and maximum pooling can be used to aggregate the spatial information of the feature map. The calculation formula is as follows: ， Where σ is the Sigmoid operation, MLP is a two-layer neural network, F is the input, AvgPool is the average pooling, and MaxPool is the maximum pooling.

9. The CBAM attention mechanism according to claim 7, characterized in that: The spatial attention module compresses the channel and performs average pooling and maximum pooling in the channel dimension. MaxPool extracts the maximum value on the channel, and the number of extractions is height multiplied by width; AvgPool extracts the average value on the channel, and the number of extractions is also height multiplied by width; then the previously extracted feature maps (the number of channels is 1) are merged to obtain a 2-channel feature map, and its calculation formula is as follows: Among them, σ is the Sigmoid activation function, and 7*7 represents the size of the convolution kernel.

Images