Embryo pregnancy prediction method and system based on spatiotemporal attention and cross-modal fusion

Abstract

The present invention discloses an embryo pregnancy prediction method and system based on spatiotemporal attention and cross-modal fusion, including collecting embryo images at various developmental stages during embryo culture, extracting clinical data and medical record data of patients; preprocessing and data enhancement of embryo images to obtain image input data, data cleaning and feature engineering of clinical data to obtain clinical input data; inputting clinical input data into embryo quality assessment model and clinical medical data mining model respectively to obtain embryo morphological feature vector and clinical feature vector respectively; using cross-modal feature fusion method to fuse the features of two modalities to obtain fusion features of embryo morphological features and clinical features; inputting fusion features into a predictor to predict the probability of embryo pregnancy. The automatic analysis and prediction results of the present invention have good robustness and interpretability.

Description

Technical Field

[0001] The present invention relates to the field of medical artificial intelligence, and in particular to an embryo pregnancy prediction method and system based on spatiotemporal attention and cross-modal fusion. Background Art

[0002] A considerable portion of the world's reproductive age population faces problems of infertility and subfertility. According to the latest survey, about 13% of women and 10% of men are affected by infertility.

[0003] IVF-ET is one of the most effective and common methods of treating infertility in Assisted Reproductive Technology (ART). During an IVF cycle, multiple mature oocytes are retrieved from a woman's ovaries and fertilized in the laboratory using in vitro fertilization (IVF) or intracytoplasmic sperm injection (ICSI). These fertilized eggs (embryos) are then cultured in an incubator under optimal conditions. These embryos can be transferred into the uterus on day 3, 4, or 5 of incubation, or cryopreserved for a later transfer.

[0004] The quality of the embryo directly affects the pregnancy outcome after its transplantation. Embryo morphology assessment is one of the most effective and commonly used methods for embryo quality assessment. Embryologists grade the quality of embryos based on the internal morphological structures of the embryo, such as the blastocyst cavity, inner cell mass, and trophoblast. Therefore, embryologists usually evaluate the quality of embryos through embryo morphology and select the best quality embryos for transplantation. However, embryologists' assessment of embryo morphology is highly subjective. For inexperienced doctors, embryo assessment is very complicated, and it is inevitable that subjective factors will lead to misjudgment of the size of the blastocyst cavity, the grade of the inner cell mass and trophoblast, etc., which in turn affects the selection of embryos and leads to a lower pregnancy rate.

[0005] Although some artificial intelligence-based embryo pregnancy prediction methods have been proposed, such as the existing authorized patents CN109214375B and CN109544512B both provide embryo pregnancy prediction technologies. However, these existing methods still have the following problems:

[0006] 1. Existing methods are highly dependent on the blastocyst grading system proposed by embryologists, and usually require multiple embryologists to uniformly annotate the data set. Therefore, there is a lack of reliable and large amounts of training data, making it difficult to apply in actual engineering.

[0007] 2. Existing methods only focus on the morphology of embryos at a specific period, such as the morphological characteristics of embryos when they develop into blastocysts, or specific structures inside embryos, such as the inner cell mass and trophoblast of embryos. However, embryonic development is a dynamic process, so these methods cannot fully evaluate embryonic activity.

[0008] 3. Existing methods only focus on embryo morphological characteristics when predicting pregnancy, ignoring important factors that affect pregnancy outcomes, such as the patient's age and medical history, and are therefore difficult to apply in actual clinical scenarios. Summary of the invention

[0009] The purpose of the present invention is to provide an embryo pregnancy prediction method and system based on spatiotemporal attention and cross-modal fusion, which not only fully analyzes the morphological characteristics of the embryo throughout the development process and efficiently obtains embryo characteristics in time and space, but also makes full use of important clinical factors such as the patient's age and medical history, so as to effectively and accurately predict the embryo pregnancy rate.

[0010] In order to achieve the above object, the embryo pregnancy prediction method proposed by the present invention comprises the following steps:

[0011] S1. Collect clinical data such as the patient's age, medical history, and medication history, and collect embryo images at various developmental stages during embryo culture;

[0012] S2. Perform unified preprocessing and data enhancement on embryo images to obtain image input data, and perform data cleaning and feature engineering on clinical data to obtain clinical input data;

[0013] S3. inputting the input data into the embryo quality assessment model and the clinical medicine data mining model respectively to obtain the embryo morphological feature vector and clinical characteristics;

[0014] S4. Use cross-modal feature fusion technology to obtain the fusion features of embryonic morphological features and clinical features;

[0015] S5. Input the fused features into the predictor to predict the probability of embryo pregnancy.

[0016] The specific sub-steps of step S2 are as follows:

[0017] S201. Grayscale the embryo image to obtain an embryo grayscale image. The grayscale processing adopts the formula Gray=0.299R+0.587G+0.114B;

[0018] S202. Processing the embryo grayscale image with the Canny operator to obtain a binary edge image of the original embryo image;

[0019] S203. Detect the center of the circle in the binary edge image using Hough transform to obtain the position coordinates of the center of the circle;

[0020] S204. The original embryo image is intercepted by the position coordinates of the circle detection, so that the embryo is distributed in the middle of the image and the image resolution is 448*448, and the preprocessed embryo image is obtained;

[0021] S205. Performing data enhancement on the preprocessed embryo image, including chroma, brightness, saturation, and contrast conversion, to obtain image input data;

[0022] S206. Perform data cleaning on the clinical data, including removing and completing missing data, removing and correcting erroneous data, correcting data format, removing outliers, etc., to obtain cleaned data;

[0023] S207. Perform feature engineering on the cleaned data, including data normalization, data transformation, data dimension reduction, feature selection, etc., to obtain clinical input data;

[0024] The specific sub-steps of step S3 are as follows:

[0025] S301. ResNet-50 is used as the basic network of the embryo quality assessment model, a spatial attention module is added to the residual block, multiple images of the same embryo are input into the network, and morphological features of embryos at different stages are extracted;

[0026] S302. Performing temporal feature fusion on the features of the same embryo at different developmental stages, and obtaining the final embryo morphological feature vector through a temporal attention module;

[0027] S303. Input the clinical input data into the Adaboost model to obtain a clinical feature vector;

[0028] The specific sub-steps of step S4 are as follows:

[0029] S401. Performing data dimension reduction processing on the embryo morphological feature vector;

[0030] S402. Align and concatenate the morphological feature vector after dimension reduction with the clinical feature vector;

[0031] S403. Input the concatenated feature vector into a cross-modal feature fusion module to extract fusion features;

[0032] To achieve the above object, the present invention further provides an embryo pregnancy prediction system, the system comprising:

[0033] The data preprocessing module is used to preprocess embryo images of the same embryo at different developmental stages to obtain input images, and to perform data engineering on clinical data to obtain corresponding clinical input data;

[0034] An embryo quality assessment module, used for inputting the input image into a feature extraction network based on a spatiotemporal attention mechanism to obtain complete morphological features of the embryo;

[0035] The clinical data evaluation module is used to input clinical input data into the clinical medical data mining model to obtain effective clinical features;

[0036] The cross-modal feature fusion module is used to fuse the embryo image features and clinical features across modalities to obtain fused pregnancy features;

[0037] The embryo pregnancy prediction module is used to input the fused pregnancy characteristics into the prediction unit to obtain the embryo pregnancy rate, thereby assisting doctors in embryo selection.

[0038] Compared with the current embryo pregnancy prediction method and system, this technical solution has the following beneficial effects:

[0039] 1. The present invention innovatively uses embryo images at different stages of the entire embryonic development process and the complete embryonic structure to predict embryonic pregnancy, avoiding one-sided analysis caused by focusing only on the blastocyst stage of the embryo or only focusing on some of the internal structures of the embryo, thereby obtaining embryo quality information more comprehensively and efficiently, predicting embryonic pregnancy outcomes, and assisting doctors in quickly evaluating and selecting embryos.

[0040] 2. The present invention innovatively introduces clinical data, analyzes clinical data through a clinical medical data mining model, obtains clinical feature vectors, and performs cross-modal feature fusion with embryo image feature vectors to predict pregnancy rates, thereby improving the robustness and interpretability of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is a flow chart of an embryo pregnancy prediction method provided by an embodiment of the present invention;

[0042] Figure 2 is a network diagram of the attention mechanism in an embodiment of the present invention; DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments herein are provided for illustrative purposes only and cannot be understood as limiting the present invention.

[0044] The existing embryo pregnancy prediction methods mainly analyze the embryo images in the blastocyst period, which cannot fully utilize the information of the entire embryo development process, or only analyze specific structures such as the blastocyst cavity, inner cell mass, trophoblast, etc., which cannot fully capture the embryo morphological characteristics, resulting in the problem that the embryo pregnancy rate cannot be accurately and reliably predicted. See Figure 1 , Figure 1This is a flow chart of a method for embryo pregnancy prediction based on spatiotemporal attention and cross-modal fusion provided by an embodiment of the present invention, wherein the method comprises the following steps:

[0045] S1. Collect images of embryos at various developmental stages during embryo culture and obtain clinical data and medical records of patients;

[0046] In the embodiment, embryo images are collected at various stages of embryo development. Before embryo transplantation, embryo images are collected at different developmental stages of the embryo, and embryo images are collected before and after embryo freezing.

[0047] The patient's clinical medical data and embryo medical history data are collected, including age, height, weight, infertility factors, hormone levels and other data that may affect pregnancy, and the data are merged accordingly based on the patient.

[0048] S2. Perform preprocessing and data enhancement on embryo images to obtain image input data, and perform data cleaning and feature engineering on clinical data to obtain clinical input data;

[0049] 1) Perform unified preprocessing and data enhancement on the embryo image to obtain image input data. The preprocessing operation is as follows:

[0050] First, the embryo image is grayed using the formula Gray = 0.299R + 0.587G + 0.114B to obtain an embryo gray image. Then, the embryo gray image is processed using the Canny operator to obtain a binary edge image of the original embryo image, and the center of the circle in the binary edge image is detected using the Hough transform to obtain the position coordinates of the center of the circle. Finally, the original embryo image is intercepted using the coordinates so that the embryo is distributed in the middle of the image and remains intact. The image resolution is 448*448, and the preprocessed embryo image is obtained.

[0051] Data enhancement is performed on the preprocessed embryo images, including chromaticity transformation, brightness transformation, saturation transformation, contrast transformation, etc.

[0052] 2) Perform data cleaning and feature engineering on clinical data to obtain clinical input data. The data cleaning process is as follows:

[0053] First, perform statistical analysis on the clinical data, then remove or complete the corresponding missing data based on the statistical analysis results, check and correct the erroneous data, and finally check and correct the data format, remove outliers, etc., to obtain the clinical data after data cleaning.

[0054] Perform feature engineering on the clinical data after the above data cleaning. The specific process includes:

[0055] First, select appropriate data normalization formulas for different features, including: Where μ is the mean of all sample data, σ is the standard deviation of all sample data, and x is the original sample data; Among them, max is the maximum value of the sample data, and min is the minimum value of the sample data; perform data transformation on the corresponding features, such as discretizing continuous features; perform data dimensionality reduction and data selection on the data to reduce the number and dimension of clinical data features.

[0056] In specific implementation, S2 can be implemented using the following sub-steps:

[0057] S201. Grayscale the embryo image to obtain an embryo grayscale image. The grayscale processing adopts the formula Gray = 0.299R + 0.587G + 0.114B, where Gray is the grayscale, and R, G, and B are the corresponding channel values ​​respectively;

[0058] S202. Processing the embryo grayscale image with the Canny operator to obtain a binary edge image of the original embryo image;

[0059] S203. Detect the center of the circle in the binary edge image using Hough transform to obtain the position coordinates of the center of the circle;

[0060] S204. The original embryo image is intercepted by the position coordinates of the circle detection, so that the embryo is distributed in the middle of the image and the image resolution is 448*448, and the preprocessed embryo image is obtained;

[0061] S205. Performing data enhancement on the preprocessed embryo image, including chroma, brightness, saturation, and contrast conversion, to obtain image input data;

[0062] S206. Perform data cleaning on the clinical data, including removing and completing missing data, removing and correcting erroneous data, correcting data format, removing outliers, etc., to obtain cleaned data;

[0063] S207. Perform feature engineering on the above cleaned data, including data normalization, data transformation, data dimension reduction, feature selection, etc., to obtain clinical input data.

[0064] S3. Input the input data into an embryo quality assessment model and a clinical medical data mining model respectively, to obtain an embryo morphological feature vector and a clinical feature vector respectively.

[0065] 1) Input embryo image data into the embryo quality assessment model based on spatiotemporal attention mechanism to obtain embryo morphological characteristics. The attention module structure can be found in Figure 2Among them, Residual is a residual block, which consists of a 3×3 convolution layer and an activation layer; Global pooling is a global pooling layer, which converts the feature map of size H×W×C output by the residual block into 1×1×C (where H, W, and C represent the height, width, and number of channels of the feature map, respectively); FC is a fully connected layer, which reduces the number of channels of the feature map by scaling parameter r. Then the number of channels is restored to C; ReLU and Sigmoid are two activation functions respectively; the Scale module restores the size of the 1×1×C feature map output by the Sigmoid layer to H×W×C, and finally adds it to the input feature map X. The output result is recorded as

[0066] Furthermore, the temporal attention is calculated as follows:

[0067] Mt(F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F)))

[0068] Where Mt() represents temporal attention, F represents input features, σ() represents the sigmoid function, AvgPool() and MaxPool() represent average pooling and maximum pooling respectively, and MLP() represents multi-layer perceptron.

[0069] The spatial attention is calculated as follows:

[0070] Ms(F′)=σ(f 7×7 ([AvgPool(F′);MaxPool(F′)]))

[0071] Where Ms() represents spatial attention, F′ is the feature map output by the temporal attention module, σ() represents the sigmoid function, and f 7 × 7 () represents a 7×7 convolution, AvgPool() and MaxPool() represent average pooling and maximum pooling respectively.

[0072] 2) Input the clinical data after data cleaning and feature engineering into the clinical medical data mining model to obtain the clinical feature vector;

[0073] In specific implementation, S3 can be implemented using the following sub-steps:

[0074] S301. ResNet-50 is used as the basic network of the embryo quality assessment model, a spatial attention module is added to the residual block, multiple images of the same embryo are input into the network, and morphological features of embryos at different stages are extracted;

[0075] S302. Performing temporal feature fusion on the features of the same embryo at different developmental stages, and obtaining the final embryo morphological feature vector through a temporal attention module;

[0076] S303. Input the clinical input data into the Adaboost model to obtain a clinical feature vector.

[0077] S4. Use cross-modal feature fusion technology to obtain the fusion features of embryonic morphological features and clinical features. The calculation method of fusion features is as follows:

[0078] E(F1,F2)=σ(MLP(Maxpool([f 1×1 (F1); f 1×1 (F2)])))

[0079] Where E() represents the fusion feature, F1 and F2 represent the embryonic morphological feature vector and clinical feature vector respectively, σ() represents the sigmoid function, MaxPool() represents the maximum pooling, and f 1×1 () represents a 1×1 convolution.

[0080] In specific implementation, S4 can be implemented using the following sub-steps:

[0081] S401. Performing data dimension reduction processing on the embryo morphological feature vector;

[0082] S402. Align and concatenate the morphological feature vector after dimension reduction with the clinical feature vector;

[0083] S403. Input the concatenated feature vector into a cross-modal feature fusion module to extract fusion features.

[0084] S5. Input the fused features into the predictor to predict the probability of embryo pregnancy.

[0085] In specific implementation, the method proposed in the technical solution of the present invention can be implemented by technical personnel in this field using computer software technology to realize automatic operation process. System devices for implementing the method, such as computer-readable storage media storing the corresponding computer program of the technical solution of the present invention and computer equipment running the corresponding computer program, should also be within the scope of protection of the present invention.

[0086] In some possible embodiments, a system for predicting embryo pregnancy based on spatiotemporal attention and cross-modal fusion is provided, comprising the following modules:

[0087] Image data processing module, used to collect embryo images at various developmental stages during embryo culture and extract clinical data and medical records of patients;

[0088] The clinical data processing module is used to preprocess and enhance the embryo images to obtain image input data, and to perform data cleaning and feature engineering on the clinical data to obtain clinical input data;

[0089] A morphological feature extraction module is used to input clinical input data into an embryo quality assessment model and a clinical medical data mining model to obtain an embryo morphological feature vector and a clinical feature vector respectively;

[0090] A cross-modal feature fusion module is used to fuse the features of the two modalities using a cross-modal feature fusion method to obtain fusion features of embryonic morphological features and clinical features;

[0091] The pregnancy outcome prediction module is used to input the fusion features into the predictor to predict the probability of embryo pregnancy.

[0092] In some possible embodiments, a system for predicting embryo pregnancy based on spatiotemporal attention and cross-modal fusion is provided, including a processor and a memory, wherein the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute a method for predicting embryo pregnancy based on spatiotemporal attention and cross-modal fusion as described above.

[0093] In some possible embodiments, a system for predicting embryo pregnancy based on spatiotemporal attention and cross-modal fusion is provided, including a readable storage medium having a computer program stored thereon. When the computer program is executed, a method for predicting embryo pregnancy based on spatiotemporal attention and cross-modal fusion as described above is implemented.

[0094] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the scope of the claims.

Claims

1. A method for embryo pregnancy prediction based on spatiotemporal attention and cross-modal fusion, characterized in that: The following steps are involved: S1. Collect embryo images at various developmental stages during embryo culture and extract the patient's clinical data and medical records; S2. Perform preprocessing and data enhancement on embryo images to obtain image input data, and perform data cleaning and feature engineering on clinical data to obtain clinical input data; S3. Input the clinical input data obtained in step S2 into the embryo quality assessment model and the clinical medicine data mining model, respectively, to obtain an embryo morphological feature vector and a clinical feature vector; S4. Using cross-modal feature fusion to fuse the features of the two modalities, we can obtain the fusion features of embryonic morphological features and clinical features; S5. Input the fusion features obtained in step S4 into the predictor to predict the probability of embryo pregnancy; The step S2 includes the following sub-steps: S201. grayscale the embryo image to obtain an embryo grayscale image; S202. Processing the embryo grayscale image obtained in S201 with the Canny operator to obtain a binary edge image of the original embryo image; S203. Using Hough transform to detect the center of the circle in the binary edge image obtained in S202, obtain the position coordinates of the center of the circle; S204. The original embryo image is intercepted by the position coordinates obtained by the circle center detection in S203, so that the embryo is distributed in the middle of the image, and a preprocessed embryo image is obtained; S205. Performing data enhancement on the embryo image preprocessed in S204, including chroma, brightness, saturation, and contrast conversion, to obtain image input data; S206. Perform data cleaning on the clinical data, including removing and completing missing data, removing and correcting erroneous data, correcting data format, and removing outliers to obtain cleaned data; S207. Perform feature engineering on the cleaned data obtained in S206, including data normalization, data transformation, data dimension reduction, and feature selection, to obtain clinical input data; The step S3 includes the following sub-steps: S301. ResNet-50 is used as the basic network of the embryo quality assessment model, a spatial attention module is added to the residual block, multiple images of the same embryo are input into the network, and morphological features of embryos at different stages are extracted; S302. Fusing the features of the same embryo at different developmental stages in the time domain, and obtaining the final embryo morphological feature vector through the temporal attention module; S303. Input the clinical input data into the Adaboost model to obtain a clinical feature vector; The step S4 includes the following sub-steps: S401. Performing data dimension reduction processing on the embryo morphological feature vector; S402. Align and concatenate the morphological feature vector after dimension reduction with the clinical feature vector; S403. Input the concatenated feature vector into a cross-modal feature fusion module to extract fusion features.

2. An embryo pregnancy prediction system based on spatiotemporal attention and cross-modal fusion, characterized by: The method for embryo pregnancy prediction based on spatiotemporal attention and cross-modal fusion as claimed in claim 1 comprises the following modules: Image data processing module, used to collect embryo images at various developmental stages during embryo culture and extract clinical data and medical records of patients; The clinical data processing module is used to preprocess and enhance the embryo images to obtain image input data, and to perform data cleaning and feature engineering on the clinical data to obtain clinical input data; A morphological feature extraction module is used to input clinical input data into an embryo quality assessment model and a clinical medical data mining model to obtain an embryo morphological feature vector and a clinical feature vector respectively; A cross-modal feature fusion module is used to fuse the features of the two modalities using a cross-modal feature fusion method to obtain fusion features of embryonic morphological features and clinical features; The pregnancy outcome prediction module is used to input the fusion features into the predictor to predict the probability of embryo pregnancy.

Images