Multimodal fusion and adaptive adjustment of embryo development quality prediction method and system

Abstract

The application discloses a kind of multi-modal fusion and adaptive adjustment's embryo development quality prediction method and system, construct the multi-modal prediction model including embryo image sequence branch and table data branch, embryo image sequence branch removes redundant information using principal component analysis, morphological information is extracted from spatial dimension and time dimension to carry out inflation, erosion and gradient operation, using multi-channel multi-filter convolution and Mamba module extracts space-time feature and global information;Table data branch is encoded respectively to numerical type and category type characteristics and uses Mamba module to extract global information;Through cross-entropy loss, the model is trained;In the test stage, unsupervised entropy minimization is carried out preliminary adjustment, according to the change of entropy before and after adjustment, sample weight is calculated, and further adjustment is carried out using weighted entropy minimization.The application significantly improves the accuracy and robustness of embryo development quality prediction.

Description

Technical Field

[0001] This invention relates to the fields of assisted reproductive technology and medical artificial intelligence, specifically to a method and system for predicting embryo development quality using multimodal fusion and adaptive adjustment. Background Technology

[0002] In vitro fertilization (IVF-ET) is currently the mainstream method in assisted reproductive technology. This technique first involves fertilization of an egg with sperm in an artificially controlled in vitro environment, followed by early embryo culture. Once the embryo reaches a critical developmental stage, embryos with good developmental status and high developmental potential are selected and implanted into the mother's uterus. Throughout the process, accurate assessment of embryo quality and developmental potential is crucial through observation of embryo development. Time-lapse imaging monitoring systems play a vital role in this process. These systems can periodically capture images of the cultured embryos from multiple focal planes, creating a complete video sequence of embryonic development. This is currently the primary technique for observing and recording the dynamic process of embryonic development.

[0003] In recent years, with the rapid development of deep learning technology, representative deep learning architectures such as convolutional neural networks, long short-term memory networks, and Transformers have been widely applied to embryo development quality prediction tasks. However, existing methods still have the following shortcomings when processing embryo development data: First, embryo images at the same developmental stage often exhibit extremely high similarity, resulting in a large amount of redundant information in embryo development videos, and existing methods lack effective mechanisms for removing this redundancy. Second, embryo images at different developmental stages exhibit typical morphological features, such as the number of cell divisions and the degree of fragmentation, but existing methods have failed to fully mine and utilize these morphological features. Furthermore, existing methods also have significant shortcomings in modeling the spatiotemporal global dependencies of embryo development videos, making it difficult to effectively capture the long-term temporal evolution patterns during embryo development. These problems severely restrict the performance improvement of embryo development quality prediction.

[0004] On the other hand, most existing methods for predicting embryonic development quality rely solely on videos of embryonic development, neglecting the significant impact of clinically relevant information on the prediction results. In fact, tabular clinical data such as patient age, number of retrieved oocytes, and endometrial type are of considerable reference value for assessing embryonic development quality. Although multimodal fusion technology has seen some development in the medical field, research specifically targeting embryonic development prediction scenarios and effectively fusing video and tabular data remains relatively scarce.

[0005] Furthermore, in practical clinical applications, the challenge of inconsistent data distribution also arises. Time-lapse incubators require autofocus when acquiring embryonic development images; poor autofocus can lead to varying degrees of blurring in the acquired images. Simultaneously, due to the heterogeneous nature of embryonic development data—numerical and categorical features exhibiting different distribution characteristics, and the dependencies between features often being complex and unknown prior—the distribution of multimodal embryonic development data during the testing phase may differ from that of the training data, thus affecting the model's predictive performance. Therefore, adaptive adjustments to the model are necessary during the testing phase to address this distribution inconsistency issue. Summary of the Invention

[0006] This invention proposes a multimodal fusion and adaptive adjustment method and system for predicting embryonic development quality, in order to solve the problems of data redundancy, insufficient mining of morphological and global information, insufficient fusion of multimodal information, and inconsistent distribution of test data and training data in existing technologies.

[0007] To address the aforementioned technical problems, this invention provides a method for predicting embryonic development quality using multimodal fusion and adaptive adjustment, comprising the following steps: Step S1: Construct an embryo development quality prediction model based on multimodal fusion. The embryo development quality prediction model includes an embryo image sequence branch and a tabular data branch. The prediction result is output after fusing the features of the embryo image sequence branch and the tabular data branch. Step S2: Obtain a training dataset containing embryo image sequences and tabular data, and train the embryo development quality prediction model by minimizing cross-entropy loss to obtain the trained model parameters; Step S3: Input the embryo image sequence and table data to be evaluated in batches into the trained embryo development quality prediction model, perform preliminary adjustment of the embryo development quality prediction model by unsupervised entropy minimization, calculate the sample weights based on the changes in prediction entropy before and after the preliminary adjustment, perform further adjustment of the embryo development quality prediction model by unsupervised weighted entropy minimization, and output the final prediction result.

[0008] Preferably, the embryo image sequence branching uses principal component analysis to perform dimensionality reduction processing on the embryo image sequence, removing redundant information and retaining the main feature information.

[0009] Preferably, the embryo image sequence branch performs dilation, erosion, and gradient operations on the dimensionality-reduced embryo image sequence from the spatial dimension to extract morphological and detail information of the image. The results of the dilation, erosion, and gradient operations are then subjected to multi-channel, multi-filter convolution processing. The three convolution processing results are added and fused together, and the global information of the spatial dimension is extracted using the Mamba module.

[0010] Preferably, the embryo image sequence branch performs dilation, erosion, and gradient operations on the dimensionality-reduced embryo image sequence from the time dimension to extract morphological and detail information at different stages of embryonic development. The results of the dilation, erosion, and gradient operations are then subjected to multi-channel, multi-filter convolution processing. The three convolution processing results are added and fused together, and the global information in the time dimension is extracted using the Mamba module.

[0011] Preferably, the table data branch divides the table data into numerical features and category features, and encodes them separately. The global information of the encoded numerical features and category features is extracted using the Mamba module.

[0012] Preferably, the formula for calculating the cross-entropy loss in step S2 is: ; In the formula, and They represent the first Embryo image sequences and tabular data for each sample, Indicates the first The sample belongs to the first The true label of the class, This indicates that the embryo development quality prediction model is effective for the first... The sample belongs to the first Predicted probability of class The parameters represent the embryo development quality prediction model. Indicates the total number of categories. Indicates the number of samples. This represents the optimized model parameters.

[0013] Preferably, step S3 uses the trained model parameters. The embryo development quality prediction model is initialized, and then optimized using unsupervised entropy minimization to obtain the initially adjusted model parameters. : ; ; ; In the formula, This represents the sample to be predicted. The parameters represent the embryo development quality prediction model. Indicates the sample The predicted probability distribution express The Middle Predicted probability of class express entropy, Indicates the total number of categories. Indicates category index, This indicates the number of samples in each batch. This is the regularization coefficient.

[0014] Preferably, the formula for calculating the sample weights in step S3 is: ; In the formula, This indicates that the trained model parameters are used. For the sample Predicted entropy This indicates that the model parameters have been initially adjusted. For the sample Predicted entropy.

[0015] Preferably, the objective function for minimizing the unsupervised weighted entropy in step S3 is: ; ; In the formula, Represents weighted entropy. Indicates sample The weight, This represents the final model parameters after adjustment.

[0016] This invention also provides a multimodal fusion and adaptive adjustment embryo development quality prediction system, comprising: An embryo development quality prediction module includes an embryo image sequence branch and a tabular data branch. The embryo image sequence branch is used to perform dimensionality reduction processing on the embryo image sequence using principal component analysis, and performs dilation, erosion and gradient operations to extract morphological information from the spatial and temporal dimensions, respectively. Multi-channel multi-filter convolution and Mamba module are used to extract spatiotemporal features and global information. The tabular data branch is used to encode numerical features and category features respectively and use Mamba module to extract global information. The prediction result is output after fusing the features of the embryo image sequence branch and the tabular data branch. An adaptive adjustment module is used to receive embryo image sequences and tabular data to be evaluated, perform an initial adjustment to the embryo development quality prediction module using unsupervised entropy minimization, calculate sample weights based on the changes in prediction entropy before and after the initial adjustment, perform a second adjustment to the embryo development quality prediction module using unsupervised weighted entropy minimization, and output the final prediction result.

[0017] The beneficial effects of the present invention include at least the following: (1) To address the problem of insufficient information mining of embryo image sequence, this invention uses principal component analysis to remove redundant information of embryo image sequence while retaining the main feature information. It uses dilation, erosion and gradient operations combined with multi-channel multi-filter convolution to fully extract the morphological information and spatiotemporal information interaction features of the spatial and temporal dimensions of embryo image sequence. It uses the Mamba module to fully extract the global information of embryo image sequence, thereby effectively improving the classification accuracy of embryo image sequence branches.

[0018] (2) To address the problem that clinical tabular data is not fully integrated into embryo image sequences, which restricts model performance, this invention adopts a multimodal fusion strategy of embryo image sequences and tabular data. The tabular data branch divides the tabular data into numerical features and category features according to the data characteristics and encodes them separately. Then, the Mamba module is used to extract global information for the two types of features, which effectively improves the overall performance of embryo development quality prediction.

[0019] (3) To address the issue of inconsistent distribution of multimodal embryo image sequences and tabular data between the testing and training phases, this invention first uses unsupervised entropy minimization to make preliminary adjustments to the prediction model. Then, based on the changes in the prediction entropy of the samples before and after the adjustment, the samples are adaptively weighted. Finally, unsupervised weighted entropy minimization is used to further adjust the model, which effectively improves the prediction accuracy and robustness of the model in the distribution offset scenario. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall network architecture of the multimodal fusion and adaptive adjustment embryo development quality prediction method provided in the embodiments of the present invention. Detailed Implementation

[0021] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0022] like Figure 1As shown, this embodiment of the invention provides a method for predicting embryonic development quality based on multimodal fusion and adaptive adjustment. The method includes the following steps: Step S1: Construct an embryo development quality prediction model based on multimodal fusion.

[0023] The multimodal embryo development quality prediction model constructed in this invention includes two parallel processing branches: an embryo image sequence branch and a tabular data branch. The two branches extract and encode features from input data of different modalities, and finally fuse the features of the two branches to output the prediction result of embryo development quality.

[0024] Specifically, the processing flow for the embryo image sequence branch is as follows: First, due to the extremely high similarity between embryo images at the same developmental stage, the original embryo development video sequence contains a large amount of redundant information. Therefore, this invention first employs Principal Component Analysis (PCA) to perform dimensionality reduction on the embryo image sequence, effectively removing redundant information while retaining the main feature information of the embryo image sequence, thus reducing the computational complexity of subsequent processing.

[0025] Subsequently, this invention extracts morphological features from embryonic image sequences from both spatial and temporal dimensions. In the spatial dimension, three morphological operations—dilation, erosion, and gradient—are performed on each frame of the dimensionality-reduced embryonic image sequence to extract morphological structural and detail information. Dilation expands cell boundary regions, enhancing the overall structural features of the embryo; erosion shrinks boundary regions, highlighting the core morphological features; and gradient effectively extracts edge details. The results of dilation, erosion, and gradient operations are then subjected to multi-channel, multi-filter convolution processing to achieve interaction of spatial information from embryos at different developmental stages and extraction of multi-modal features. The results of the three convolution processes are then fused, comprehensively utilizing the morphological information from dilation, erosion, and gradient. Finally, the Mamba module is used to process the fused features, fully extracting the global information of the embryonic image sequence in the spatial dimension.

[0026] In the temporal dimension, this invention employs the same processing strategy for feature extraction from embryonic image sequences. Specifically, dilation, erosion, and gradient operations are performed along the time axis to extract morphological and detailed information at different stages of embryonic development, capturing the temporal evolution pattern of the embryo from fertilized egg to blastocyst. The results of the dilation, erosion, and gradient operations are then subjected to multi-channel, multi-filter convolution processing to achieve the interaction of embryonic development time information at different image locations and the extraction of multi-modal features. The results of the three convolution processes are then added and fused, and the Mamba module is used to fully extract the global information of the embryonic image sequence in the temporal dimension. The Mamba module, based on a state-space model, has the advantage of linear computational complexity compared to the quadratic computational complexity of the Transformer, making it more suitable for processing long sequences of embryonic development video data.

[0027] For the tabular data branch, this invention, based on the characteristics of clinical tabular data, divides the data into two categories: numerical features and categorical features, which are processed separately. Numerical features include continuous numerical variables such as patient age and number of retrieved oocytes, while categorical features include discrete categorical variables such as endometrial type. After targeted encoding of the numerical and categorical features, the Mamba module is used to extract global information for both types of features, capturing the complex nonlinear dependencies between features. Finally, the results of Mamba processing of the numerical and categorical features are fused and input into the classification head along with the features from the embryo image sequence branch to obtain the final embryo development quality prediction result.

[0028] Step S2: Obtain a training dataset containing embryo image sequences and tabular data, and train the embryo development quality prediction model by minimizing cross-entropy loss to obtain the trained model parameters.

[0029] This invention obtains a training dataset containing multiple pairs of embryo image sequences and corresponding tabular data. Each pair of training samples includes embryonic development video sequences, clinical tabular data, and corresponding embryo quality annotation information. A multimodal embryonic development quality prediction model is trained in a supervised manner by minimizing the cross-entropy loss between the model's predictions and the actual annotation information.

[0030] The formula for calculating cross-entropy loss is: ; In the formula, and They represent the first Embryo image sequences and tabular data for each sample, Indicates the first The sample belongs to the first The true label of the class, This indicates that the embryo development quality prediction model is effective for the first... The sample belongs to the first Predicted probability of class The parameters represent the embryo development quality prediction model. Indicates the total number of categories. Indicates the number of samples. This represents the optimized model parameters.

[0031] In this embodiment, the hyperparameters for the training process are set as follows: the learning rate is set to 10. -4 The batch size was set to 32, and the number of training rounds was set to 200.

[0032] Step S3: Input the embryo image sequences and tabular data to be evaluated into the trained embryo development quality prediction model in batches. Use unsupervised entropy minimization to make preliminary adjustments to the embryo development quality prediction model. Calculate the sample weights based on the changes in prediction entropy before and after the preliminary adjustments. Use unsupervised weighted entropy minimization to make further adjustments to the embryo development quality prediction model and output the final prediction results.

[0033] During the testing phase, due to differences in the autofocus performance of time-varying incubators and the complexity of the heterogeneous distribution of tabular data, the distribution of test data may deviate from that of training data. To address this issue, this invention employs an adaptive adjustment mechanism to optimize the model online during the testing phase.

[0034] First, the embryo image sequences and tabular data to be evaluated are input in batches into the trained embryo development quality prediction model, and the trained model parameters are used. The model is initialized, and then an unsupervised entropy minimization strategy is used to initially adjust the model. The objective function for unsupervised entropy minimization is: ; The entropy of the predicted output is defined as: ; The average output for each batch is defined as: ; In the formula, This represents the sample to be predicted. The parameters represent the embryo development quality prediction model. Indicates the total number of categories. Indicates category index, Indicates the sample The predicted probability distribution express The Middle Predicted probability of class express entropy, This indicates the number of samples in each batch. The regularization coefficient is . This represents the model parameters obtained after unsupervised entropy minimization optimization. The objective function reduces prediction uncertainty by minimizing the entropy of single-sample predictions, while preventing the model from collapsing into a single class by subtracting the entropy of the batch average output.

[0035] To further improve the accuracy of predicting incorrect samples after model adjustment, this invention introduces an adaptive weighting mechanism. Sample weights are calculated based on the changes in sample prediction entropy before and after the initial adjustment: ; In the formula, Indicates sample The weight, This indicates that the trained model parameters are used. Calculated samples Predicted entropy, This indicates that the model parameters have been initially adjusted. Calculated samples The predicted entropy. The core idea of ​​this weight design is: if the predicted entropy of a sample decreases after the initial adjustment (i.e., This indicates that the prediction confidence of this sample has increased, suggesting it may be a more reliable sample; therefore, a higher weight is assigned. Greater than 1; conversely, if the predicted entropy of a sample increases or remains unchanged after the initial adjustment, it indicates that the sample may contain noise or be a difficult sample, and therefore its weight should be reduced. Less than or equal to 1.

[0036] Finally, the model is further adjusted using unsupervised weighted entropy minimization, with the objective function being: ; The weighted entropy is defined as follows: ; In the formula, This represents the final model parameters obtained after unsupervised weighted entropy minimization optimization. Through an adaptive weighting mechanism, the model can automatically identify and strengthen the contribution of reliable samples while weakening the influence of noisy samples, thereby further improving prediction accuracy.

[0037] In summary, the adaptive adjustment process of this invention forms from arrive Then The parameter progressive optimization path: These are the initial model parameters obtained during the training phase. For The intermediate parameters are obtained after unsupervised entropy minimization optimization, starting from the given parameters. For The final model parameters are obtained after unsupervised weighted entropy minimization optimization, starting from the given parameters. The final model parameters are then used. It performs predictions and outputs the final prediction results of embryonic development quality.

[0038] To verify the effectiveness of the method of this invention, ablation experiments and comparative experiments were conducted using collected multimodal embryonic development data. The ablation experiment results showed that the multimodal fusion method of this invention achieved an accuracy of 88.7%. After removing the tabular data branch, the accuracy decreased to 86.5%, indicating that the introduction of tabular data can effectively improve the classification performance of the model. After adding an adaptive adjustment mechanism to the multimodal fusion, the accuracy further improved to 91.2%, demonstrating that adaptive adjustment can effectively address the problem of inconsistent distribution between test and training data.

[0039] Furthermore, to illustrate the advantages of the embryo image sequence processing network used in this invention, it was compared with commonly used CNNs and Transformers. The results show that the CNN method achieves an accuracy of 80.7%, and the Transformer method achieves an accuracy of 83.1%, both lower than the method of this invention. The experimental results fully demonstrate the superior performance of the multimodal fusion and adaptive adjustment method proposed in this invention on the task of predicting embryo development quality.

[0040] The present invention also provides an embryo development quality prediction system based on multimodal fusion and adaptive adjustment, the system comprising an embryo development quality prediction module and an adaptive adjustment module.

[0041] The embryo development quality prediction module comprises an embryo image sequence branch and a tabular data branch. The embryo image sequence branch uses principal component analysis to reduce the dimensionality of the embryo image sequence, performing dilation, erosion, and gradient operations to extract morphological information from both spatial and temporal dimensions. Multi-channel, multi-filter convolution and the Mamba module are then used to extract spatiotemporal features and global information. The tabular data branch encodes numerical and categorical features separately and uses the Mamba module to extract global information. The prediction result is output after feature fusion from both branches.

[0042] The adaptive adjustment module receives the embryo image sequence and tabular data to be evaluated, performs an initial adjustment to the embryo development quality prediction module using unsupervised entropy minimization, calculates the sample weights based on the changes in prediction entropy before and after the initial adjustment, and further adjusts the embryo development quality prediction module using unsupervised weighted entropy minimization, outputting the final prediction result.

[0043] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described; only preferred embodiments of the present invention are illustrated. The descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. As long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification.

[0044] It should be noted that those skilled in the art can make various modifications and improvements without departing from the inventive concept, and these all fall within the scope of protection of this invention. Therefore, the scope of protection of this invention should be determined by the appended claims.

Claims

1. A method for predicting embryonic development quality using multimodal fusion and adaptive adjustment, characterized in that, Includes the following steps: Step S1: Construct an embryo development quality prediction model based on multimodal fusion. The embryo development quality prediction model includes an embryo image sequence branch and a tabular data branch. The prediction result is output after fusing the features of the embryo image sequence branch and the tabular data branch. Step S2: Obtain a training dataset containing embryo image sequences and tabular data, and train the embryo development quality prediction model by minimizing cross-entropy loss to obtain the trained model parameters; Step S3: Input the embryo image sequence and table data to be evaluated in batches into the trained embryo development quality prediction model, perform preliminary adjustment of the embryo development quality prediction model by unsupervised entropy minimization, calculate the sample weights based on the changes in prediction entropy before and after the preliminary adjustment, perform further adjustment of the embryo development quality prediction model by unsupervised weighted entropy minimization, and output the final prediction result.

2. The method for predicting embryonic development quality according to claim 1, characterized in that, The embryo image sequence branch uses principal component analysis to reduce the dimensionality of the embryo image sequence, remove redundant information and retain the main feature information.

3. The method for predicting embryonic development quality according to claim 2, characterized in that, The embryo image sequence branch performs dilation, erosion, and gradient operations on the dimensionality-reduced embryo image sequence in the spatial dimension to extract morphological and detail information of the image. The results of the dilation, erosion, and gradient operations are then subjected to multi-channel, multi-filter convolution processing. The three convolution processing results are added and fused together, and the global information in the spatial dimension is extracted using the Mamba module.

4. The method for predicting embryonic development quality according to claim 3, characterized in that, The embryo image sequence branch performs dilation, erosion, and gradient operations on the dimensionality-reduced embryo image sequence from the time dimension to extract morphological and detailed information at different stages of embryonic development. The results of the dilation, erosion, and gradient operations are then subjected to multi-channel, multi-filter convolution processing. The three convolution processing results are added and fused together, and the global information in the time dimension is extracted using the Mamba module.

5. The method for predicting embryonic development quality according to claim 1, characterized in that, The tabular data branch divides the tabular data into numerical features and category features, and encodes them separately. The global information of the encoded numerical features and category features is extracted using the Mamba module.

6. The method for predicting embryonic development quality according to claim 1, characterized in that, The formula for calculating the cross-entropy loss in step S2 is as follows: ； In the formula, and They represent the first Embryo image sequences and tabular data for each sample, Indicates the first The sample belongs to the first The true label of the class, This indicates that the embryo development quality prediction model is effective for the first... The sample belongs to the first Predicted probability of class The parameters represent the embryo development quality prediction model. Indicates the total number of categories. Indicates the number of samples. This represents the optimized model parameters.

7. The method for predicting embryonic development quality according to claim 6, characterized in that, Step S3 uses the trained model parameters The embryo development quality prediction model is initialized, and then optimized using unsupervised entropy minimization to obtain the initially adjusted model parameters. : ； ； ； In the formula, This represents the sample to be predicted. The parameters represent the embryo development quality prediction model. Indicates the sample The predicted probability distribution express The Middle Predicted probability of class express entropy, Indicates the total number of categories. Indicates category index, This indicates the number of samples in each batch. This is the regularization coefficient.

8. The method for predicting embryonic development quality according to claim 7, characterized in that, The formula for calculating the sample weights in step S3 is as follows: ； In the formula, This indicates that the trained model parameters are used. For the sample Predicted entropy This indicates that the model parameters have been initially adjusted. For the sample Predicted entropy.

9. The method for predicting embryonic development quality according to claim 8, characterized in that, The objective function for minimizing the unsupervised weighted entropy in step S3 is: ； ； In the formula, Represents weighted entropy. Indicates sample The weight, This represents the final model parameters after adjustment.

10. A multimodal fusion and adaptive adjustment embryo development quality prediction system, characterized in that, include: An embryo development quality prediction module includes an embryo image sequence branch and a tabular data branch. The embryo image sequence branch is used to perform dimensionality reduction processing on the embryo image sequence using principal component analysis. It performs dilation, erosion, and gradient operations to extract morphological information from the spatial and temporal dimensions, respectively. The results of the dilation, erosion, and gradient operations are then subjected to multi-channel, multi-filter convolution processing. The three convolution processing results are added and fused. The Mamba module is used to extract global information from the spatial and temporal dimensions, respectively. The tabular data branch is used to encode numerical features and category features, respectively, and uses the Mamba module to extract global information. The prediction result is output after fusing the features from the embryo image sequence branch and the tabular data branch. An adaptive adjustment module is used to receive embryo image sequences and tabular data to be evaluated, perform an initial adjustment to the embryo development quality prediction module using unsupervised entropy minimization, calculate sample weights based on the changes in prediction entropy before and after the initial adjustment, perform a second adjustment to the embryo development quality prediction module using unsupervised weighted entropy minimization, and output the final prediction result.

Images