Embryo evaluation method and apparatus based on multi-modal data

Abstract

The application relates to an embryo evaluation method and device based on multi-modal data. The method for training a machine learning model comprises the following steps: acquiring image information of a training embryo sample and text information corresponding to the training embryo sample; acquiring image features of the image information; acquiring text features of the text information; acquiring matching degrees of the image features and the text features; and training the model based on the high matching degrees of the image features and the text features from the same training embryo sample, so as to obtain the machine learning model. The machine learning model can comprehensively utilize image and text information, more comprehensively evaluate embryo quality, improve evaluation accuracy and evaluation efficiency, and reduce artificial subjective error.

Description

Technical Field

[0001] This invention relates to the field of bioinformatics, specifically to a method and apparatus for embryo assessment based on multimodal data, and more specifically, to a method for training a machine learning model, an apparatus for training a machine learning model, a system for assessing embryo quality, a computing device, and a computer-readable storage medium. Background Technology

[0002] In vitro fertilization (IVF) is one of the most common methods for treating infertility. The process includes ovulation induction, egg retrieval, fertilization, embryo transfer, and implantation. However, this technology currently only achieves a pregnancy rate of approximately 35%. Embryo quality assessment is a crucial part of the IVF process. Doctors select the most viable embryos based on their morphology for implantation into the mother's uterus. However, subjective differences in how doctors score embryos often lead to variations in assessment results; furthermore, other physiological information, treatment plans, and related indicators such as hormone levels, follicle count and morphology, and endometrial thickness can also affect embryo quality and thus the final pregnancy outcome.

[0003] Currently, some companies and scholars both domestically and internationally have conducted research on embryo quality assessment based on artificial intelligence algorithms. Their methods mainly focus on using computer vision technology to extract features from static or time-series images of embryos and classify them. An embryo is a three-dimensional tissue containing multiple components. For example, when an embryo grows to the blastocyst stage on day five, it develops various structures such as the inner cell mass, trophoblast, and zona pellucida. Different structures will continue to evolve into fetuses or placenta, having different impacts on the final pregnancy outcome. Using only image information cannot refine the relationships and differences between these tissues, thus failing to accurately and effectively assess the growth of each tissue structure and quantitatively evaluate the final embryo quality. Embryo growth depends not only on in vitro culture but also on maternal physiological indicators, including the quantity and quality of follicles before and after ovulation induction, age, and hormone levels. However, to date, no machine learning model has been found that comprehensively assesses embryo quality using multimodal data.

[0004] Therefore, there is an urgent need in this field to develop a machine learning model for quantitative assessment of embryo quality based on multimodal data, in order to improve the efficiency and accuracy of embryo testing. Summary of the Invention

[0005] This application is made by the inventors based on their findings regarding the following problems and facts:

[0006] To address the problems of low efficiency, poor accuracy, and significant subjective variability in existing embryo quality assessment methods, the inventors have constructed a multimodal assessment model by extracting and combining multiple hyperparameters from image and text information. This model maximizes the use of image and text information for embryo quality assessment. Furthermore, the multimodal model employs a probability calculation method different from traditional classification models. This improvement allows the model to more fully utilize the extracted hyperparameter information, thereby enhancing retrieval accuracy and efficiency.

[0007] The present invention aims to at least partially solve one of the above-mentioned technical problems.

[0008] Therefore, in a first aspect, the present invention proposes a method for training a machine learning model for evaluating embryo quality. According to an embodiment of the present invention, the machine learning model includes an image feature extractor sub-model, a text feature extractor sub-model, and a matching degree prediction sub-model. The method includes: acquiring image information of training embryo samples and text information corresponding to the training embryo samples; inputting the image information into the image feature extractor sub-model to obtain image features of the training embryo samples; inputting the text information into the text feature extractor sub-model to obtain text features of the training embryo samples; inputting the image features and the text features into the matching degree prediction sub-model, the matching degree prediction model outputting the matching degree of the image features and the text features; and training the image feature extractor sub-model, the text feature extractor sub-model, and the matching degree prediction sub-model based on the high matching degree of the image features and the text features from the same training embryo samples to obtain the machine learning model.

[0009] The machine learning model for embryo quality assessment trained using the above method can comprehensively utilize image and text information to assess embryo quality more comprehensively and improve assessment accuracy. In addition, the automated assessment method of this machine learning model reduces human subjective error and improves assessment efficiency. Since this machine learning model is trained on a large amount of training data, it can provide highly accurate embryo quality assessment results.

[0010] It should be noted that in this application, the image feature extractor sub-model uses the Swin-Tiny model as the image encoder. Through a series of hierarchical operations such as convolutional layers and pooling layers, it extracts features from the original image and represents them as a series of feature maps. These feature maps can be used for tasks such as image classification and object detection.

[0011] It should be noted that the text feature extractor sub-model refers to a machine learning model used to extract useful features from text data. These models can convert text data into numerical vectors for use in various machine learning tasks, such as classification, clustering, and information retrieval. In this application, the text feature extractor model is selected to train the text dataset to learn the most effective feature extraction method.

[0012] It should be noted that the matching degree prediction sub-model is based on a deep learning method to predict the similarity between the image dataset and the text dataset. By employing supervised learning, it learns the similarity between the image set and the text set using the training dataset, and then uses this learning to predict new similarity values.

[0013] According to embodiments of the present invention, the method for training a machine learning model described above may further include at least one of the following technical features:

[0014] According to an embodiment of the present invention, the machine learning model further includes an embryo quality assessment sub-model, which is used to generate embryo quality assessment results. By automatically assessing embryo morphology, cell number, cell symmetry, and other characteristics through computer vision technology and artificial intelligence algorithms, the burden of manual assessment is reduced, and accuracy and reliability are improved.

[0015] According to an embodiment of the present invention, the quality assessment result is selected from at least one of graphic description, assessment score, assessment level, and regional assessment. Specifically, the quality assessment result includes:

[0016] 1) Graphic description: The quality information of the embryo is described by combining text and images, thereby simplifying the analysis process.

[0017] 2) Assessment level: The quality of embryos is assessed by classifying them into levels such as excellent, good, and average.

[0018] 3) Assessment score: The quality of the embryo is scored, and a score above a certain threshold is defined as a high-quality embryo.

[0019] 4) Regional assessment: By assessing different locations or stages of embryo images, the quality of the embryo at a specific time point can be further determined, which helps doctors analyze the reasons for the excellent quality of the embryo.

[0020] It should be noted that different assessment methods can yield different quality assessment results. Therefore, when conducting quality assessments, it is necessary to select an appropriate assessment method and analyze and judge it in conjunction with the specific circumstances.

[0021] According to an embodiment of the present invention, the pre-trained model for the image and text information is selected from the CLIP model. Leveraging the advantage of the CLIP model's ability to process both text and images simultaneously, this embodiment of the invention selects the CLIP model for large-scale training on embryonic images and text information. Based on the versatility of the CLIP model, different types of images and text can be selectively input for training, thereby enhancing the model's robustness. This ensures accurate results even when inputting special images.

[0022] According to an embodiment of the present invention, the image feature extractor sub-model is selected from the Swin-Tiny model. The inventors chose the Swin-Tiny model based on its advantages of lightweight design, strong generalization performance, and high training efficiency.

[0023] According to an embodiment of the present invention, the image feature extractor sub-model is obtained through supervised training based on the Swin-Tiny model, using at least one of the following information from the image: basal follicles, post-ovulation induction follicles, pre-implantation endometrium, oocytes, cleavage stage, and blastocyst stage as feature parameters. According to an embodiment of the present invention, the image feature extractor sub-model developed based on the Swin-Tiny model extracts information such as basal follicles, post-ovulation induction follicles, pre-implantation endometrium, oocytes, cleavage stage, and blastocyst stage from the input image, and uses these information as feature parameters for supervised training. By learning these feature parameters, it identifies and classifies different types of follicles and endometrium, thereby helping doctors better assess embryo quality.

[0024] According to an embodiment of the present invention, the text feature extractor sub-model is selected from the text Transformer model. The text Transformer model, as a type of deep learning model, is widely used in natural language processing. In this embodiment, the text Transformer model is selected as the model for processing text information because the Transformer employs a multi-head self-attention mechanism, which can effectively model long-term dependencies and better handle long texts. Furthermore, the Transformer can perform fully parallel computations, processing data faster during both training and inference, reducing training time costs and resources.

[0025] According to an embodiment of the present invention, the text feature extractor sub-model is trained under supervision based on a text Transformer model, using at least one of the following as feature parameters: physiological information, hormone information, medication information, follicle information, and embryonic development information in the text. Specifically, the sub-model is trained using previously labeled relevant text datasets to learn how to extract effective feature parameters related to physiological, hormone, medication, follicle, and embryonic development information, thereby better processing and analyzing text data in subsequent tasks.

[0026] According to an embodiment of the present invention, the text features of the machine learning model are selected from patients.

[0027] According to an embodiment of the present invention, the matching degree prediction sub-model is trained based on the similarity between image features and text features. The training of this sub-model is based on the similarity between image features and text features; that is, the matching degree between two feature vectors is determined by comparing their similarity. It is trained using a known matching degree dataset to predict the similarity between two unknown images or text, thereby determining their matching degree.

[0028] According to an embodiment of the present invention, the matching degree corresponds to the similarity between image features and text features.

[0029] For example, in the embryo quality assessment model trained in this application, images and text information are input, and the quality of the embryo is predicted by analyzing the input images and text information. When outputting results, the user can choose to output the top N (N is a positive integer, such as 1, 2, 3, 4, 5, etc.) similar images and corresponding text information based on their similarity ranking. It should be noted that the model can also modify the output results to embryo quality scores, embryo quality ratings, or embryo quality assessments based on continuous time-point images. These different output formats provide more options to suit different clinical needs. Furthermore, users can also choose to combine the output images and corresponding text information with embryo quality assessment scores, ratings, and continuous time-point images for evaluation.

[0030] According to an embodiment of the present invention, the image information further includes an ultrasound image.

[0031] It should be noted that the image information described in this application includes, but is not limited to, ultrasound images.

[0032] According to an embodiment of the present invention, the ultrasound image includes B-mode ultrasound.

[0033] According to embodiments of the present invention, the image information of the machine learning model is selected from at least one of a single image, a series of images, or different regions within an image under a microscope. According to embodiments of the present invention, the image information of the machine learning model is trained using a single RGB image, a series of RGB images, and a portion of a single RGB image under a microscope, in order to make a more comprehensive and accurate quality assessment. By analyzing the growth and development status of the embryo throughout its entire lifecycle, the generated assessment report has a higher rating accuracy rate and can assist reproductive physicians in selective embryo transfer in practical clinical applications.

[0034] In a second aspect, the present invention provides an apparatus for training a machine learning model. According to an embodiment of the present invention, the apparatus is used to assess embryo quality, and the apparatus includes: an information acquisition module for acquiring image information of training embryo samples and text information corresponding to the training embryo samples; a feature acquisition module for acquiring image features and text features of the training embryo samples; a matching degree prediction module for acquiring the matching degree between the image features and the text features; and a training module for training the machine learning model based on a high matching degree between the image features and the text features from the same training embryo samples; wherein the machine learning model is trained using the method described in the first aspect of the present invention.

[0035] The device described in the embodiments of the present invention has the following advantages:

[0036] 1) Improve the accuracy of embryo quality assessment: The aforementioned device can simultaneously acquire image and text information of the embryo, and perform feature extraction and matching degree prediction. Based on large-scale training data, it can significantly increase the accuracy of embryo quality assessment.

[0037] 2) High degree of automation: The aforementioned device can automatically complete the embryo quality assessment process, saving labor costs (and avoiding subjective errors of observers) and time costs.

[0038] 3) Support for training machine learning models: The aforementioned device can also train machine learning models on datasets based on the acquired image and text features, thereby increasing the robustness of the models.

[0039] 4) Wide range of applications: The aforementioned device can be applied to different types of embryo quality assessment tasks and has versatility.

[0040] According to embodiments of the present invention, the apparatus for training machine learning models described above may further include at least one of the following technical features:

[0041] According to an embodiment of the present invention, the image information further includes an ultrasound image.

[0042] According to an embodiment of the present invention, the ultrasound image includes B-mode ultrasound.

[0043] In a third aspect, the present invention proposes a system for assessing embryo quality. According to an embodiment of the invention, the system includes: an information acquisition unit for acquiring image information of training embryo samples and corresponding text information of the training embryo samples; a feature acquisition unit for acquiring image features of the training embryo samples and text features of the training embryo samples; a matching degree prediction unit for acquiring the matching degree of the image features and the text features; and an embryo assessment unit for assessing embryo quality based on a machine learning model. The machine learning model is obtained by training using the method described in the first aspect of the invention. According to an embodiment of the invention, the system has the advantage of automatically assessing embryo quality, reducing the influence of subjective human factors. Furthermore, the system uses a machine learning model for assessment, which can more accurately predict the developmental capacity and health status of the embryo, improving the embryo transfer success rate, and also helping to reduce unnecessary medical expenses and time costs.

[0044] According to embodiments of the present invention, the above-described system for assessing embryo quality may further include at least one of the following technical features:

[0045] According to an embodiment of the present invention, the image information further includes an ultrasound image.

[0046] According to an embodiment of the present invention, the ultrasound image includes B-mode ultrasound.

[0047] According to an embodiment of the present invention, the quality assessment result is selected from at least one of the following: graphic description, assessment score, assessment level, and regional assessment matching degree.

[0048] In a fourth aspect, the present invention provides a computing device. According to an embodiment of the invention, the computing device includes: a processor and a memory; the memory for storing a computer program; and the processor for executing the computer program to implement the method described in the first aspect of the invention. According to an embodiment of the invention, the processor runs a program corresponding to the executable program code stored in the memory to train a machine learning model. The device can execute computer programs quickly, making it more efficient. Furthermore, the device can store various types of computer programs as needed, enhancing flexibility in processing computer programs. More importantly, compared to other high-end computing devices, this computing device may be much less expensive because it is implemented using a combination of common hardware components and open-source software.

[0049] It should be noted that the device described in this application is scalable. For example, in practical applications, if a larger storage space or more powerful processing capability is required, the computing device can be expanded by adding additional memory or processors; it can also support multi-task parallel processing, thereby improving computing efficiency.

[0050] In a fifth aspect, the present invention provides a computer-readable storage medium. According to an embodiment of the invention, the storage medium includes computer instructions that, when executed by a computer, cause the computer to implement the method described in the first aspect of the invention. The computer program contained in the computer-readable storage medium can more accurately process complex information contained in embryo images and text information, and based on extensive prior training, can provide a more accurate result. Furthermore, the computer-readable storage medium can store a large amount of data, avoiding human error and ensuring the accuracy and reliability of the data. Moreover, it can be accessed anytime and anywhere to facilitate the retrieval of user test records from different times, etc.

[0051] It should be noted that, in this application, the logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, a computer-readable medium can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory. The various computer-readable storage media described in this invention can represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" can include, but is not limited to, wireless channels and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0052] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0053] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0054] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0055] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0056] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0057] Figure 1 This is a diagram of a machine learning model training apparatus according to an embodiment of the present invention;

[0058] Figure 2 This is a flowchart of the embryo quality assessment system according to an embodiment of the present invention;

[0059] Figure 3 This is the embryo evaluation method flow described in the embodiments of the present invention;

[0060] Figure 4 This is a structural diagram of the multimodal learning network model according to an embodiment of the present invention;

[0061] Figure 5This is a confusion matrix diagram of the classification, image retrieval, and image-text retrieval results according to Embodiment 1 of the present invention;

[0062] Figure 6 This is the similarity result between each word and the local image according to Embodiment 1 of the present invention; wherein, the darker the color, the higher the similarity. Detailed Implementation

[0063] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0064] Definitions and Explanations

[0065] To facilitate understanding of the invention, certain technical and scientific terms are specifically defined below. Unless otherwise expressly defined elsewhere in this document, all other technical and scientific terms used herein have the meanings commonly understood by one of ordinary skill in the art to which this invention pertains.

[0066] In this document, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0067] In this document, the terms “comprising” or “including” are open-ended expressions, meaning that they include the contents specified in this invention, but do not exclude other aspects.

[0068] In this document, the terms “optionally,” “optionally,” or “optionally” generally refer to an event or condition that may, but may not, occur, and the description includes both cases in which the event or condition occurs and cases in which the event or condition does not occur.

[0069] In this application, the term "multimodal data" includes data composed of multiple information formats, such as images, text, sound, and other different types of data. In this document, the multimodal embryo assessment model is a machine learning model based on artificial intelligence algorithms that analyze, process, and fuse the aforementioned different types of data (embryo image information, text information, etc.) to achieve a more accurate and comprehensive assessment of embryo quality.

[0070] Methods for training machine learning models

[0071] According to an embodiment of the present invention, the machine learning model is used for embryo quality assessment. The machine learning model includes an image feature extractor sub-model, a text feature extractor sub-model, and a matching degree prediction sub-model. The method includes: acquiring image information of training embryo samples and text information corresponding to the training embryo samples; inputting the image information into the image feature extractor sub-model to obtain image features of the training embryo samples; inputting the text information into the text feature extractor sub-model to obtain text features of the training embryo samples; inputting the image features and the text features into the matching degree prediction sub-model, wherein the matching degree prediction model outputs the matching degree of the image features and the text features; and training the image feature extractor sub-model, the text feature extractor sub-model, and the matching degree prediction sub-model based on the high matching degree of the image features and the text features from the same training embryo samples to obtain the machine learning model.

[0072] Specifically, for ease of understanding, the technical solution of this application will be explained and described in detail below:

[0073] In this application, the machine learning model training process specifically includes:

[0074] 1) Image preprocessing: Image preprocessing aims to enhance contrast and eliminate noise interference to improve image quality and accuracy.

[0075] For example, in a multi-embryo culture mode, the original RGB embryo image is first converted to a grayscale image, and then the grayscale image is binarized. The rectangular region of the outer boundary of the blastocyst is located using the OpenCV algorithm, and the rectangular region is cropped from the original RGB blastocyst image. The background image is then cropped based on the rectangular binary mask image. Finally, to load batch images during network training, this application adjusts the pixel size of the ultrasound and embryo images to a fixed pixel size by padding, thereby maintaining the aspect ratio of the images.

[0076] It's important to note that the term "binarization" is a fundamental preprocessing technique in digital image processing, used to convert grayscale images into binary images. In a binary image, each pixel has only two possible values, typically 0 and 1, but other polarities are also possible. Specifically, grayscale values ​​are usually divided into two categories by comparing them to a threshold, setting pixels with values ​​greater than the threshold to 1, and pixels with values ​​less than or equal to the threshold to 0. In a binary image, there are clear distinctions between adjacent pixel values, and the contours and features are more defined, allowing for better image analysis and processing.

[0077] 2) Text paradigm processing: In order to recognize the text in the image, the text must be paradigm processed.

[0078] This invention designs a text paradigm generation system specifically for embryo-related tasks. It integrates patient and related embryo information into ultrasound and embryo image data to generate an aligned paradigm. For example, for ovarian ultrasound images from ovulation induction, it provides the number and size of follicles; for embryo microscopic images, it provides information such as shooting conditions, growth time, cell cluster location and morphology, and the presence of fragmented vacuoles. This textual information provides rich self-supervised information for the corresponding images.

[0079] 3) Extraction of global and local features of text in images.

[0080] This invention utilizes image feature extractors and text feature extractors to extract global and local features of images and text, enabling the learning of image and text representations within the same multimodal feature space. The pre-trained models for both the image and text feature extractors employ a vision-language joint learning modality model, such as CLIP (Contrastive Language-Image Pre-Training), to learn alignment representations between text and images through pre-training on large-scale data. These alignment representations are then used to fine-tune the embryonic task.

[0081] Specifically, such as Figure 3 As shown, the image input includes ultrasound images of the patient during the basal follicular phase and before embryo transfer during the stimulation phase, obtained through ultrasound, and time-series images of embryonic growth and development obtained through optical microscopy. The text input includes patient physiological information, hormone information, medication information, follicular information, and embryonic development information.

[0082] In the image feature extractor sub-model, the Swin-Tiny Transformer is selected to extract features from the image. The image is segmented into a sequence of image patches to accommodate the model's multi-head attention mechanism. Each image patch obtains local features through the feature extractor, and the average of the embedding representations of all image patches is used to obtain the global image features.

[0083] In the text feature extractor sub-model, the text Transformer model is used as the text feature extractor to obtain robust global text representations and local text representations (such as the entire text, words, or sentences).

[0084] During the architecture training phase, text feature extractors and image feature extractors are used to extract text description features and image features respectively. The similarity between text and image features is used as the score of the current category, and then cross-entropy loss is used to optimize the model.

[0085] 4) Embryo quality assessment: This involves using classification models, regression models, reinforcement learning, or retrieval subtasks to assess embryo quality. Depending on the type of data and the purpose, for example, classification models might be suitable for ranking embryos from poor to good, while regression models might be used to predict embryo quality scores.

[0086] Downstream subtasks include image and text retrieval, quantitative assessment, classification, and segmentation, tailored to the needs of doctors. The differences between these subtasks are as follows:

[0087] a. Image and text retrieval: Similar to an X-ray examination, the input image generates a report using an AI algorithm. The prediction model takes the input image and / or text information such as physiological blood values, calculates the similarity between image features and all text description features, selects the category corresponding to the text with the highest feature similarity as the model's prediction result, and ultimately achieves a text description of the embryo.

[0088] b. Quantitative Assessment Task: The predictive model is input with relevant patient test images and physiological blood values, etc. The model, along with a regression head, uses a regression task to quantitatively predict embryo quality, such as on a percentage scale. Different embryos from different patients will have different predicted scores. Based on the prediction results, doctors can determine whether to proceed with transfer or the next cycle of treatment.

[0089] c. Classification task: Similar to the regression task, the prediction model is input with text information such as relevant patient test images and physiological blood values, and uses a classification task to rate the embryo, such as level three. Doctors can then decide on embryo transfer or treatment for the next cycle based on the embryo score.

[0090] d. Recognition and segmentation task: The prediction model is input with relevant patient test images and physiological blood values ​​and other text information to mark the corresponding regions of the images and embryo morphology descriptions, such as the number of blastomeres during the cleavage stage, the morphology of the cell mass and trophoblast during the blastocyst stage, and the location and number of fragment cavities, to help doctors quantify and refine the assessment of embryo quality.

[0091] According to embodiments of the present invention, the above method is used to purchase a multimodal data embryo assessment model to achieve embryo rating, thereby helping doctors to perform IVF treatment more accurately and improve the success rate of their treatment.

[0092] Device for training machine learning models

[0093] According to an embodiment of the present invention, the device is used to assess embryo quality.

[0094] like Figure 1As shown, the device includes: an information acquisition module S100, used to acquire image information of training embryo samples and text information corresponding to the training embryo samples; a feature acquisition module S200, used to acquire image features of training embryo samples and text features of the training embryo samples; a matching degree prediction module S300, used to acquire the matching degree of the image features and the text features; and a training module S400, used to train the machine learning model based on the high matching degree of the image features and the text features from the same training embryo samples.

[0095] It should be noted that the information acquisition module S100 is connected to the feature acquisition module S200, the feature acquisition module S200 is connected to the matching degree prediction module S300, and the matching degree prediction module S300 is connected to the training module S400.

[0096] System for assessing embryo quality

[0097] According to an embodiment of the present invention, the system is used to assess embryo quality.

[0098] like Figure 2 As shown, the information acquisition unit S500 is used to acquire image information of training embryo samples and text information corresponding to the training embryo samples; the feature acquisition unit S600 is used to acquire image features of training embryo samples and text features of training embryo samples; the matching degree prediction unit S700 is used to acquire the matching degree of the image features and the text features; and the embryo evaluation unit S800 is used to evaluate embryo quality based on a machine learning model.

[0099] It should be noted that the information acquisition unit S500 is connected to the feature acquisition unit S600, the feature acquisition unit S600 is connected to the matching degree prediction unit S700, and the matching degree prediction unit S700 is connected to the embryo evaluation unit S800.

[0100] computing devices

[0101] According to an embodiment of the present invention, the computing device includes: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program to implement the aforementioned method for training a machine learning model.

[0102] Computer-readable storage media

[0103] According to an embodiment of the present invention, the storage medium includes computer instructions that, when executed by a computer, cause the computer to implement the aforementioned method for training a machine learning model.

[0104] Embodiments of the present invention will now be described in more detail, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the invention, and should not be construed as limiting the invention.

[0105] The embodiments will be described in detail below. Where specific techniques or conditions are not specified in the embodiments, they shall be performed in accordance with the techniques or conditions described in the literature in this field or in accordance with the product manual.

[0106] Example 1: Validation of the Embryo Quality Assessment Model

[0107] The dataset used for training in this invention comes from a top-tier hospital in Shanghai. This invention constructs an image-text multimodal model based on blastocyst images and textual information (physiological information, hormone information, medication information, follicle information, embryonic development information, etc.) to evaluate embryo quality. The specific steps are as follows:

[0108] 1) Image data acquisition

[0109] The dataset contains 395 static RGB images of blastocysts under a microscope (without age or other physiological selectivity). The blastocyst images were taken 115–120 hours (day 5) or 135–140 hours (day 6) after in vitro fertilization using a standard inverted optical microscope system (model: Nikon Eclipse Ti-U). These images may contain variations in viewing angle, zoom level, color, contrast, etc.

[0110] 2) Text Data Acquisition

[0111] In this embodiment, the embryo quality assessment model is trained using static images of blastocysts on day 5 or 6 and the corresponding graded text for their inner cell mass.

[0112] 3) Model parameter tuning

[0113] To improve the blastocyst quality assessment performance of the multimodal network, this invention trained the model for 300 iterations using the Adam optimizer (decay rate: 5e-4, momentum: 0.95). Furthermore, an early stopping strategy was applied to prevent overfitting. The batch size was 32, the learning rate for the image encoder was 7.5e-6, the learning rate for the text encoder was 7.5e-7, and the temperature coefficient was 1.0 for both.

[0114] This invention redesigns text labels for the inner cell mass of the blastocyst, replacing traditional letter ratings, based on text descriptions from the Gardner rating system and the Time-lapse system, as shown in Table 1. These text labels provide more information to help assess blastocyst quality. By combining image data and text labels, this invention successfully constructs an image-text-based multimodal model to more comprehensively and accurately assess blastocyst quality.

[0115] Table 1: Text labels replace corresponding letter ratings

[0116]

[0117] 4) Model performance verification

[0118] To verify the effectiveness of the model, its performance was evaluated using four metrics: accuracy, precision, recall, and F1-score. In all experiments, 5-fold cross-validation was performed to increase the model's robustness. Compared to unimodal networks, the text description of the blastocyst includes more semantic information. Furthermore, the performance of different image encoders used in the classification experiments was compared. Experimental results demonstrate the effectiveness of the proposed multimodal method.

[0119] 5) Results Analysis

[0120] The results are shown in Table 2. The multimodal Swin-Tiny model outperformed the best single-modal classification model (+1.5%) and image retrieval model (+1.2%).

[0121] Table 2: Model Performance Comparison Results

[0122]

[0123]

[0124] To further analyze the model's scoring ability, this invention examined the recall rate for each level. For example... Figure 3 As shown in the figure, this model achieved the best performance in predicting grades A and C compared to the other two models. In clinical practice, grade A and C blastocysts are crucial for embryologists when selecting blastocysts. Furthermore, the model exhibited the fewest cross-grade prediction errors, i.e., a blastocyst labeled A but predicted as C, or vice versa. These findings suggest that the model can utilize textual information about blastocysts for more accurate blastocyst evaluation.

[0125] This invention also evaluates the effectiveness of the text word by word by calculating the similarity between each word and the patch image embedding. For example... Figure 4As shown, the words most relevant to the image patch within each level are sorted from smallest to largest by their maximum similarity. Words with higher similarity, such as vacuoles in level C, play a more crucial role.

[0126] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0127] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A method for training a machine learning model, said machine learning model being used to assess embryo quality, characterized in that, The machine learning model includes an image feature extractor sub-model, a text feature extractor sub-model, and a matching degree prediction sub-model; the method includes: Acquire image information of training embryo samples and text information corresponding to the training embryo samples; The image information is input into the image feature extractor sub-model to obtain the image features of the training embryo samples; and The text information is input into the text feature extractor sub-model in order to obtain the text features of the training embryo samples; The image features and text features are input into the matching degree prediction sub-model, and the matching degree prediction model outputs the matching degree between the image features and the text features; and Based on the high matching degree of the image features and text features from the same training embryo samples, the image feature extractor sub-model, the text feature extractor sub-model, and the matching degree prediction sub-model are trained to obtain the machine learning model; The image feature extractor sub-model is selected from the Swin-Tiny model; The image feature extractor sub-model is obtained by supervised training based on the Swin-Tiny model, using at least one of the following as feature parameters: basal follicles, post-ovulation follicles, pre-implantation endometrium, oocytes, cleavage stage, and blastocyst stage information in the image.

2. The method according to claim 1, characterized in that, The machine learning model further includes an embryo quality assessment sub-model, which is used to generate embryo quality assessment results.

3. The method according to claim 2, characterized in that, The quality assessment results are selected from at least one of the following: graphic description, assessment score, assessment level, and regional assessment.

4. The method according to claim 1, characterized in that, The pre-trained models for the image and text information are selected from the CLIP model.

5. The method according to claim 1, characterized in that, The text feature extractor sub-model is selected from the text Transformer model.

6. The method according to claim 5, characterized in that, The text feature extractor sub-model is obtained by supervised training based on the text Transformer model, using at least one of the physiological information, hormone information, medication information, follicle information, and embryonic development information in the text as feature parameters.

7. The method according to claim 6, characterized in that, The text features of the machine learning model are selected from patients.

8. The method according to claim 1, characterized in that, The matching degree prediction sub-model is trained based on the similarity between image features and text features.

9. The method according to claim 8, characterized in that, The matching degree corresponds to the similarity between image features and text features.

10. The method according to claim 1, characterized in that, The image information further includes ultrasound images.

11. The method according to claim 10, characterized in that, The ultrasound images include B-mode ultrasound.

12. The method according to claim 1, characterized in that, The image information of the machine learning model is selected from at least one of a single image or a series of images or different regions within an image taken under a microscope.

13. An apparatus for training a machine learning model, characterized in that, The device is used to assess embryo quality, and the device includes: An information acquisition module is used to acquire image information of training embryo samples and text information corresponding to the training embryo samples. A feature acquisition module is used to acquire image features and text features of training embryo samples; A matching degree prediction module is used to obtain the matching degree between the image features and the text features; A training module that trains the machine learning model based on a high degree of matching between the image features and the text features from the same training embryo samples; The machine learning model is trained using the method described in any one of claims 1 to 12.

14. The apparatus according to claim 13, characterized in that, The image information further includes ultrasound images.

15. The apparatus according to claim 14, characterized in that, The ultrasound images include B-mode ultrasound.

16. A system for assessing embryo quality, characterized in that, include: An information acquisition unit is used to acquire image information of training embryo samples and text information corresponding to the training embryo samples; The feature acquisition unit is used to acquire the image features and text features of the training embryo samples; A matching degree prediction unit is used to obtain the matching degree between the image features and the text features; Embryo assessment unit, used to assess embryo quality based on machine learning models; The machine learning model is obtained by training using the method described in any one of claims 1 to 12.

17. The system according to claim 16, characterized in that, The image information further includes ultrasound images.

18. The system according to claim 17, characterized in that, The ultrasound images include B-mode ultrasound.

19. The system according to claim 16, characterized in that, The quality assessment is selected from at least one of the following: graphic description, assessment score, assessment level, and regional assessment.

20. A computing device, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program to implement the method as described in any one of claims 1 to 12.

21. A computer-readable storage medium, characterized in that, The storage medium includes computer instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1 to 12.

Images