Automated evaluation of quality assurance metrics for assisted reproductive procedures

Abstract

Systems and methods for assigning quality parameters to germ cell structures are provided. An image of the germ cell structure is obtained. The image of the germ cell structure is provided to a neural network to generate a value representing the morphology of the germ cell structure. The value is compared to a predetermined standard to provide a quality assurance metric representing one of a medical person, an institution, a growth medium, and an identity of the germ cell structure.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Patent Application Serial No. 62 / 897,043, filed on September 6, 2019, entitled “DEEP NEURAL NETWORK-ENABLED EMBRYOTRACKING AND SPECIMEN IDENTIFICATION SYSTEM FOR ERROR MINIMIZATION IN IVF PRACTICES” and U.S. Provisional Patent Application Serial No. 62 / 897,043, filed on September 6, 2019, entitled “AUTOMATED QUALITY ASSESSMENT OF INDIVIDUAL EMBRYOLOGISTS PERFORMING ICSI USING DEEP LEARNING-ENABLED FERTILIZATION AND EMBRYO GRADING No. 62 / 897,045, filed on September 6, 2019, entitled “MONITORING OF HUMAN EMBRYO CULTURE CONDITIONS USING A DEEP LEARNING-DERIVED KEY PERFORMANCE INDICATOR (KPI)”; and No. 62 / 897,053, filed on September 6, 2019, entitled “DEEP LEARNING-ENABLED PREDICTION OF FERTILIZATION BASED ON OOCYTE MORPHOLOGICAL QUALITY.” The entire contents of each of these applications are incorporated herein by reference in their entirety. Technical Field

[0003] The present invention relates generally to the field of medical decision support, and in particular to automated assessment of quality assurance metrics for assisted reproductive procedures. Background Art

[0004] Infertility is an underestimated healthcare problem, affecting more than 48 million couples worldwide and is a cause of pain, depression and discrimination. Although assisted reproductive technologies (ART) (such as in vitro fertilization (IVF)) have alleviated the burden of infertility to a certain extent, their efficiency is very low, with an average success rate of 26% reported in the United States in 2015. IVF remains an expensive solution, with costs between $7,000 and $20,000 per ART cycle in the United States, which is usually not covered by insurance. Moreover, many patients require multiple IVF cycles to achieve pregnancy.

[0005] Data analysis is a key part of an effective quality assessment (QA) program for assisted reproductive procedures. Routine review of identified key performance indicators (KPIs) is important to ensure laboratory functioning and, perhaps more importantly, to detect potential problems so they can be corrected promptly. Fertilization assessment is the primary outcome used to measure the efficiency of intracytoplasmic sperm injection (ICSI) embryologists. However, tracking the developmental fate of ICSI-derived embryos can provide a more complete picture of how the procedure is performed. Summary of the invention

[0006] According to one aspect of the present invention, a method for assigning a quality parameter to a germ cell structure is provided. An image of the germ cell structure is obtained. The image of the germ cell structure is provided to a neural network to generate a value representing the morphology of the germ cell structure. The value is compared to a predetermined standard to provide a quality assurance metric representing one of a medical person, an institution, a growth medium, and an identity of the germ cell structure.

[0007] According to another aspect of the invention, a system includes a processor, an output device, and a non-transitory computer readable medium storing machine executable instructions for assigning quality assurance metrics to embryos. The machine executable instructions include: an imager interface that receives an image of the embryo from an associated imager; and a convolutional neural network that determines a value representing the likelihood of a successful outcome for the embryo based on the image of the embryo. A quality analysis component calculates a value representing the performance of one of a medical professional, an institution, and a growth medium on a plurality of embryos, and compares the value to a threshold value to generate a quality assurance metric. A user interface for displaying the quality assurance metric to a user at the output device.

[0008] According to another aspect of the present invention, a system includes a processor, an output device, and a non-transitory computer readable medium storing machine executable instructions for assigning a value representing the quality of an oocyte. The machine executable instructions include: an imager interface that receives an image of an oocyte from an associated imager; and a convolutional neural network that determines a value representing one of a probability of successful fertilization of the oocyte and a position of a polar body of the oocyte based on the image of the oocyte. A user interface displays the value representing the probability of successful fertilization of the oocyte to a user at the output device.

[0009] According to another aspect of the invention, a system includes a processor, an output device, and a non-transitory computer-readable medium storing machine-executable instructions for confirming the identity of a germ cell structure. The machine-executable instructions include an imager interface that receives, from an associated imager, a first image of a germ cell structure taken at a first time and a second image of the germ cell structure taken at a second time. A neural network generates an identifier keyword based on the first image of the germ cell structure and generates a value representing the morphology of the germ cell structure based on the second image. An identity verification component compares the value representing the morphology of the germ cell structure with the identifier keyword to determine whether the germ cell structure belongs to a patient associated with the identifier keyword. A user interface displays to a user at the output device the determination of whether the germ cell structure belongs to the patient associated with the identifier keyword. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The foregoing and other features of the present invention will become apparent to those skilled in the art to which the present invention relates by reading the following description made with reference to the accompanying drawings, in which:

[0011] Figure 1 A system for assigning quality assurance metrics to germ cell structures is shown;

[0012] Figure 2 Example implementations of systems for assigning quality assurance metrics to embryos are provided;

[0013] Figure 3 Example implementations of systems for assigning quality metrics to oocytes are provided;

[0014] Figure 4 An example implementation of a system for verifying the identity of a germ cell structure is shown;

[0015] Figure 5 A method for assigning quality assurance metrics to germ cell structures is shown;

[0016] Figure 6 A method for assigning quality assurance metrics to embryos is shown;

[0017] Figure 7 A method for assigning a value representing the quality of an oocyte is shown;

[0018] Figure 8 Methods for verifying the identity of germ cell structures are shown; and

[0019] Fig. 9 is a schematic block diagram of an exemplary system illustrating hardware components capable of implementing examples of the systems and methods disclosed herein. DETAILED DESCRIPTION

[0020] As used herein, "germ cell structure" refers to an oocyte before or after fertilization. Thus, the term is intended to encompass both oocytes and embryos at any stage of development prior to implantation into a patient or subject.

[0021] As used herein, a "quality assurance metric" refers to a continuous, ordinal, or categorical value of adherence to an established set of practices during the assisted reproductive process.

[0022] Current computer vision methods for embryo and oocyte assessment are semi-automatic, limited to measuring specific parameters that provide metrics that require further analysis by an embryologist, and require tightly controlled imaging systems. Previous attempts to develop systems using machine learning methods in assisted reproduction required intensive image preprocessing followed by manually guided feature segmentation for classification. Due to the reliance of machine learning methods on image processing and segmentation, this approach suffers from the same limitations as computer vision techniques.

[0023] Here, we overcome this challenge by employing a deep neural network pre-trained on a larger set of images for transfer learning of the classification of germ cell structures at clinically relevant developmental stages. Unlike existing computer-assisted algorithms for assessing germ cell structures, the systems and methods provided herein allow for automatic feature selection and analysis at the pixel level without any assistance by embryologists. In one example, a convolutional neural network was applied to identify shape, structural, and texture variations among morphologically complex germ cell structures. The system is resilient to variations in image illumination and quality due to the use of multiple instruments for data acquisition.

[0024] Figure 1A system 100 for assigning quality assurance metrics to germ cell structures is shown. The system 100 includes an imager 102 that acquires images of germ cell structures. For example, the imager 102 may include one or more cameras capable of producing images in the visible or infrared range, paired with appropriate optics to provide images of germ cell structures. In practice, the imager 102 may be implemented as part of a time-lapse oocyte / embryo imaging system, capturing images of germ cell structures over multiple days of development. In one embodiment, the imager 102 includes an accessory for a mobile device that operates with a camera of the mobile device to provide images of germ cell structures. The housing for the accessory may be 3D printed using polylactic acid having dimensions of 82×34×48 mm. An acrylic lens may be included in the housing to provide appropriate magnification of the image.

[0025] In another embodiment, the imager 102 can be implemented as a stand-alone system having an optical housing 3D printed from polylactic acid and an overall size of 62×92×175 mm. The housing contains electronic circuitry with a white light emitting diode, a three-volt battery, and a single-pole double-throw switch. The germ cell structure is transilluminated with a 10x plan achromatic objective for image magnification and a complementary metal-oxide-semiconductor (CMOS) image sensor for image data acquisition. The CMOS sensor can be connected to a single-board computer to process the captured images. The imager 102 can be connected to a mobile device via a wireless connection (e.g., Wi-Fi, Bluetooth, or similar connection) for data processing and visualization.

[0026] One or more images obtained at imager 102 are provided to neural network 104, which calculates at least one output value representing the morphology of the germ cell structure based on the image of the germ cell structure. For example, the output value can represent the quality of the germ cell structure based on the morphology of the germ cell structure, or a keyword that identifies the embryo based on its morphological characteristics. It should be understood that the neural network can be implemented as software instructions stored on a non-transitory computer readable medium and executed by an associated processor. In one example, neural network 104 can be implemented on a cloud computing system.

[0027] In one embodiment, the neural network 104 can be a convolutional neural network, which is a feed-forward artificial neural network including a convolutional layer that effectively applies convolution to the values ​​at the previous layer of the network to emphasize groups of features within the image. In a convolutional layer, each neuron is connected only to an appropriate subset of neurons in the previous layer, called the receptive field of the neuron. In the example shown, the convolutional neural network is implemented using the Xception architecture. In one embodiment, at least one chromaticity value (e.g., a value of an RGB color channel, a YCrCb color channel, or a grayscale brightness) associated with each pixel is provided as an initial input to the convolutional neural network.

[0028] In another embodiment, the neural network 104 can be implemented as a recursive neural network. In a recursive neural network, the connections between nodes in the network are selected to form a directed graph along a sequence, allowing it to exhibit dynamic temporal behavior. In another embodiment, the neural network 104 is implemented and trained as a discriminant network in a generative adversarial model, where the generative neural network and the discriminant network provide each other with feedback so that the generative neural network generates increasingly complex samples for the discriminant network to try to classify.

[0029] In another application, graph neural networks can be used. Graph neural networks are connection models that capture the dependencies of graphs through messages passed between the nodes of the graph. Unlike standard neural networks, graph neural networks retain states that can represent information from their neighborhoods with arbitrary depth. Graph neural networks are able to model the relationships between nodes in a graph and produce digital representations. In yet another embodiment, an autoencoder-based neural network can be used. An autoencoder is an unsupervised generative model that trains a neural network to represent input data in a useful form. In one embodiment, an autoencoder can be trained to reconstruct the input layer at the output layer, where alternative representations of the data are generated in the hidden layers of the network.

[0030] The inventors have discovered that the predictive power of the neural network 104 can be enhanced by combining the neural network 104 with another expert system (not shown). In practice, any of a variety of expert systems can be used in combination with a convolutional or recursive neural network, including support vector machines, random forests, self-organizing maps, fuzzy logic systems, data fusion processes, integration methods, rule-based systems, genetic algorithms, and artificial neural networks. It should be understood that additional expert systems can be trained based on features from multiple stages of embryonic development as well as features external to the image (such as biometric parameters of an egg donor, sperm donor, or recipient of an embryo).

[0031] Output values ​​from neural network 104 can be provided to quality assurance component 106, which compares the output values ​​with predetermined standards to provide quality assurance metrics. Quality assurance metrics can represent, for example, the performance of one of medical personnel, institutions and growth medium in extracting oocytes and fertilizing them and incubating embryos to implantation points, wherein predetermined standards represent the quality of germ cell structures. When used in this manner, quality assurance metrics can be used as a substitute for the measured results that may require a large amount of additional time to confirm, such as successful fertilization or pregnancy. It should be understood that in this case, descriptive statistics can be generated from multiple germ cell structures assessed to provide quality assurance metrics for given medical personnel, institutions or growth medium.

[0032] In one embodiment, neural network 104 is trained on multiple images or image groups of the oocyte taken before fertilization, and these images or image groups are classified as the first class representing the normal fertilization of embryo or the second class representing the abnormal fertilization of embryo.For the purpose of this application, the embryo of normal fertilization is the embryo comprising two pronuclei, and the embryo of abnormal fertilization is the embryo with any other number of pronuclei.The output of neural network can be a continuous value representing the quality of embryo, and this continuous value can be compared with a threshold value, so that fertilization is designated as "successful" or "unsuccessful".Therefore, in this embodiment, the performance of the embryologist or mechanism performing the oocyte extraction program can be evaluated as the percentage of the extracted oocyte that causes successful fertilization.Alternatively, the output of neural network can be used for generating the percentage of the expected successful fertilization when performing fertilization program (such as intracytoplasmic sperm injection (ICSI)) for embryologist or mechanism, and the quality assurance metric can be determined by comparing this expected value with the actual success rate of embryologist or mechanism.

[0033] In another embodiment, neural network 104 is trained on a plurality of embryo images or image groups taken at selected one or more stages of embryonic development, and these images or image groups are classified as representing successful pregnancy or representing unsuccessful pregnancy. In one example, each embryo is represented by the image taken at the third development day of embryo (for example 70 hours after fertilization). The output of neural network can be a continuous value representing the quality of embryo, and this continuous value can be compared with a threshold value, so that fertilization is designated as "successful" or "unsuccessful". Therefore, in this embodiment, the performance of the embryologist or mechanism of carrying out fertilization program or the growth culture medium for hatching embryo can be evaluated as the percentage of the fertilized egg mother cell that expects to produce successful pregnancy. Alternatively, the output of neural network can be used to produce the expected percentage of successful implantation of embryologist, mechanism or culture medium when carrying out embryo implantation, and quality metric can be determined by comparing this expected value with the actual success rate of embryologist, mechanism or culture medium.

[0034] In another embodiment, the output of neural network 104 is based on the feature of the morphology representing germ cell structure, and the identity of germ cell structure is represented as an identifier keyword. In this embodiment, the image of the germ cell structure taken at the first time can be provided to the neural network to produce the first identifier keyword, and the dish containing the germ cell structure can be marked with the identifier keyword provided. When it is desired to confirm the identity of the embryo in the dish, for example, just before being transferred to the recipient uterus, another image can be collected and provided to the neural network to generate the second identifier keyword. These two keywords can be compared to determine whether the embryo is correctly marked, and this information can be provided to the user as a quality assurance measure. In one example, the first time can be 113 hours after insemination, and the second time can be 115 hours after insemination.

[0035] Quality assurance metrics and any associated values ​​that may be of interest to the user in assuring the quality of the institution, medical professionals, or culture media involved in the assisted reproductive process may be provided to the user at an associated user interface 108. For example, user interface 108 may include at least one output device, such as a display, and appropriate software stored on a non-transitory medium and executed by an associated processor for receiving the output of neural network 104 and presenting it at the output device. In one embodiment, user interface 108 may include a mobile device that wirelessly communicates with the neural network.

[0036] Figure 2 An example implementation of a system 200 for allocating quality assurance metrics to embryos is provided. An important aspect of assisted reproductive technology is the conditions of laboratory embryo culture. The clinical outcome of an in vitro fertilization (IVF) cycle may be the best indicator of system efficiency, while ongoing pregnancy rates provide the most powerful marker of embryo quality. Several early developmental stage markers are widely used to monitor culture conditions; however, their relevance to clinical outcomes is unclear. Similarly, data analysis is a key part of an IVF quality assessment (QA) project. Regular reviews of identified key performance indicators (KPIs) are very important for ensuring the normal operation of the laboratory, and perhaps more importantly, for identifying potential problems so as to correct them in a timely manner. Fertilization assessment is the main result for measuring the efficiency of intracytoplasmic sperm injection (ICSI) embryology personnel. However, tracking the developmental fate of embryos obtained by ICSI can provide a more complete description of how this program is performed. Current quality assessments require manual inspection and recording of fertilization status and embryo development scores. These processes are labor-intensive and highly subjective in nature. Further, certain developmental indicators (such as pregnancy outcome results) are only available after significant delays.

[0037] The system 200 shown can be used for monitoring the alternative method of the KPI of the facilities, personnel and growth medium in the IVF laboratory, without the need for manual assistance. This system can also be used to detect the difference in the implantation potential of the developing embryo. The ability to accurately predict embryo implantation allows the practice of detecting and correcting the changes in culture conditions and technical proficiency in a few weeks faster than those relying on pregnancy results. In the example shown, the quality assurance metric is a classification parameter representing the expected success of the embryo implantation under the situation of given its morphological characteristics. The quality assurance metric of multiple embryos can be used to assess the impact of given medical professionals, growth medium or mechanism on the quality of the embryo, and allows to remedy the defects in any one of these factors. For example, in the case where growth medium provides a high-quality embryo percentage lower than a threshold value, growth medium can be replaced. In the case where an embryologist or a mechanism (such as ICSI) performing a fertilization procedure provides a high-quality embryo percentage lower than a threshold value, the personnel at the embryologist or mechanism can be retrained or subjected to additional supervision.

[0038] In this example, system 200 includes imager 202 that acquires images of each embryo on the third development day. However, it should be understood that the specific time at which images of the embryos are captured can vary based on the desired application, and in some embodiments, multiple images are captured at different stages of development to provide a set of images of each embryo. For example, in one embodiment, images captured on the first development day can be used to evaluate the embryos, particularly in the case of evaluating the performance of medical personnel or institutions during fertilization. Imager 202 can be used in a manner similar to Figure 1 The image captured at the imager 202 may be provided to the embryo analysis system 210.

[0039] The embryo analysis system 210 includes a processor 212, an output device 214 such as a display, and a non-transitory computer readable medium 220 storing executable instructions for providing quality assurance metrics representative of the performance of a medical professional, institution, or growth medium. The executable instructions include an imager interface 222 that receives images from the imager 202 and provides them in an appropriate form to a convolutional neural network 224 for analysis.

[0040] In one embodiment, training group embryo images can be generated from images or image groups representing known results of implantation of each embryo and embryo, and used to train convolutional neural network 224, so that the output of each image is a key performance indicator value representing the possibility that the implantation of the embryo will lead to a successful outcome. In the embodiment shown, the successful outcome is a successful pregnancy, but in some embodiments, the successful outcome can be that the embryo develops into a blastocyst or reaches the selected grade of the embryo at a selected developmental point (such as the fifth day of development). It should be understood that the feature of each embryo can include values ​​outside the image, such as biometric parameters of the patient, and images provided to convolutional neural network 224. In one example, the value output by convolutional neural network 224 can be a continuous value representing the possibility of a successful pregnancy, or a categorical value representing one or more ranges of possibilities. For example, a key performance indicator can be represented as a categorical parameter, which can adopt a first value indicating a possible successful pregnancy after implantation and a second value indicating that a successful pregnancy may not be possible.

[0041] The output of convolutional neural network 224 is provided to quality analysis component 226, which evaluates the performance of medical professionals, mechanism or growth medium. In the example shown, quality analysis component 226 maintains the running mean on the limited window, and this running mean represents the mean value of the key performance indicator value on the embryo in the limited window or has been indicated as the percentage of the embryo that may produce a successful pregnancy by convolutional neural network. For example, the running mean can be kept on the generated embryo of the limited time period or limited quantity. Quality analysis component 226 can monitor this running mean, and determines whether it meets a predetermined threshold value, so that quality assurance measurement is provided. For example, the predetermined threshold value of a successful pregnancy can be fifty percent.

[0042] In one example, the quality assurance metric is a categorical value that takes a first value when a moving average exceeds a threshold value and a second value when the moving average is below the threshold value. In another example, the quality assurance value remains at the first value until the running average drops below the threshold value within a predetermined amount of time or number of injections. The quality assurance metric and any other values ​​of interest (such as the running average) can be provided to the user at the output device 214 via an associated user interface 228. In one example, an alert can be provided to the user as an email, SMS message, or similar communication over a local area network or wide area network connection each time the quality assurance value changes from a first value to a second value.

[0043] Figure 3Provided is an example implementation of a system 300 for allocating quality metrics to oocytes. Failure of oocyte fertilization may be associated with both male and female factors. However, for some women, especially women with premature ovarian failure, reduced ovarian reserve or genetic diseases, donating eggs may be the only option for giving birth to healthy children. Adding donor eggs to the cycle significantly increases the out-of-pocket expenses of patients. Obtaining high-quality eggs with a higher chance of success may help reduce the uncertainty of patients while potentially improving pregnancy rates. At present, there is no objective system that can assess oocyte quality and predict its developmental potential. The system 300 shown can be used to accurately predict the fertilization potential of oocytes, and therefore allows the selection of the highest quality oocytes for fertilization and implantation. It can also be used to compare the performance of an institution or embryologist with the expected success rate in the case of the predicted fertilization potential of a given oocyte used to assess the performance of an institution or embryologist in the fertilization process. In the example shown, the system 300 can produce a continuous parameter representing the expected success of oocyte fertilization given its morphological characteristics.

[0044] System 300 includes an imager 302 that acquires an image of each oocyte prior to fertilization. Imager 302 may be configured in a manner similar to Figure 1 The images captured at the imager 302 may be provided to an oocyte analysis system 310, which includes a processor 312, an output device 314 (such as a display), and a non-transitory computer readable medium 320 storing executable instructions for providing a value representing the fertilization potential of an oocyte. The executable instructions include an imager interface 322 that receives images from the imager 302 and provides them in an appropriate form to a convolutional neural network 324 for analysis.

[0045] In one embodiment, a training group of oocyte images can be generated according to one or more images representing the known fertilization results of each oocyte and the oocyte, and used to train a convolutional neural network 324, so that the output of each image is a value representing the possibility that the fertilization of the oocyte will lead to a successful result. In the embodiment shown, the fertilization result of each training image is determined 18 hours after insemination. It should be understood that the feature of each embryo can include a value (such as a biometric parameter of the donor of the oocyte) outside the image, and the feature extracted from the convolutional neural network 324. It should be understood that the value output by the convolutional neural network 324 can be a continuous value representing the possibility of successful fertilization, or a classification value representing one or more ranges of possibilities. In one example, the value output by the convolutional neural network 324 is a classification parameter, which can adopt a first value indicating that successful fertilization is possible and a second value indicating that successful fertilization is impossible. The output of the convolutional neural network 324 is provided to the user at the output device 314 by an associated user interface 326.

[0046] In another embodiment, the output of the convolutional neural network 324 can represent the position of the polar body on the oocyte. Intracytoplasmic sperm injection is a procedure including metaphase (MII) oocyte arrangement, sperm selection and fixation, and injection of sperm at a precise position that does not interfere with the mitotic spindle. The spindle is located near the polar body extruded and cannot be visualized using a bright field microscope. Therefore, standard practice is to arrange oocytes based on the position of the polar body and inject sperm at ninety degrees from this visible structure. In this embodiment, the convolutional neural network is trained on multiple oocyte images with known positions of polar bodies, marked with categories representing discrete segments of the imaged oocyte. In one example, 12 categories were used, each of which represented a 30 degree segment of the imaged oocyte. The output of the convolutional neural network of the new image is the category representing the segment in which the polar body is located.

[0047] Figure 4 An example implementation of a system 400 for verifying the identity of an embryo is shown. Using an electronic witness system in an IVF laboratory is a recommended practice for improving traceability and reducing incorrect embryos being implanted in patients. This error is difficult to detect, and usually becomes apparent when couples give birth to babies with genetic makeup that is significantly different from their own. Therefore, the actual number of errors is difficult to estimate, and this type of biological sample misidentification may be disastrous for clinics, clinic staff, and especially patients. IVF laboratories are concerned about legal liability, and witness systems can play a vital role in minimizing identification errors.

[0048] The illustrated system 400 utilizes a multi-level traceability system that employs hardware-based radio frequency identification (RFID) tags combined with a software-based deep convolutional neural network that recognizes unique morphological features in a patient's oocytes and embryos. To this end, the system 400 includes an imager 402 that acquires images of germ cell structures at specific developmental stages. It should be understood that the specific time at which images of germ cell structures are captured can vary based on the desired application, and in some embodiments, multiple images are captured at different stages of development to provide a set of images for each germ cell structure. The imager 402 can be configured in a manner similar to Figure 1 The image captured at the imager 402 may be provided to the digital witness system 410.

[0049] The digital witness system 410 includes a processor 412, an output device 414 such as a display, and a non-transitory computer-readable medium 420 storing executable instructions for verifying the identity of an embryo or oocyte. The executable instructions include an imager interface 422, which receives images from the imager 402 and provides them to a neural network 424 in an appropriate form for analysis. The neural network can be implemented as, for example, a convolutional neural network, a graph neural network, and a neural network based on an autoencoder, or a generative adversarial neural network. In one embodiment, a neural network 424 can be pre-trained on a set of images to provide morphological features based on the provided images so that the output associated with a given image of an embryo or oocyte is an identifier keyword representing the morphology of the embryo or oocyte. The morphology of the embryo or oocyte can be stable in different developmental periods, so that the identifier keyword generated from the image taken at the first time is expected to match the identifier keyword generated from the image taken at the second time. In one example, an identifier keyword can be generated for a given germ cell structure at the first time and attached to a dish containing the germ cell structure, such as by encoding the identifier keyword in an RFID tag attached to the dish.

[0050] The identity verification component 426 compares the two identifier keywords to determine whether they represent the same germ cell structure. In one example, a second identifier key can be generated at a critical point in the assisted reproductive process, such as immediately before the fertilization of the oocyte or immediately before the embryo is transferred to the patient. The identity verification component 426 can generate a quality assurance metric representing the possibility that the second identifier keyword represents the same germ cell structure as the first identifier keyword, and therefore represents the possibility of the patient associated with the first identifier keyword. In one example, the quality assurance metric is classified, wherein the first value represents the match between the first identifier keyword and the second identifier keyword, and the second value represents that the first identifier keyword fails to match the second identifier keyword. Alternatively, the quality assurance metric can be a continuous value representing the possibility that the two keys represent the same embryo. The quality assurance metric can be provided to the user at the output device 414 via the associated user interface 428. In one example, whenever the quality assurance value assumes that the germ cell structure is misidentified as belonging to the wrong patient, the alarm can be provided to the user as an email, SMS message or similar communication via a local area network or wide area network connection.

[0051] In view of the structural and functional characteristics described above, refer to Figures 5 to 8 The method according to various aspects of the present invention will be better understood. Although for the sake of simplicity of description, Figures 5 to 8 The method is shown and described as being performed sequentially, but it should be understood and appreciated that the present invention is not limited to the order shown, because according to the present invention, some aspects can occur in different orders and / or simultaneously with other aspects shown and described herein. Moreover, not all of the features shown are required to implement a method according to one aspect of the present invention.

[0052] Figure 5 A method 500 for assigning quality assurance metrics to germ cell structures is shown. At 502, images of the germ cell structures are obtained. In practice, each germ cell structure can be represented by a set of one or more images taken at different times during the development of the germ cell structure, including, for example, before fertilization, during the first day of embryonic development, during the third day of embryonic development, during the fifth day of embryonic development, and just before implantation of the embryo. Generally, the images are obtained by visible light microscopy, although other forms of imaging may also be used depending on the application.

[0053] At 504, the image of the germ cell structure is provided to a neural network to generate a value representing the morphology of the germ cell structure. In one embodiment, the neural network is a convolutional neural network. In other embodiments, the neural network can be a generative adversarial neural network, a graph neural network, or an autoencoder-based neural network. In some examples, the steps at 502 and 504 can be repeated to provide corresponding values ​​representing the morphology of multiple germ cell structures. Further, in some embodiments, a set of multiple images can be obtained for each germ cell structure and provided to the neural network, wherein each image in the set of multiple images represents a germ cell structure at a different stage of development.

[0054] At 506, the values ​​representing the morphology of the plurality of germ cell structures are compared to predetermined criteria to provide a quality assurance metric representing one of the identity of the medical personnel, the institution, the growth medium, and the germ cell structure. In one example, the predetermined criteria is determined by obtaining a second image of the germ cell structure at the time the image was acquired at 506 and providing the second image to a neural network to provide the predetermined criteria as an identifier key for the germ cell structure. The identifier key can then be encoded on a radio frequency identification (RFID) tag and attached to the dish in which the germ cell structure is stored. In one embodiment, the quality assurance metric is a categorical value having a first value indicating that the first image and the second image represent the same germ cell structure when the unique key matches the predetermined criteria and a second value indicating that the first image and the second image represent different germ cell structures when the unique key does not match the predetermined criteria.

[0055] In another example, a representative value can be generated according to a plurality of values ​​representing the form of a plurality of germ cell structures, and this representative value is compared with a predetermined standard. In one example, each in a plurality of germ cell structures can be an embryo, which is the result of the fertilization process performed by a given embryologist or performed at a given institution, and each value representing the form of a given germ cell structure represents that the implantation of the embryo will produce the possibility of a successful pregnancy. The representative value can represent the percentage of a plurality of embryos that have been indicated as possibly producing a successful pregnancy, and the quality assurance metric can be the classification value of adopting the first value when the percentage is higher than a threshold value and adopting the second value when the percentage is lower than a threshold value. In one example, when the quality assurance metric transitions to the second value from the first value, the user can be changed.

[0056] Similarly, each of the multiple germ cell structures can be an embryo cultured in a given growth medium, and each value representing the morphology of a given germ cell structure represents the likelihood that the implantation of the embryo will produce a successful pregnancy. The representative value can represent the percentage of multiple embryos that have been indicated as likely to produce a successful pregnancy, and the quality assurance metric can be a categorized value that adopts a first value when the percentage is above a threshold and adopts a second value when the percentage is below a threshold. In one example, when the quality assurance metric transitions from a first value to a second value, the user can be changed. At 508, the quality assurance metric can be displayed to the user.

[0057] Figure 6 Shown is a method 600 for distributing quality assurance metrics to embryos.At 602, obtain multiple images of corresponding multiple embryos associated with medical professionals, institutions or growth medium.For example, multiple embryos can all be fertilized by a given embryologist or at a given institution or cultivated in identical growth medium.Each embryo can be represented by a group of one or more images taken at different times (comprising, for example, the first development day of embryos and the third day of embryos) during the development of the embryo.Generally speaking, images are obtained by visible light microscopes, although other forms of imaging can also be adopted according to application.At 604, each of a plurality of images is provided to a convolutional neural network, to generate corresponding multiple values ​​of the possibility of representing the successful result of the embryo (for example, developing into blastocysts, reaching a specific grade on the fifth day of development or successful pregnancy when implanting).In certain embodiments, a group of multiple images can be obtained for each germ cell structure and provided to a neural network, wherein each image in this group of multiple images represents the germ cell structure in different developmental stages.

[0058] At 606, from a plurality of values ​​representing the probability of the result of embryo success, a value representing the performance of one in medical professional, mechanism and growth medium is determined on a plurality of embryos. In one example, each of a plurality of values ​​is a classification parameter representing the probability range of successful result, and the percentage of the value representing a given category is generated as a representative value. In another example, each of a plurality of values ​​is a continuous parameter, and the representative value is the mean value (for example, mean or median) on a plurality of values. At 608, the value is compared with a threshold value, to generate a quality assurance metric. For example, the quality assurance metric can adopt a first value when the representative value exceeds a threshold value, and the quality assurance metric adopts a second value when the representative value does not exceed the threshold value. At 610, the quality assurance metric is displayed to the user at an output device.

[0059] Figure 7A method 700 for assigning a value representing the quality of an oocyte is shown. At 702, an image of an oocyte is acquired from an associated imager. Generally, the image is obtained by a visible light microscope, although other forms of imaging may be used depending on the application. At 704, the image is provided to a convolutional neural network that determines a value representing the likelihood of successful fertilization of the oocyte based on the image of the oocyte. In one example, the convolutional neural network is trained on an image of an oocyte with a known outcome and labeled with the fertilization status of an embryo resulting from fertilization of an oocyte 18 hours after insemination. At 706, a value representing the likelihood of successful fertilization of the oocyte is displayed to a user at an output device.

[0060] Figure 8 A method 800 for verifying the identity of a germ cell structure is shown. At 802, a first image of a germ cell structure is obtained at a first time. In practice, each germ cell structure can be represented by a set of one or more images taken at different times during the development of the germ cell structure, including, for example, before fertilization, during the first embryonic development day, during the third embryonic development day, during the fifth day of embryonic development, and immediately before implantation of the embryo. At 804, the first image is provided to a neural network to provide an identifier keyword K representing the morphology of the germ cell structure. In one example, the neural network is a convolutional neural network. In another example, the neural network is a generative adversarial neural network, a graph neural network, or an autoencoder-based neural network.

[0061] At 806, a second image of the germ cell structure is obtained at a second time. At 808, the second image is provided to a neural network to provide a value V representing the morphology of the germ cell structure. At 810, a determination is made whether the value matches the identifier keyword. If yes (Y), then at 812 it is determined that the germ cell structure belongs to the patient associated with the identifier keyword. If no (N), then at 814 it is determined that the germ cell structure does not belong to the patient associated with the identifier keyword and an error may have occurred.

[0062] Fig. 9 It shows that it can be implemented Figures 1 to 8 An example system 900 of example hardware components of the systems and methods disclosed in Figure 1 Schematic block diagram of a system for assigning quality assurance metrics to germ cell structures as shown in FIG. System 900 may include various systems and subsystems. System 900 may be a personal computer, a laptop computer, a workstation, a computer system, an appliance, an application-specific integrated circuit (ASIC), a server, a server blade center, a server cluster, etc.

[0063] The system 900 may include a system bus 902, a processing unit 904, a system memory 906, memory devices 908 and 910, a communication interface 912 (e.g., a network interface), a communication link 914, a display 916 (e.g., a video screen), and an input device 918 (e.g., a keyboard and / or a mouse). The system bus 902 may communicate with the processing unit 904 and the system memory 906. Additional storage devices 908 and 910, such as hard drives, servers, independent databases, or other non-volatile memories, may also communicate with the system bus 902. The system bus 902 interconnects the processing unit 904, the memory devices 906-910, the communication interface 912, the display 916, and the input device 918. In some examples, the system bus 902 also interconnects additional ports (not shown), such as a universal serial bus (USB) port.

[0064] System 900 may be implemented in a computing cloud. In this case, features of system 900 (such as processing unit 904, communication interface 912, and memory devices 908 and 910) may represent a single hardware instance or multiple hardware instances, where applications are executed on multiple (i.e., distributed) hardware instances (e.g., computers, routers, memories, processors, or a combination thereof). Alternatively, system 900 may be implemented on a single dedicated server.

[0065] The processing unit 904 may be a computing device and may include an application specific integrated circuit (ASIC). The processing unit 904 executes a set of instructions to implement the operations of the examples disclosed herein. The processing unit may include a processing core.

[0066] Additional storage devices 906, 908, and 910 can store data, programs, instructions, text or compiled forms of database queries and any other information that may be needed to operate the computer. Memory 906, 908, and 910 can be implemented as a computer-readable medium (integrated or removable), such as a memory card, a disk drive (CD), an optical disk, or a server accessible via a network. In some examples, memory 906, 908, and 910 may include text, images, video, and / or audio, portions of which may be available in a human-understandable format.

[0067] Additionally or alternatively, the system 900 may access external data sources or query sources through a communication interface 912 , which may be in communication with the system bus 902 and a communication link 914 .

[0068] In operation, system 900 can be used to implement one or more portions of a system for assigning quality assurance metrics to germ cell structures according to the present invention. According to some examples, computer executable logic for implementing the quality assurance system resides on system memory 906 and one or more of storage devices 908 and 910. Processing unit 904 executes one or more computer executable instructions originating from system memory 906 and storage devices 908 and 910. It should be understood that the computer readable medium may include a plurality of computer readable media each operably connected to the processing unit.

[0069] Specific details are given in the above description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments can be practiced without these specific details. For example, in order not to obscure the embodiments in unnecessary details, circuits can be shown in block diagrams. In other cases, well-known circuits, processes, algorithms, structures and techniques can be shown without unnecessary details to avoid obscuring the embodiments.

[0070] The implementation of the above-mentioned techniques, blocks, steps and means can be accomplished in various ways. For example, these techniques, blocks, steps and means can be implemented in hardware, software or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (digital signal processors, DSPs), digital signal processing devices (digital signal processing devices, DSPDs), programmable logic devices (programmable logic devices, PLDs), field programmable gate arrays (field programmable gate arrays, FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the above-mentioned functions and / or combinations thereof.

[0071] Moreover, it is noted that the embodiments can be described as processes depicted as flowcharts, flow charts, data flow diagrams, structure diagrams or block diagrams. Although flowcharts can describe operations as sequential processes, many operations can be performed in parallel or concurrently. In addition, the order of operations can be rearranged. When the operation of a process is completed, the process is terminated, but may have additional steps not included in the figure. The process can correspond to a method, function, program, subroutine, subprogram, etc. When a process corresponds to a function, its termination corresponds to the function returning to a calling function or a main function.

[0072] In addition, the embodiment can be implemented by hardware, software, scripting language, firmware, middleware, microcode, hardware description language and / or any combination thereof.When implemented with software, firmware, middleware, scripting language and / or microcode, the program code or code segment performing the necessary tasks can be stored in a machine-readable medium such as a storage medium. Code segment or machine executable instruction can represent any combination of a program, function, subroutine, program, routine, subroutine, module, software package, script, class or instruction, data structure and / or program statement.By transmitting and / or receiving information, data, independent variable, parameter and / or memory content, code segment can be coupled to another code segment or hardware circuit.Information, independent variable, parameter, data etc. can be transmitted, forwarded or transmitted by any suitable means, including memory sharing, message transmission, ticket transmission, network transmission etc.

[0073] For firmware and / or software implementations, the methods can be implemented using modules (e.g., programs, functions, etc.) that perform the functions described herein. Any tangible machine-readable medium that implements instructions can be used to implement the methods described herein. For example, software code can be stored in a memory. The memory can be implemented inside a processor or outside a processor. As used herein, the term "memory" refers to any type of long-term, short-term, and volatile, non-volatile or other storage medium, and is not limited to any particular type of memory or amount of memory, or the type of medium that stores the memory.

[0074] Moreover, as disclosed herein, the term "storage medium" may refer to one or more memories for storing data, including read only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices, and / or other machine-readable media for storing information. The terms "computer-readable medium" and "machine-readable medium" include, but are not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and / or various other storage media capable of storing, containing or carrying (multiple) instructions and / or data. It should be understood that a "computer-readable medium" or "machine-readable medium" may include multiple media, each of which is operably connected to a processing unit.

[0075] The above-described contents are examples. Of course, it is impossible to describe every conceivable combination of parts or methods, but those of ordinary skill in the art may recognize that many further combinations and permutations are possible. Therefore, the present disclosure is intended to include all such changes, modifications and variations that fall within the scope of the present application, including the appended claims. As used herein, the term "includes" means including but not limited to, and the term "including" means including but not limited to. The term "based on" refers to at least partially based on. Additionally, in the case where the present disclosure or claims narrate "one", "one", "first" or "another" element or its equivalent, it should be interpreted as including one or more than one such element, neither requiring nor excluding two or more such elements.

Claims

1. A method for assigning a quality parameter to one of a medical professional, an institution, and a growth medium, the method comprising: obtaining a plurality of images corresponding to the plurality of embryos; providing the plurality of images of the plurality of embryos to a neural network to generate, for each embryo in the plurality of embryos, a value representing a likelihood of a successful outcome for the embryo based on the morphology of the embryo to provide a set of values ​​representing a likelihood of a successful outcome for the plurality of embryos, wherein the value representing the likelihood of a successful outcome for the embryo represents a likelihood that implantation of the embryo will result in a successful pregnancy; calculating descriptive statistics representing the set of values; and The descriptive statistics are compared to predetermined standards to provide a quality assurance metric representative of the one of the medical professional, the institution, and the growth medium.

2. The method according to claim 1, wherein: Obtaining multiple images of each embryo in the plurality of embryos includes obtaining multiple images representing each embryo in the plurality of embryos, and providing the multiple images to the neural network includes providing the multiple images of each embryo in the plurality of embryos to the neural network to generate multiple values ​​representing the morphology of each embryo in the plurality of embryos.

3. The method according to claim 2 further comprises generating a representative value based on a plurality of values ​​representing each embryo.

4. The method according to claim 3, wherein: The representative value indicates the probability that implantation of the embryo will result in a successful pregnancy.

5. The method according to claim 2, wherein: Each image in the plurality of images represents an embryo at a different day of development.

6. The method according to claim 3 or 4, wherein: The plurality of embryos are cultured in a given growth medium, and the descriptive statistics represent a percentage of the plurality of embryos for which implantation of the embryos is expected to result in a successful pregnancy, and the predetermined criterion is a threshold percentage, the method further comprising alerting a user when the representative value drops below the threshold percentage.

7. The method according to claim 3 or 4, wherein: The plurality of embryos were all fertilized by a given embryologist, and the descriptive statistics represent the percentage of the plurality of embryos for which implantation of the embryos is expected to result in a successful pregnancy, and the predetermined criterion is a threshold percentage, the method further comprising alerting a user when the representative value drops below the threshold percentage.

8. The method according to claim 1, wherein: Providing the plurality of images to the neural network includes providing the plurality of images to a convolutional neural network.

9. A system comprising: an imager that generates a plurality of images of a corresponding plurality of embryos, all of the plurality of embryos being associated with one of a medical professional, an institution, and a growth medium, wherein the association means that the plurality of embryos were all fertilized by the medical professional or at the institution, or cultured in the growth medium; processor; output devices; and a non-transitory computer-readable medium storing machine-executable instructions for assigning a quality assurance metric to said one of said medical professional, said facility, and said growth medium, said machine-executable instructions comprising: an imager interface that receives the plurality of images from the imager; a convolutional neural network that determines, based on each of the plurality of images, a value representing a likelihood of a successful outcome for the embryo to provide a set of values ​​representing a likelihood of a successful outcome for the plurality of embryos, wherein the value representing the likelihood of a successful outcome for the embryo represents a likelihood that implantation of the embryo will result in a successful pregnancy; a quality analysis component that calculates a value representing performance of said one of said medical professional, said institution, and said growth medium as a descriptive statistic representing said set of values ​​representing a likelihood of a successful outcome on said plurality of embryos and compares said value to a threshold value to generate said quality assurance metric; and A user interface displays the quality assurance metrics to a user at the output device.

10. The system according to claim 9, wherein: The convolutional neural network determines as a classification parameter a value representing the likelihood of a successful outcome for the embryo, and the quality analysis component determines as a percentage of embryos among the plurality of embryos to which the convolutional neural network assigns a first value as the value representing the likelihood of a successful outcome for the embryo.

11. The system according to claim 9, wherein: The convolutional neural network determines, for each embryo, a value representing the likelihood of a successful pregnancy following implantation of the embryo.

12. The system according to claim 9, wherein: The convolutional neural network is trained on multiple images of embryos taken on the first developmental day, and the quality analysis component calculates a value representing the performance of the growth medium.

13. The system according to claim 9, wherein: The convolutional neural network is trained on a plurality of images of embryos taken on a third developmental day, and the quality analysis component calculates a value representative of the performance of one of the medical professional and the institution.

Images