Methods and systems for analyzing biological material

Abstract

The present disclosure relates to a method performed by one or more computers for analyzing an embryo during a biopsy process. The method includes, at each of a plurality of time points: generating a virtualization of the biopsy environment. The method includes, at each of the plurality of time points, processing the virtualization of the biopsy environment using a machine learning model to generate a model output that comprises an action to be performed by the agent. The method includes, at each of the plurality of time points, causing the agent to perform the action.

Description

METHODS AND SYSTEMS FOR ANALYZING BIOLOGICAL MATERIALCLAIM OF PRIORITY

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 617,239 filed on January 03, 2024, and U.S. Provisional Patent Application No. 63 / 702,779, filed on October 03, 2024, the entire contents of which are hereby incorporated by reference.TECHNICAL FIELD

[0002] The present disclosure relates to methods and systems for analyzing biological material, and more specifically, methods and systems for analyzing biological material in assisted reproduction technology (ART) procedures.BACKGROUND

[0003] When conducting an ART procedure, standard practice is to create multiple embryos and transfer the embryo that has the best chance of developing into a healthy baby back into the uterus. Embryos that are aneuploid (z.e., having abnormal, extra or missing chromosomal material which can include whole chromosomes or smaller subsections of chromosomes) are less likely to make it to birth and more likely to be bom with disorders, so genomic testing of embryos, which can identify aneuploid embryos, has become a common practice.

[0004] To carry out genomic testing, cells are removed from the embryo in a biopsy process and sent to a genomics lab for testing, and the embryo is vitrified (z.e., frozen in liquid nitrogen) while awaiting results. After receiving the results, the embryo associated with the biopsy will be thawed and transferred if viable or discarded if not viable.

[0005] The biopsy process happens in a drop of fluid on a dish. During the biopsy process, cells are removed from the trophectoderm of the embryo using a biopsy pipette. The number of cells removed during the biopsy process should be sufficient to carry out accurate genomic testing, but should be minimized to prevent damage to the embryo that may result in reduced embryo competency and live birth potential. Furthermore, while cells are removed, the embryo is exposed to a non-controlled environment with changing conditions such as temperature fluctuations that may damage the embryo. The duration that the embryo is exposed to the non-controlled environment should be minimized to prevent damage to the embryo.SUMMARYThe present disclosure is directed to systems and methods for analyzing a subject’s biological material in a lab during an ART procedure. The systems disclosed herein provide guidance and / or automation to ensure the correct amount of biological material is obtained during the biopsy process, and that the biological material is removed from the correct location of the embryo.

[0006] In a first example aspect, a method performed by one or more computers for analyzing an embryo during a biopsy process may include receiving, from a camera in realtime during the biopsy process, one or more images that each depict an embryo and a biopsy pipette. The method may include processing, in real-time using a machine learning model, the one or more images to determine an amount of biomaterial contained within the biopsy pipette; and causing, in real-time, a user interface to display an indication of the amount of biomaterial within the biopsy pipette.

[0007] In a second example aspect, one or more non-transitory computer storage media may store instructions that when executed by one or more computers cause the one or more computers to perform operations for analyzing an embryo during a biopsy process. The operations may include receiving, from a camera in real-time during the biopsy process, one or more images that each depict an embryo and a biopsy pipette; processing, in real-time using a machine learning model, the one or more images to determine an amount of biomaterial contained within the biopsy pipette; and causing, in real-time, a user interface to display an indication of the amount of biomaterial within the biopsy pipette.

[0008] In a third example aspect, a system for analyzing an embryo during a biopsy process may include a camera; one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for analyzing an embryo during a biopsy process. The operations may include receiving, from the camera in real-time during the biopsy process, one or more images that each depict an embryo and a biopsy pipette; processing, in real-time using a machine learning model, the one or more images to determine an amount of biomaterial contained within the biopsy pipette; and causing, in real-time, a user interface to display an indication of the amount of biomaterial within the biopsy pipette.

[0009] In accordance with any one of the first, second, and third aspects, the method, system, and non-transitory computer storage media for analyzing an embryo during a biopsy process may include any one of the following forms.

[0010] In some examples, processing the one or more images may include identifying one or more regions of interest in each image, wherein the one or more regions of interest can include at least one of a holding pipette, the biopsy pipette, or biomaterial.

[0011] In some examples, the method may include determining, in real-time, a segmentation mask for each region.

[0012] In some examples, the method may include causing, in real-time, the user interface to display an indication of each segmentation mask.

[0013] In some examples, the biomaterial may include at least one of an embryo, an inner cell mass, a zona pellucida, an inner boundary of the zona pellucida, trophectoderm cells within the zona pellucida, trophectoderm cells within the biopsy pipette, or a herniation.

[0014] In some examples, the method may include based on processing the one or more images, determining, in real-time, that the amount of biomaterial contained within the biopsy pipette is below a predetermined range; and in response, causing, in real-time, the user interface to display an indication that the amount of biomaterial contained within the biopsy pipette is below the predetermined range.

[0015] In some examples, the method may include based on processing the one or more images, determining, in real-time, that the amount of biomaterial contained within the biopsy pipette is above a predetermined range; and in response, causing, in real-time, the user interface to display an indication that the amount of biomaterial contained within the biopsy pipette is above the predetermined range.

[0016] In some examples, the method may include based on processing the one or more images, determining, in real-time, that the amount of biomaterial contained within the biopsy pipette is within a predetermined range; and in response, causing, in real-time, the user interface to display an indication that the amount of biomaterial contained within the biopsy pipette is within the predetermined range.

[0017] In some examples, the predetermined range is between 5 and 10 cells.

[0018] In some examples, the predetermined range is between 67% and 133% of an adequate target amount.

[0019] In some examples, the method may include based on processing the one or more images, determining, in real-time, a separation region where the biopsy pipette and the embryo are in contact; and in response, causing, in real-time, the user interface to display an indication of the separation region.

[0020] In some examples, the method may include causing, in real-time, the user interface to display a laser guide at the separation region.

[0021] In some examples, the method may include causing a laser to be directed at the separation region; and controlling the laser to be activated while the laser is directed at the separation region.

[0022] In some examples, processing the one or more images to determine an amount of biomaterial contained within the biopsy pipette comprises processing, in real-time using the machine learning model, the one or more images to determine a volume of the biomaterial.

[0023] In some examples, processing the one or more images to determine an amount of biomaterial contained within the biopsy pipette comprises processing, in real-time using the machine learning model, the one or more images to determine a number of cells contained within the biopsy pipette.

[0024] In some examples, the method may include processing, in real-time using the machine learning model, the one or more images, to determine one or more entry regions of the embryo for taking the biopsy.

[0025] In some examples, the one or more entry regions include at least one of a herniation of the embryo, or a region away from an inner cell mass of the embryo.

[0026] In some examples, the method may include causing, in real-time, the user interface to display an indication of the one or more entry regions.

[0027] In some examples, the method may include causing the biopsy pipette to be enter the embryo at one of the one or more entry regions; and causing the biopsy pipette to suction biomaterial from the embryo.

[0028] In some examples, receiving one or more images may include receiving, in realtime from the camera, a sequence of images that depicts the embryo and the biopsy pipette; and processing the one or more images may include selecting two or more images from the sequence of images; identifying one or more images in the sequence as non-selected images; and processing, in real-time using a machine learning model, the selected images to determine an amount of biomaterial contained within the biopsy pipette.

[0029] In some examples, processing, in real-time using a machine learning model, the selected images to determine an amount of biomaterial contained within the biopsy pipette may include determining a segmentation mask for each of one or more regions in each of the selected images, wherein each region depicts at least one of a holding pipette, the biopsy pipette, or biomaterial.

[0030] In some examples, processing the one or more images may include for each of the one or more non-selected images in the sequence of images: estimating a segmentation mask for each of the one or more regions in the non-selected image.

[0031] In some examples, estimating a segmentation mask for each of the one or more regions in each of the one or more non-selected images may include: for each of the one or more regions, estimating a respective location for each of a plurality of points that define the segmentation mask based on the respective location for each of the plurality of points in one or more prior images in the sequence of images.

[0032] In some examples, the method may include, for each of the one or more images: causing, in real-time, the user interface to display an indication of each segmentation mask or each estimated segmentation mask.

[0033] In some examples, the method may include determining that the biopsy pipette has separated from the embryo; determining that the amount of biomaterial within the biopsy pipette is non-zero; and in response, recording the amount of biomaterial within the biopsy pipette and an identifier for the embryo.

[0034] In some examples, the method may include training the machine learning model, wherein training comprises receiving a set of training data, wherein the training data may include: (i) a plurality of training outputs, each training output including a labeled image, wherein the labeled image includes an image depicting an embryo and a biopsy pipette and includes one or more labels corresponding to one or more regions in the labeled image, (ii) a plurality of training inputs, each training input corresponding to a training output, and each training input comprising an unlabeled image, wherein the unlabeled image includes the image of the corresponding training output, and training the machine learning model on the set of training data.

[0035] In some examples, training the machine learning model on the set of training data may include training the machine learning model to, for each training input, process the training input to generate a model output that matches the corresponding training output.

[0036] In some examples, training the machine learning model on the set of training data may include training the machine learning model to detect one or more regions in an image.

[0037] In some examples, the one or more labels correspond to a region that depicts trophectoderm cells within the biopsy pipette and represent an amount of trophectoderm cells within the biopsy pipette.

[0038] In some examples, training the machine learning model on the set of training data may include training the machine learning model to predict an amount of trophectoderm cells within the biopsy pipette in an image.

[0039] In some examples, the system may include a laser configured to be directed at a separation region depicted in the one or more images.

[0040] In some examples, the system may include a microscope configured to be directed at the embryo.

[0041] In some examples, the system may include a plurality of actuators configured to move at least one of the biopsy pipette, a holding pipette, a dish, a laser, or a microscope.

[0042] In a fourth example aspect, a method performed by one or more computers for controlling an agent interacting with a biopsy environment during a biopsy process may include at each of a plurality of time points, generating a virtualization of the biopsy environment, wherein the virtualization depicts an embryo and a biopsy pipette at the respective time point. At each of the plurality of time points, the method may include processing the virtualization of the biopsy environment using a machine learning model to generate a model output that comprises an action to be performed by the agent; and causing the agent to perform the action.

[0043] In a fifth example aspect, one or more non-transitory computer storage media may store instructions that when executed by one or more computers cause the one or more computers to perform operations for controlling an agent interacting with a biopsy environment during a biopsy process may include, at each of a plurality of time points, generating a virtualization of the biopsy environment, wherein the virtualization depicts an embryo and a biopsy pipette at the respective time point. At each of the plurality of time points, the method may include processing the virtualization of the biopsy environment using a machine learning model to generate a model output that comprises an action to be performed by the agent; and causing the agent to perform the action.

[0044] In a sixth example aspect, a system for controlling an agent interacting with a biopsy environment during a biopsy process may include a camera; one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations controlling an agent interacting with a biopsy environment during a biopsy process may include, at each of a plurality of time points, generating a virtualization of the biopsy environment, wherein the virtualization depicts an embryo and a biopsy pipette at the respective time point. At each of the plurality of time points, the method may include processing the virtualization of the biopsy environment using a machine learning model to generate a model output that comprises an action to be performed by the agent; and causing the agent to perform the action.

[0045] In accordance with any one of the fourth, fifth, and sixth aspects, the method, system, and non-transitory computer storage media for analyzing an embryo during a biopsy process may include any one of the following forms.

[0046] In some examples, generating a virtualization of the biopsy environment may include: obtaining an image depicting the embryo and the biopsy pipette; and in response, generating the virtualization of the biopsy environment from the image.

[0047] In some examples, obtaining the image may include receiving the image from a camera directed at the embryo and the biopsy pipette.

[0048] In some examples, the image may include a simulated image, and obtaining the image may include generating the simulated image from a model of the embryo.

[0049] In some examples, generating the virtualization of the biopsy environment from the image may include generating, using a segmentation machine learning model, a feature segmentation map for the image comprising one or more segmentation masks.

[0050] In some examples, the method may include displaying the feature segmentation map in the virtualization of the biopsy environment.

[0051] In some examples, the method may include identifying a portion of the image comprising the embryo based on the feature segmentation map; and in response, displaying the identified portion in the virtualization of the biopsy environment.

[0052] In some examples, generating a virtualization of the biopsy environment may include: receiving data indicating a state of the biopsy pipette; identifying a change in the state of the biopsy pipette based on the data indicating the state of the biopsy pipette; and generating the virtualization of the biopsy environment based on the change in the state of the biopsy pipette.

[0053] In some examples, generating a virtualization of the biopsy environment may include: receiving data indicating a state of a holding pipette; identifying a change in the state of the holding pipette based on the data indicating the state of a holding pipette; and generating the virtualization of the biopsy environment based on the change in the state of the holding pipette.

[0054] In some examples, causing the agent to perform the action may include controlling one or more actuators to move at least one of the biopsy pipette, a holding pipette, a dish containing the embryo, a stage supporting the embryo, a laser, or a microscope.

[0055] In some examples, causing the agent to perform the action may include controlling one or more actuators to move the biopsy pipette to contact the embryo at a region of the embryo.

[0056] In some examples, causing the agent to perform the action may include controlling one or more actuators to cause suction to be applied to the embryo through the biopsy pipette.

[0057] In some examples, causing the agent to perform the action may include controlling one or more actuators to cause suction to be applied to the embryo through a holding pipette.

[0058] In some examples, causing the agent to perform the action may include controlling one or more actuators to cause a laser to be directed at a region of the embryo.

[0059] In some examples, causing the agent to perform the action may include causing the laser to be activated while the laser is directed at the region of the embryo.

[0060] In some examples, the model output may include a probability distribution for a plurality of possible actions.

[0061] In some examples, causing the agent to perform the action may include selecting the action to be performed by the agent from the plurality of possible actions.

[0062] In some examples, the machine learning model may be trained to generate a model output that maximizes a total reward based on one or more reward scores at each time point.

[0063] In some examples, each reward score may be based on any one or more of: an amount of biomaterial contained within the biopsy pipette, a time elapsed for the biopsy process, a proximity of a sample locus to one or more identified sampling regions of the embryo, a number of times the embryo was touched by the biopsy pipette, and a proximity of a holding pipette to one or more identified holding placement regions.

[0064] In some examples, the agent is a human operator, and causing the agent to perform the action may include providing data representing instructions to the human operator to perform the action.

[0065] In some examples, the system may include a camera, and generating a virtualization of the biopsy environment may include: receiving an image depicting the embryo and the biopsy pipette from the camera; and in response, generating the virtualization of the biopsy environment from the image using a segmentation machine learning model.

[0066] In some examples, the system may include a laser configured to be directed at a separation region depicted in the virtualization, the separation region indicating a region where the biopsy pipette and the embryo are in contact.

[0067] In some examples, the system may include a microscope configured to be directed at the embryo.

[0068] In some examples, the system may include a plurality of actuators configured to move at least one of the biopsy pipette, a holding pipette, a dish, a laser, or a microscope.

[0069] Systems and methods described in the present disclosure can include one or more of the following advantages.

[0070] In some examples, the system described herein can improve the accuracy of determining the number of cells contained within a biopsy pipette. For example, the system described in this specification can determine, in real-time, an amount of biomaterial, e.g., the number of cells, within the biopsy pipette. The system can thus accurately determine the number of cells within the biopsy pipette, while minimizing the amount of time the embryo is exposed to the non-controlled environment.

[0071] In addition, the system described in this specification can provide guidance to an embryologist on where to remove cells from the embryo, which can improve the accuracy of the biopsy process. In some examples, the system described herein can improve training of embryologists. For example, the guidance provided by the system regarding the number of cells contained within a biopsy pipette and the location to remove cells from an embryo can be used for confirmation during the training of an embryologist. The system can thus reduce the amount of training needed, or make training more efficient, reducing the barrier to entry and allowing more embryologists to perform biopsy processes. The guidance can also be used by the embryologist for confirmation during the biopsy process. The system can thus allow the embryologist to be more confident in their skills.

[0072] In some examples, the system described herein can reduce the risk of damage to an embryo during the biopsy process. For example, the system described in this specification can determine, in real-time, that the amount of biomaterial within the biopsy pipette is sufficient for testing and, as a result, the amount of biomaterial withdrawn from the embryo during the biopsy is minimized which reduces the risk of damage to the embryo. In addition, the system can determine a separation region and can cause a user interface to display an indication of the separation region so that the embryologist can direct a laser at the separation region, reducing the risk of damage by enabling more accurate and efficient separation of the sample from embryo.

[0073] In some examples, the system provides real-time feedback to the embryologist and thereby prevents errors from occurring. For example, while the embryologist is extracting material from the embryo, the system can notify the embryologist before the embryologist removes the biopsy pipette from the embryo.

[0074] In some examples, the system described herein can automate the biopsy process and thereby prevents errors from occurring and minimizes damage to the embryo. For example, the system can include actuators for the holding pipette and the biopsy pipette. The system thus reduces the time and embryologist training required to perform a biopsy, and can provide for a streamlined ART procedure. In some of these examples, the system can automate the biopsy process by determining an action to be performed at each of multiple time points during the biopsy process. The system can perform the actions using the actuators, for example.

[0075] In some examples, the system may be incorporated easily into existing work spaces, and may be retrofitted to work with existing cameras and microscopes. For example, the system can determine an amount of biomaterial in a biopsy pipette from images taken under different conditions, including lighting conditions, image quality, and scaling factors. Further, in some examples, the system can be used for analyzing other biological materials in an ART procedure or other process.

[0076] As used herein, the terms “top,” “bottom,” “upper,” “lower,” “above,” and “below” are used to provide a relative relationship between structures. The use of these terms does not indicate or require that a particular structure must be located at a particular location in the apparatus.

[0077] Some examples may be described using the expression “coupled” and “connected” along with their derivatives. For example, some arrangements may be described using the term “coupled” to indicate that two or more elements are in direct physical or electrical contact. The term “coupled,” however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. The examples described herein are not limited in this context.

[0078] Other aspects, features, and advantages of the present disclosure will be apparent from the following detailed description, figures, and claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0079] FIG. 1 A is a schematic diagram of a system for analyzing biological material in an ART procedure.

[0080] FIG. IB is a block diagram of an example embryo analysis system.

[0081] FIGS. 2A-2B show example images and indications generated by the embryo analysis system of FIG. IB that can be displayed by a user interface.

[0082] FIG. 3 is a flow chart of an example process for analyzing biological material.

[0083] FIG. 4 is a block diagram of another example embryo analysis system.

[0084] FIG. 5 shows an example image of biomaterial that has been removed from an embryo.

[0085] FIG. 6 is a block diagram of an example computing system.

[0086] FIG. 7 is a diagram of example approximating cylinders for determining the amount of biomass within a biopsy pipette.

[0087] FIG. 8 shows an example virtualization of an environment for analyzing biological material in an ART procedure.

[0088] FIG. 9 is a diagram of an example action machine learning model.

[0089] FIG. 10 is a flow chart of an example process for controlling an agent interacting with an environment during a biopsy process.DETAILED DESCRIPTION

[0090] FIG. 1 A depicts a system for analyzing biological material, in particular, for determining the amount of biomaterial in a biopsy pipette in real-time during a biopsy process, which reduces instances of error in the biopsy process. As used herein, a real-time operation may describe an operation that is performed with minimal delay, taking into account the limitations of the computing system(s) performing the operation. For example, the systems described herein can determine the amount of biomaterial within a biopsy pipette and display an indication of the amount of biomaterial contained in the biopsy pipette within 10-60 milliseconds of capturing an image of the biopsy pipette using a camera.

[0091] In FIG. 1A, a system 100 for performing a biopsy of a subject’s biological material in a lab during an ART procedure includes an embryo analysis system 140 including a camera 142 that is coupled to a microscope 146 and the computer 144. The microscope 146 shown is an inverted stereoscope. The microscope 146 is arranged on a work surface 121, and the computer 144 is disposed underneath or near the work surface 121. The camera 142 is coupled to the computer 144, and may be coupled to a user interface 112. The embryo analysis system 140 is configured to monitor and record a biopsy process being performed. The embryo analysis system 140 is described in further detail below with reference to FIG. IB.

[0092] The camera 142 continuously or periodically captures images of one or more dishes on a platform 110, such as dish 122, and delivers the images to the computer 144. In some examples, the dish 122 can be placed over a light source 124. The camera 142 is mounted to a port of the microscope 146 so that the camera 142 has a magnified view of anembryo on the dish 122. As depicted in FIG. 1 A, the camera 142 can be coupled to a body 145 of the microscope 146. In the illustrated example system, the camera 142 is a webcam, but may be any suitable camera such as a video camera.

[0093] The computer 144 is communicatively coupled to the camera 142 by a wired and / or wireless connection, such as via Bluetooth™, or radio communication (e.g., Wi-Fi). The embryo analysis system 140 is configured to deliver real-time feedback in the form of visual indications, prompts, and / or alerts to the embryologist. This real-time feedback can be delivered through the user interface 112, which is coupled to the computer 144, and / or through audible feedback. The user interface 112 can include a screen 114 that can display indications to the embryologist. For example, the user interface 112 can be configured to temporarily display a message or color on the screen 114 when prompted by the embryo analysis system 140. For example, the screen 114 can display visual indications 208, 210, 212, 214, 216, 218, and 260 that include the amount of biomaterial detected within a biopsy pipette, or whether the amount of biomaterial detected within the biopsy pipette is adequate. The screen 114 can also display visual indications 208, 210, 212, 214, 216, 218, and 260 that include a target region for separating the biopsy pipette from the embryo or a target region for where the biopsy pipette can enter the embryo.

[0094] The user interface 112 can include a speaker 116 that is configured to play a sound to deliver messages when prompted by the embryo analysis system 140. For example, the embryo analysis system 140 can cause the user interface 112 to deliver messages such as whether the amount of biomaterial detected within the biopsy pipette is adequate via the speaker 116. In other examples, the system 100 may include a separate speaker and / or a separate user interface communicatively coupled to the embryo analysis system 140 to display or deliver messages.

[0095] As will be described in detail herein, the computer 144 of the embryo analysis system 140 receives one or more images from the camera 142, and the computer 144 processes the one or more images to determine information such as regions depicted in the image, the amount of biomaterial in the biopsy pipette, entry regions, and separation regions. The embryo analysis system 140 provides data representing the information to the user interface 112 for display to an embryologist.

[0096] The embryo analysis system 140 can be implemented as computer programs on one or more computers in one or more locations in which the systems, components, and techniques described below are implemented. The embryo analysis system 140 includes one or more processors and data storage or memory devices that define a machine learning modeland stores instructions executed by the one or more processors. More specifically, the embryo analysis system 140 uses imaging software and / or machine learning to image, process, and identify characteristics of embryos and pipettes. The embryo analysis system 140 analyzes biomaterial within the embryo and the biopsy pipette. In the illustrated example of FIG. IB, the embryo analysis system 140 includes the computer 144, the camera 142, a machine learning model 160, an interpolation engine 162, and a memory 164.

[0097] Referring to FIG. IB, the embryo analysis system 140 is configured to use the camera 142 to obtain image data 130 of a dish 122 that can have one or more drops of liquid. The image data 130 can include a sequence of images with one or more images. The image data 130 can also include video data with one or more frames or images. The camera 142 provides the image data 130 to the computer 144, and the computer 144 utilizes the machine learning model 160 and / or the interpolation engine 162 to process the images and output data related to the images, such as data representing particular regions of the image and / or the amount of biomaterial within the biopsy pipette. The computer 144 can process the output of the machine learning model 160 and / or the interpolation engine 162 to determine indications data 170 to display to the embryologist on a user interface (e.g., user interface 112). For example, indications data 170 can include data representing visual indications 208, 210, 212, 214, 216, 218, and 260 as shown in FIGS. 2A-2B.

[0098] The computer 144 can include memory 164 such as a physical data storage device or a logical data storage area. The embryo analysis system 140 can store data defining the machine learning model 160 and the interpolation engine 162 in the memory 164. The embryo analysis system 140 can also store image data 130 generated by the camera 142 in the memory 164. The embryo analysis system 140 can also store outputs from the machine learning model 160 and the interpolation engine 162 in the memory 164.

[0099] The embryo analysis system 140 is configured to use the camera 142 to obtain image data 130 of a dish 122. For example, the image data can include a video that includes one or more images. The one or more images can depict an embryo and pipettes such as a holding pipette 202 or a biopsy pipette 204 as depicted in FIGS. 2A-2B. In some examples, the embryo analysis system 140 can include the microscope 146 and the camera 142 can obtain an image of the dish 122 through the microscope 146 such that the image of the dish 122 is magnified. The embryo analysis system 140 provides the one or more images of the dish 122 to the computer 144. The computer 144 can receive the image data 130 and provide the image data 130 to the machine learning model 160. In examples where the dish includes more than one drop of liquid, the computer 144 can divide the one or more images intosmaller images that each include one drop of liquid. The computer 144 can provide the smaller images to the machine learning model 160.

[0100] The machine learning model 160 can be configured to process the one or more images generated by the camera 142. For example, the machine learning model 160 can determine regions in the one or more images, such as the holding pipette 202, the biopsy pipette 204, or biomaterial. The machine learning model 160 can determine separation regions at which an embryologist should separate a biopsy from the embryo. The machine learning model 160 can also determine entry regions from which the embryologist should take the biopsy. The machine learning model 160 can distinguish between different types of biomaterial depicted in the image, such as an inner cell mass (ICM), a zona pellucida, an inner boundary of the zona pellucida, trophectoderm cells within the zona pellucida, trophectoderm cells within the biopsy pipette, or a herniation. The machine learning model 160 can also determine a segmentation mask for each region identified by the machine learning model 160. Each segmentation mask can define a bounded area of the image that depicts the identified region, and each segmentation mask can include class data that specifies the identified region (e.g., holding pipette 202, biopsy pipette 204, or type of biomaterial).

[0101] In some implementations, the machine learning model 160 is trained by a training system to determine the amount of biomaterial within the biopsy pipette. For example, the machine learning model 160 can determine the segmentation mask for trophectoderm cells within the biopsy pipette to determine the amount of biomaterial within the biopsy pipette. In some examples, the machine learning model 160 can determine the amount of biomaterial within the biopsy pipette based on a volume of material identified within the biopsy pipette and the magnification level of the microscope 146, the known dimensions of the biopsy pipette, and a segmentation mask size for individual cells in the image. For example, the machine learning model 160 can determine the volume of the biopsy pipette from the known dimensions. The machine learning model 160 can determine a volume of the biomaterial based on the proportion of the biopsy pipette that is filled by the biomaterial. The machine learning model 160 can determine a number of cells of the biomaterial contained within the biopsy pipette based on the magnification level of the microscope 146 and the segmentation mask size for individual cells in the image.

[0102] For example, the machine learning model 160 can determine a volume of the biomaterial within the biopsy pipette, Vsampie, and volume of a model cell, Vceii, in units of pixels3. As an example, the volume of the biomaterial within the biopsy pipette can be determined by calculating the Reimann integral of approximating cylinders in an imagegenerated by camera 142. For example, before performing the biopsy procedure, the volume of the biomaterial within the biopsy pipette, Vsampie, can be set to 0 pixels and the camera 142 can be initialized to begin capturing images of the dish 122. The machine learning model 160 can receive images from the camera 142 throughout the biopsy process and processes each image to divide each image into a series of approximating cylinders. Each approximating cylinder 710 can include one column of pixels, as depicted in FIG. 7. For example, each approximating cylinder 710 in the image can have a height of one pixel.

[0103] FIG. 7 shows a diagram 700 of example approximating cylinders 710 for determining the volume of the biomass within the biopsy pipette 204. For each approximating cylinder 710 in the image, the machine learning model 160 can identify whether the approximating cylinder 710 overlaps the segmentation mask for the biomass within the biopsy pipette. An indication for the segmentation mask for the biomass within the biopsy pipette is depicted as indication 212. If the approximating cylinder 710 does not overlap the segmentation mask for the biomass within the biopsy pipette, then the volume of the biomaterial within the approximating cylinder 710 is estimated as zero. If the approximating cylinder 710 overlaps the segmentation mask for the biomass within the biopsy pipette, such as the example approximating cylinders 710a, 710b, and 710c, the volume of the biomaterial within the approximating cylinder 710 can be estimated using the number of pixels of the approximating cylinder 710 that overlap the segmentation mask for the biomass within the biopsy pipette. For example, the approximating cylinders 710a, 710b, and 710c, have a height, h, of one pixel. Each of the approximating cylinders 710a, 710b, and 710c can have a different diameter, cZcyiinder - The diameter dcyiinder indicates the number of pixels of the approximating cylinder 710 that are within the segmentation mask for the biomass within the biopsy pipette. For example, dcyiinder for a particular approximating cylinder 710 can be the difference between the row number of the pixel at the top of the segmentation mask for the biomass within the biopsy pipette, and the row number of the pixel at the bottom of the segmentation mask for the biomass within the biopsy pipette.

[0104] The volume of each approximating cylinder can be calculated as , andVsampie can be calculated as:all cylinders '7

[0105] The machine learning model 160 can determine a volume of a model cell Vceii by determining the average cell radius r. For example, the cell radius can be determined as aproportion of the pipette width. As another example, the cell radius can be determined as the one half of the average width of segmentation masks of the biopsy cells. The volume of the 4'777’ model cell can thus be calculated as — •

[0106] The machine learning model 160 can then determine a number of cells, ncell, of the biomaterial contained within the biopsy pipette as ncell«

[0107] For example, the known dimensions of the biopsy pipette can be a default value. In some implementations, the embryo analysis system 140 can also receive the dimensions of the biopsy pipette from an embryologist through the user interface 112.

[0108] As an example, the machine learning model 160 can include one or more deep neural networks (DNN) or convolutional neural networks (CNN). The machine learning model 160 can detect proposed regions of interest in the image. For example, the machine learning model 160 can include a region proposal network that detects proposed regions of interest in the image using any of a variety of appropriate methods such as a sliding window method, an image pyramid method, or a selective search algorithm. In some implementations, the dimensions of each proposed region of interest can be scaled to a constant size for downstream processing, for example, classifying each of the proposed regions of interest, or other multi-step downstream processes.

[0109] The machine learning model 160 can then process each proposed region of interest to determine a bounding area for the proposed region of interest, a class assignment, and a probability for the class assignment. For example, the machine learning model 160 can extract features from each proposed region of interest. The machine learning model 160 can include a CNN that has been configured to convert each proposed region of interest into a vector or matrix of features. In some examples, the machine learning model 160 can use the final network prediction as the vector or matrix of features. In some examples, the machine learning model 160 can extract a layer of perceptrons from the CNN to obtain the vector or matrix of features. The machine learning model 160 can then classify each proposed region of interest into a class. The classes can include, for example, a holding pipette class, a biopsy pipette class, or a type of biomaterial class such as an inner cell mass class, a zona pellucida class, etc. For example, the machine learning model 160 can use a statistical or machine learning method such as a support vector machine or logistic regression to classify each ofthe proposed regions of interests. In some examples, the machine learning model 160 can process each proposed region of interest using region of interest pooling to extract features for each proposed region of interest, and determine bounding area, class assignment, and corresponding probability of the class assignment for the proposed region of interest.

[0110] As another example, the machine learning model 160 can determine bounding areas, class assignments, and probabilities for the class assignments for an image using a convolutional neural network. For example, the machine learning model 160 can determine bounding areas, class assignments, and probabilities for the class assignments for an image using a You Only Look Once (YOLO) model.[oni] The machine learning model 160 can generate segmentation masks for one or more of the proposed regions of interest based on the bounding areas for the proposed regions of interest. For example, the machine learning model 160 can generate segmentation masks for proposed regions of interest that have been assigned a certain class assignment, and with the highest probability among regions of interest that were assigned the certain class assignment. The machine learning model 160 can thus generate a segmentation mask for regions such as the holding pipette 202, the biopsy pipette 204, or types of biomaterial such as the inner cell mass or the zona pellucida.

[0112] In some examples, the machine learning model 160 can include multiple models for processing the image data 130 received from the camera 142. For example, the machine learning model 160 can include an image segmentation model that determines segmentation masks for regions such as the embryo, different regions within the embryo, the biopsy pipette, and the biomaterial within the biopsy pipette given an image. The machine learning model 160 can include another model that determines an entry region of the embryo for taking the biopsy given at least the segmentation masks generated by the image segmentation model. The machine learning model 160 can include another model that determines, based at least on the segmentation masks, a separation region of the embryo for separating the biopsy pipette from the embryo. The machine learning model 160 can include another model that determines, based at least on the segmentation masks, the amount of biomaterial contained within the biopsy pipette.

[0113] For each image, the machine learning model 160 can output, for example, data representing the segmentation masks, data representing the amount of biomaterial within the biopsy pipette, data representing the separation region, and data representing the entry region of the embryo for taking the biopsy.

[0114] In some examples, the embryo analysis system 140 can process different images in a sequence of images, such as a video, using different methods. For example, the embryo analysis system 140 can select images from the sequence of images to process using the machine learning model 160 and can determine segmentation masks for one or more regions only in the selected images. The embryo analysis system 140 can process the images that were not selected using a different method that uses less computing power and less computing time than the machine learning model 160. For example, the embryo analysis system 140 can estimate segmentation masks for the one or more regions for the images that were not selected, rather than using machine learning to generate segmentation makes for the images that were not selected.

[0115] As an example, the embryo analysis system 140 can estimate segmentation masks for one or more images using interpolation. The embryo analysis system 140 can provide the image data 130 to the interpolation engine 162. For example, for a given image in a sequence of images of the image data 130 generated by the camera 142, the embryo analysis system 140 can estimate segmentation masks for the given image based on segmentation masks determined by the embryo analysis system 140 for one or more prior images in the sequence of images of the image data 130. For example, the segmentation masks for one or more prior images can have been determined by the machine learning model 160, or estimated by the interpolation engine 162. Each segmentation mask can be defined by multiple points that make up a polygon of the segmentation mask. For example, each point can be represented by a location in an image, e.g., coordinates. Each point can be identified by a unique identifier such that the interpolation engine 162 can track each point across different images. The interpolation engine 162 can predict the location of each of the multiple points for the given image based on the locations of each of the multiple points in prior images. For example, the machine learning model 160 can output segmentation masks for a first image in a video generated by the camera 142. The interpolation engine 162 can compute a first covering, for example a convex hull, for each segmentation mask in the first image and can obtain the points that define the first covering. For a second image in the video, the machine learning model 160 can determine a second covering for each segmentation mask, and obtain points that define the second covering so that the points of the second covering are close to the points of the first covering. Each particular point of the second covering can have the same corresponding identifier as the closest point of the first covering. For a third image in the video, the interpolation engine 162 can estimate the location for each point of the third covering based on the change in position of each point from the first covering to the secondcovering. Each segmentation mask for the third image can thus be defined by estimated locations for each point based on the change in position of each point from the first covering to the second covering of the first and second images.

[0116] In some examples, the interpolation engine 162 can use any appropriate type of interpolation, spline, or regression to estimate the segmentation masks for a given image. For example, the interpolation engine 162 can set the points of the segmentation mask for a given image to have the same locations as the points of a prior segmentation mask.

[0117] The interpolation engine 162 can output data representing the estimated segmentation mask. The machine learning model 160 can process the data representing the estimated segmentation masks to determine the amount of biomaterial within the biopsy pipette, for example.

[0118] In some examples, the computer 144 can downsample the one or more images before providing the one or more images to the machine learning model 160 and / or the interpolation engine 162. The segmentation masks and estimated segmentation masks are thus determined at the downsampled size. The computer 144 can then upsample the segmentation masks for display over the original image. Downsampling the images can reduce the size of the images and decrease the computing resources and time required for the machine learning model 160 and the interpolation engine 162 to process the images.

[0119] The embryo analysis system 140 can process the outputs of the machine learning model 160 and the interpolation engine 162 to determine indications to display to the embryologist, for example, through the user interface 112. For example, the computer 144 can cause the user interface 112 to display the image from the camera 142 and the indications for segmentation masks for the image. For example, the computer 144 can provide data representing the image from the camera 142 and data representing the indications for segmentation masks for the image generated by the machine learning model 160 or the interpolation engine 162 for display by the user interface 112. For example, the computer 144 can provide data representing indications 208, 210, and 212 for the segmentation masks as part of the indications data 170 to the user interface 112.

[0120] Referring to FIG. 2A, the embryo analysis system 140 can determine segmentation masks for regions of the image 200. FIG. 2A shows a holding pipette indication 208 that indicates a segmentation mask for the holding pipette 202, a biopsy pipette indication 210 that indicates a segmentation mask for the biopsy pipette 204, and a biomaterial within the biopsy pipette indication 212 that indicates a segmentation mask for biomaterial 211 within the biopsy pipette. Referring to FIG. 2B, the embryo analysis system140 can determine segmentation masks for regions of the image 250. FIG. 2B shows the holding pipette indication 208, the biopsy pipette indication 210, and the biomaterial within the biopsy pipette indication 212.

[0121] As another example, the computer 144 can process the image data 130 generated by the camera 142 to determine the amount of biomaterial within the biopsy pipette. In addition, the computer 144 can determine whether the amount of biomaterial within the biopsy pipette is less than, within, or greater than a threshold range. The threshold range can represent a recommended amount of biomaterial to take from the embryo during the biopsy process. For example, the threshold range can be defined by 5-10 cells. The threshold range can also be defined by a volume range or a percentage of an adequate target amount of biomaterial. For example, if the amount of biomaterial within the biopsy pipette determined by the machine learning model 160 is less than the threshold range, the computer 144 can determine that the amount of biomaterial is inadequate. If the amount of biomaterial within the biopsy pipette is within the threshold range, the computer 144 can determine that the amount of biomaterial is adequate. If the amount of biomaterial within the biopsy pipette is greater than the threshold range, the computer 144 can determine that the amount of biomaterial is over the threshold range. The computer 144 can provide the data representing the amount of biomaterial within the biopsy pipette and the determination of the amount of biomaterial relative to the threshold range for display by the user interface 112. For example, the computer 144 can provide data representing an amount indication 214 for the amount of biomaterial within the biopsy pipette as part of the indications data 170 to the user interface 112. The computer 144 can also provide data representing a relative amount indication 216, 260 for the determination of the amount of biomaterial relative to the threshold range as part of the indications data 170 to the user interface 112.

[0122] Referring to FIG. 2A, the embryo analysis system 140 can determine that the amount of biomaterial within the biopsy pipette is 110% of the target adequate amount. The embryo analysis system 140 can determine the amount of biomaterial within the biopsy pipette as a percentage of the target adequate amount, a volume, or a number of cells. FIG. 2 A shows the amount indication 214 as “1.10,” or 110% of the target adequate amount, for example. The embryo analysis system 140 can determine whether the amount of biomaterial within the biopsy pipette is within a threshold range. In the example of FIG. 2A, the embryo analysis system 140 can determine that 110% of the target adequate amount is within the threshold range, and, in response, the user interface 112 can display a relative amountindication 216 that indicates that the amount of biomaterial is adequate, as depicted in FIG. 2A.

[0123] In response to determining that the amount of biomaterial 211 within the biopsy pipette is adequate, the embryo analysis system 140 can cause the user interface 112 to display a separation region indication 218 that indicates a separation region where the biopsy pipette 204 and the embryo 206 are in contact. FIG. 2A shows a separation region indication 218.

[0124]

[0125] The image 250 of FIG. 2B is similar to the image 200 of FIG. 2A, but depicts a smaller amount of biomaterial 211 within the biopsy pipette 204. For example, the embryo analysis system 140 can determine that 70% of the target adequate amount is present within the biopsy pipette. FIG. 2B shows the amount indication 214 as “0.7,” or 70% of the target adequate amount, for example. The embryo analysis system 140 can determine whether the amount of biomaterial 211 within the biopsy pipette is within a threshold range. In the example of FIG. 2B, the embryo analysis system 140 can determine that 70% of the target adequate amount is below the threshold range and, in response, the embryo analysis system 140 can cause the user interface 112 to display a relative amount indication 260 that indicates that the amount of biomaterial is inadequate to the user interface 112, as depicted in FIG. 2B. Because the amount of biomaterial contained within the biopsy pipette 204 in FIG. 2B is inadequate, the embryo analysis system 140 does not cause the user interface 112 to display data representing the separation region. FIG. 2B thus does not show a separation region indication 218 that is shown in FIG. 2A.

[0126] In some examples, the embryo analysis system 140 can determine that the amount of biomaterial 211 within the biopsy pipette is above the threshold range, and can cause the user interface 112 to display a relative amount indication 260 that represents there is too much biomaterial 211 within the biopsy pipette to the user interface.

[0127] As another example, the embryo analysis system 140 can determine that the biopsy pipette has not yet entered the embryo and, in response, the embryo analysis system 140 can determine one or more entry regions for the biopsy pipette. The embryo analysis system 140 can cause the user interface 112 to display indications for the one or more entry regions to the user interface 112. For example, the computer 144 can provide data representing an entry indication for each entry region as part of the indications data 170 to the user interface 112.

[0128] The computer 144 can also provide the output of the machine learning model 160 and / or the interpolation engine 162 to other components of the embryo analysis system 140, as described below with reference to FIG. 4.

[0129] Further, the embryo analysis system 140 can detect when an embryo enters or exits the field of view of the camera 142. The embryo analysis system 140 can also detect when a pipette enters or exits the field of view of the camera 142. For example, the embryo analysis system 140 can be configured to process an image from the camera 142 at a regular interval using the machine learning model 160. If the machine learning model 160 generates a segmentation mask for an embryo, the computer 144 can determine that a biopsy process may begin soon, and as a result, can instruct the camera 142 for take one or more actions to capture additional image data 130 related to the biopsy process. For example, in response to detecting the presence of an embryo within the image data 130 generated by the camera 142, the computer 144 can be configured to send instructions to the camera 142 to begin taking video, or to begin taking images at a higher frequency. The computer 144 can also be configured to begin processing images from the camera 142 at a higher frequency. Similarly, if the machine learning model 160 generates a segmentation mask for an embryo 206 and a biopsy pipette 204 or holding pipette 202, the computer 144 can determine that the biopsy process is beginning. As a result, the computer 133 can use the machine learning model 160 and the interpolation engine 162 to perform real-time analysis of the biopsy process, and / or provide real-time feedback to the embryologist. For example, in response to detecting the presence of an embryo 206 and a biopsy pipette 204 or holding pipette 202, the computer 144 can be configured to turn on a laser in preparation for separating the biopsy pipette 204 and the embryo 206.

[0130] If the machine learning model 160 generates a segmentation mask for the biopsy pipette 204 that is at a threshold distance away from the segmentation mask for the embryo 206, and the machine learning model 160 generates a segmentation mask for biomaterial 211 within the biopsy pipette 204, the computer 144 can determine that cells have been extracted from the embryo 206 and the biopsy pipette 204 has separated from the embryo 206. In response to determining that the biopsy pipette 204 has separated from the embryo 206, the computer 144 can be configured to send instructions to the camera 142 to stop recording video, or to take images at a lower frequency. The computer 144 can also store data representing the amount of biomaterial 211 within the biopsy pipette 204 and an identifier for the embryo 206 in memory 164. 1

[0131] The computer 144 can also be configured to start recording the amount of time that has elapsed since the embryo analysis system 140 first determines that an embryo 206 has entered the field of view of the camera 142. The computer 144 can be configured to continue recording the amount of time until the computer 144 determines that cells have been extracted from the embryo 206. If the computer 144 has not determined that cells have been extracted from the embryo 206 and the amount of time that has elapsed is greater than a threshold amount of time that represents an ideal maximum time for the biopsy process, the computer 144 can be configured to provide a prompt to the user interface 112 that indicates the embryo 206 has been exposed for too long of a period of time. For example, the user interface 112 can display an alert or notification that signals to the embryologist that the embryo 206 should be stored.

[0132] FIGS. 2A-2B show example images and indications that can be displayed by a user interface. The image 200 shows a holding pipette 202, a biopsy pipette 204, and an embryo 206. Data representing indications such as the amount indication 214, the biopsy pipette indication 210, the holding pipette indication 208, the biomaterial within the biopsy pipette indication 212, the separation region indication 218, and the relative amount indications 216, 260 can have been provided by the embryo analysis system 140 to the user interface 112 as described above with reference to FIG. IB. Data representing the images 200, 250 can also have been provided by the embryo analysis system 140 so that the indications can be displayed as overlaid with the images. The example images and indications can be displayed by the user interface 112 on the screen 114 of FIG. 1 A, for example.

[0133] FIG. 3 is a flow chart of an example process 300 for analyzing biological material. The process 300 can be performed by the embryo analysis system 140 described above with reference to FIGS. 1 A and IB.

[0134] The system receives one or more images that each depict an embryo and a biopsy pipette (310). The system can receive the one or more images from a camera in real-time during the biopsy process.

[0135] The system processes the one or more images to determine an amount of biomaterial contained within the biopsy pipette (320). The system can process the one or more images in real-time using a machine learning model such as the machine learning model 160 of FIG. IB.

[0136] For example, the system can process the one or more images to identify one or more regions of interest in each image. The regions of interest can include a holding pipette,a biopsy pipette, or biomaterial such as an embryo, an inner cell mass, a zona pellucida, an inner boundary of the zona pellucida, trophectoderm cells within the zona pellucida, trophectoderm cells within the biopsy pipette, or a herniation.

[0137] The system can also process the one or more images to determine, in real-time, a segmentation mask for each region. A segmentation mask can be defined by multiple points that represent locations on an image. Each segmentation mask can cover the regions of an image that correspond to a particular class. The classes can include, for example, a holding pipette, a biopsy pipette, or biomaterial such as an embryo, an inner cell mass, a zona pellucida, an inner boundary of the zona pellucida, trophectoderm cells within the zona pellucida, trophectoderm cells within the biopsy pipette, or a herniation.

[0138] The amount of biomaterial contained within the biopsy pipette can be defined by a volume, a number of cells, or a percentage of an adequate target amount. For example, the system can process the one or more images to determine a volume of biomaterial contained within the biopsy pipette, a number of cells of biomaterial contained within the biopsy pipette, or a percentage of an adequate target amount of the biomaterial contained within the biopsy pipette.

[0139] In some examples, the one or more images can be part of a sequence of images such as a video. The system can select two or more images in the sequence of images to process using the machine learning model. For example, the system can process, in real-time using the machine learning model, the selected images to determine an amount of biomaterial contained within the biopsy pipette by determining a segmentation mask for each of one or more regions in each of the selected images. For example, each region can depict a holding pipette, the biopsy pipette, or biomaterial. The system can identify one or more images in the sequence as non-selected.

[0140] The system can process each of the one or more non-selected images in the sequence of images to estimate a segmentation mask for each of the one or more regions determined for the selected images. For example, for a particular non-selected image, the system can estimate a respective location for each of multiple points that define the segmentation mask based on the respective location for each of the multiple points in one or more prior images in the sequence of images. For example, the respective location for each of the multiple points in one or more prior images can be determined from the segmentation mask for each region for the one or more prior images. Each segmentation mask can have been generated by the machine learning model 160, or an interpolation engine 162 as described above with reference to FIG. 1.

[0141] The system can cause a user interface to display an indication of the amount of biomaterial within the biopsy pipette (330). The system can cause the user interface to display the indication in real-time. For example, the system can provide data representing the indication of the amount of biomaterial to the user interface. The system can also provide data representing the image that the system processed to determine the amount of biomaterial so that the indication can be overlaid with the image.

[0142] The system can also cause the user interface to display an indication of each segmentation mask. For example, the system can provide data representing each segmentation mask to the user interface. In examples where the system estimates segmentation masks for one or more images, the system can cause the user interface to display an indication of each segmentation mask or each estimated segmentation mask. For example, the system can provide data representing each segmentation mask or estimated segmentation mask to the user interface.

[0143] The system can further determine, in real-time, whether the amount of biomaterial contained within the biopsy pipette is acceptable based on processing the one or more images. For example, the system can compare the amount of biomaterial contained within the biopsy pipette with a predetermined range. The predetermined range can be defined by volume (e.g., pL), for example, a number of volumetric pixels (pixels3), number of cells, or proportion of an adequate target amount, for example. For example, the predetermined range can be between 5 and 10 cells, or between 67% and 133% of the adequate target amount.

[0144] The system can determine that the amount of biomaterial contained within the biopsy pipette is below the predetermined range. That is, the amount of biomaterial contained within the biopsy pipette is less than the lower end of the predetermined range. In response, the system can cause the user interface to display an indication that the amount of biomaterial contained within the biopsy pipette is below the predetermined range. An example indication is shown above with reference to FIG. 2B.

[0145] The system can determine that the amount of biomaterial contained within the biopsy pipette is above the predetermined range. That is, the amount of biomaterial contained within the biopsy pipette is greater than the higher end of the predetermined range. In response, the system can cause the user interface to display an indication that the amount of biomaterial contained within the biopsy pipette is above the predetermined range.

[0146] The system can determine that the amount of biomaterial contained within the biopsy pipette is within the predetermined range. That is, the amount of biomaterial contained within the biopsy pipette is in between the lower end and the higher end of thepredetermined range. In response, the system can cause the user interface to display an indication that the amount of biomaterial contained within the biopsy pipette is within the predetermined range. An example indication is shown above with reference to FIG. 2A.

[0147] In examples where the system determines that the amount of biomaterial contained within the biopsy pipette is within the predetermined range, the system can further determine, in real-time, a separation region where the biopsy pipette and the embryo are in contact based on processing the one or more images. For example, the system can determine a region where the segmentation masks for the biopsy pipette and the embryo are within a threshold distance to each other or meet each other as the separation region. The system can cause the user interface to display an indication of the separation region. An example indication of the separation region is shown above with reference to FIG. 2A. In some examples, the system can cause the user interface to display a laser guide at the separation region.

[0148] The system can further determine that the biopsy pipette has separated from the embryo and that the amount of biomaterial within the biopsy pipette is non-zero. In response to determining that the amount of biomaterial within the biopsy pipette is non-zero, the system can record the amount of biomaterial within the biopsy pipette and an identifier for the embryo.

[0149] In some examples, the system can determine one or more entry regions of the embryo for taking the biopsy by processing, in real-time, the one or more images using the machine learning model 160 of FIG. IB. Each of the one or more entry regions can be a region of the embryo that is conducive to obtaining trophectoderm cells without damaging the inner cell mass. For example, an entry region can be a herniation of the embryo or a region of the embryo that is away from the inner cell mass of the embryo. The system can further cause, in real-time, the user interface to display an indication of the one or more entry regions. For example, the system can provide data representing the one or more entry regions to the user interface.

[0150] In some examples, the system can determine that the biopsy pipette has entered the embryo at a region other than the one or more entry regions and provide feedback to the embryologist. For example, the system can determine that the segmentation mask for the biopsy pipette and the segmentation mask for the embryo are within a threshold distance at a region that is not near one of the entry regions. The system can cause, in real-time, the user interface to display an alert that the position of the biopsy pipette should be adjusted.

[0151] In some examples, the system can perform a validation process for the accuracy of the machine learning model 160. For example, the system can use the machine learning model 160 to determine the amount of biomaterial within the biopsy pipette 204. After the biopsy pipette 204 is separated from the embryo, the biomaterial within the biopsy pipette 204 can be removed from the biopsy pipette for further processing. For example, an embryologist or an embryo analysis system 400 described with reference to FIG. 4 can perform further processing. For example, the biomaterial may be tested to quantify the number of cells present using a stain such as DAPI. The biomaterial may also be tested to determine the amount of DNA present using a quantitative PCR (qPCR).

[0152] An example of staining to quantify the number of cells present is shown in FIG. 5. FIG. 5 shows an image 500 of the biomaterial that has been removed from an embryo using a biopsy pipette, and removed from the biopsy pipette for further testing. FIG. 5 shows an image 550 of the same biomaterial that is depicted in image 500, with a stain that more clearly delineates individual cells 560a-f.

[0153] The system can validate the accuracy of the machine learning model 160 by comparing the output from the machine learning model 160 representing the amount of biomaterial removed from the embryo with the quantity of biomaterial removed from the embryo that is determined through standard testing techniques, such as staining. In some examples, the system may determine that the accuracy of the machine learning model 160 does not meet a threshold accuracy based on comparing the output of the machine learning model 160 with standard testing techniques, and, in response, the system can re-train the model, for example, using the training system described below, on new or previously unseen training data.

[0154] The machine learning model can have any appropriate machine learning model architecture that enables the machine learning model to perform its described functions. For instance, the machine learning model can be implemented, for example, as a neural network model, or a random forest model, or a support vector machine model, or a decision tree model, or a linear regression model, etc. In implementations, where the machine learning model is implemented as a neural network model, the machine learning model can include any appropriate types of neural network layers (e.g., fully connected layers, convolutional layers, attention layers, etc.) in any appropriate number (e.g., 5 layers, 10 layers, or 50 layers) and connected in any appropriate configuration (e.g., as a linear sequence of layers). In implementations where the machine learning model is implemented as a decision tree model,the machine learning model can include any appropriate number of vertices, and can implement any appropriate splitting function at each vertex.

[0155] The machine learning model can include a set of machine learning model parameters. For instance, for a machine learning model implemented as a neural network model, the set of machine learning model parameters can define the weights and biases of the neural network layers of the machine learning model. As another example, for a machine learning model implemented as a decision tree, the set of machine learning model parameters can define parameters of a respective splitting function used at each vertex of the decision tree. To generate a model output, the machine learning model can process a model input in accordance with values of the set of machine learning model parameters.

[0156] A training system can train a machine learning model on a set of training data. More specifically, the training system can determine trained values of the set of machine learning model parameters of the machine learning model by a machine learning training technique.

[0157] The training system uses a training engine to train the set of machine learning model parameters of the machine learning model on a set of training data. The training engine can train the machine learning model using any machine learning training technique appropriate for the architecture of the machine learning model. For instance, if the machine learning model is implemented as a neural network model, then the training engine can train the machine learning model using stochastic gradient descent.

[0158] The training system can receive a set of training data to train the machine learning model. The training data can include multiple training outputs. Each training output can include a labeled image. Each labeled image can include an image depicting an embryo and a biopsy pipette and one or more labels corresponding to one or more regions in the labeled image. For example, the labeled image can include labels corresponding to the region of the embryo, the region of the holding pipette, and the region of the biopsy pipette. The one or more labels can also represent an amount of biomaterial within the biopsy pipette. For example, one of the labels can correspond to the region that depicts trophectoderm cells within the biopsy pipette, and can include information representing the amount of biomaterial within the pipette, such as the number of cells.

[0159] The one or more labels can have been annotated by an embryologist, for example. The one or more labels can also have been annotated by the training system. For example, the training system can receive a video of a biopsy process performed by an embryologist.The training system can label the region of the embryo that the embryologist chose to take the biopsy from as the entry region in one or more images that depict the embryo from the video.

[0160] The training data can also include multiple training inputs. Each training input can correspond to a training output. Each training input can include an unlabeled image that includes the image of the corresponding training output. That is, each training input includes the same image as the corresponding training output, without the labels of the training output.

[0161] For each training input and corresponding training output, the training engine trains the machine learning model to process the training input to generate a model output that matches the training output. For example, given a training input, the machine learning model can generate a model output that includes labels predicted by the machine learning model for the unlabeled image of the training input. More specifically, the training engine trains the machine learning model, by a machine learning training technique, to optimize an objective function that measures an error between: (i) the labels of the model output generated by the machine learning model for the training input, and (ii) the labels of the training output. The objective function can measure the error between a model output and a training output in any appropriate way, e.g., as a squared error or as an absolute error.

[0162] The training system can thus train the machine learning model to detect one or more regions in an image. The training system can also train the machine learning model to depict the amount of biomaterial within the biopsy pipette.

[0163] In some examples, the training system can perform techniques to prevent the machine learning model from overfitting to the set of training data, such as regularization and / or pruning.

[0164] In some examples, the training system can perform data augmentation to generate additional training inputs and corresponding training outputs. For example, the training system can perform modifications on the received set of training data to generate additional training inputs and corresponding training outputs. For example, the training system can rotate, flux, degrade, mutate, or inject noise into the images.

[0165] In some examples, the set of training data can include training inputs and training outputs with different types of images. For example, the images can include images taken under a variety of conditions, such as lighting, image resolution, scale factors, quality of the imaging chip, or type of the imaging chip. With a variety of types of images, the training system can train the machine learning model to be more robust to different conditions that may exist in the system 100 of FIG. 1 A.

[0166] The images can also include images of different embryos and pipettes. With a variety of embryos and pipettes, the training system can train the machine learning model to be more robust to previously unseen embryos and pipettes that may be analyzed.

[0167] A number of implementations have been described. While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what is being claimed, which is defined by the claims themselves, but rather as descriptions of features that may be specific to particular implementations of particular inventions. It will be understood that various modifications may be made.

[0168] In some examples, the embryo analysis system 140 can be used as a training tool for embryologists. For example, the embryo analysis system 140 can provide an estimate of a skill level of an embryologist. For example, the embryo analysis system 140 can record the actions of the embryologist, such as the region of the embryo that the embryologist took the biopsy from and the amount of biomaterial that the embryologist removed from the embryo. The embryo analysis system 140 can compare the region of the embryo that the embryologist took the biopsy from with the one or more entry regions determined by the embryo analysis system 140. The embryo analysis system 140 can determine whether the amount of biomaterial removed from the embryo is within the predetermined threshold. The embryo analysis system 140 can estimate the skill level of the embryologist based on how close the region that the embryologist selected to take the biopsy is to the entry regions determined by the embryo analysis system 140, and / or based on whether the amount of biomaterial removed is within the predetermined threshold, and / or based on how far the amount of biomaterial is from the predetermined threshold. The embryo analysis system 140 can also suggest areas of improvement, such as choosing a region of the embryo to take the biopsy from, to the embryologist. For example, the embryo analysis system 140 can provide data representing the embryologist’s skill level and suggestions for areas of improvement to the user interface for display.

[0169] In some examples, the embryo analysis system 140 can generate artificial biopsy sessions for training embryologists. For example, the embryo analysis system 140 can generate an image or a sequence of images of embryos. For example, the embryo analysis system 140 can generate images for artificial biopsy sessions using a generative model such as a diffusion model, a transformer-based model, or a Generative Adversarial Network (GAN). The images can have been generated based on training data for the machine learning model 160 or from images captured during previous biopsy sessions. The embryo analysis system 140 can provide data representing the generated images to the user interface 112, forexample, that is positioned in the field of view of the camera and so that the embryologist can simulate interacting with the embryo. For example, the user interface 112 can include a screen that is positioned in the field of view of the camera, or a projection of the image that is positioned in the field of view of the camera. The embryologist can practice using the manipulators to adjust the position of the holding pipette and the biopsy pipette, or practice adjusting the position of the laser, for example.

[0170] In some examples, the embryo analysis system 140 can be configured to identify stages of the biopsy process and record data representing each stage. For example, the embryo analysis system 140 may begin recording (for example, begin storing images captured by the camera 142 and related data) when an embryo is detected in the field of view of the camera. The embryo analysis system 140 can identify the frame and timestamp when the embryo is detected as a first stage of the biopsy process. As another example, the embryo analysis system 140 may also detect when a biopsy pipette is in the field of view of the camera. The embryo analysis system 140 can identify the frame and timestamp when the biopsy pipette is detected as a second stage of the biopsy process. The embryo analysis system 140 can record the frames between the start of the first stage and the start of the second stage as belonging to the first stage. The embryo analysis system 140 can record the frames between the start of the second stage and the start of a third stage as belonging to the second stage. The data representing each stage can be stored in memory of the computer 144, for example. The data representing each stage can be used to score the performance of the embryologist performing the biopsy process, for example. An example calculation of a score for the performance of the embryologist performing the biopsy process, measured as the total reward, is described below with reference to FIG. 9. As another example, the data representing each stage can be used in an auditing process. In some implementations, the data representing each stage can be used to further train and improve the machine learning model to detect stages of a biopsy procedure (e.g., when a biopsy procedure is started).

[0171] In another example, the embryo analysis system 140 can include a microscope with an integrated graphical overlay that provides feedback and guidance while viewing the dish 122 through the microscope. Specifically, graphical overlay can incorporate augmented reality (AR) technology. For example, the microscopes 146 can incorporate AR by providing a transparent screen or beam splitter disposed between an embryologist’s eye and what is being read with the microscope 146. The AR technology can be coupled with the camera 142 to allow the embryologist to see the outputs of the machine learning model 160 without needing to look up from the microscope 146. In another example, the embryologist could usea microscope configured with a display screen instead of eyepieces. In this case, graphical information could be overlaid onto that display screen.

[0172] In some examples, the user interface can be a user interface of the computer. For example, the computer can include a user interface that includes a screen for displaying images and indications.

[0173] FIG. 4 is a block diagram of another example embryo analysis system 440. The embryo analysis system 440 is similar to the embryo analysis system 140 described above with reference to FIG. IB, but is configured to automatically carry out the biopsy process. The embryo analysis system 440 is also referred to herein as an agent. The agent 440 is configured to automatically carry out the biopsy process by interacting with a biopsy environment that the biopsy process is performed in. The biopsy environment includes, for example, equipment such as the dish 122, a holding pipette, and a biopsy pipette. The environment can also include the embryo on the dish 122.

[0174] The embryo analysis system 440 also includes robotic components 408. The robotic components 408 can include a laser 410 and manipulators 412. The laser 410 can be configured to be directed at a certain location, e.g., coordinates, and to be activated to separate an embryo and a biopsy pipette inserted into the embryo. The manipulators 412 can be configured to move equipment such as the dish 122, the holding pipette, or the biopsy pipette. The manipulators 412 can also be configured to move the laser 410, or the microscope 146. Referring to FIG. 1 A, manipulators 412 are depicted as being directed at the dish 122.

[0175] In some examples, the robotic components 408 can be controlled by an operator or by the embryo analysis system 440. For example, the operator and the embryo analysis system 440 can share control of the robotic components 408 during the biopsy process.

[0176] The computer 144 of the embryo analysis system 440 also includes an instruction engine 402. The instruction engine 402 is configured to generate instructions 480 for robotic components 408 based on the outputs of the machine learning model 160 and the interpolation engine 162.

[0177] For example, the machine learning model 160 may output data representing a separation region. The instruction engine 402 can generate instructions 480 that cause the laser 410 to be directed at the separation region. For example, the instruction engine 402 can determine coordinates that correspond to the location of the separation region on the dish, and provide the coordinates to the laser 410. After the laser 410 is directed at the separation region, the instruction engine 402 can generate instructions 480 that activate the laser 410 andprovide the instructions 480 to the laser 410 to separate the biomaterial in the pipette from the rest of the embryo being sampled. In some examples, after the laser 410 is directed at the separation region, the computer 144 may cause a confirmation message to be displayed at a user interface 112. The computer 144 can receive an input from the embryologist that indicates the embryologist would like to activate the laser. The instruction engine 402 can then generate instructions 480 that activate the laser 410 and provide the instructions to the laser 410.

[0178] As another example, the machine learning model 160 may output data representing one or more entry regions, and in response to receiving the entry regions, the instruction engine 402 can generate instructions 480 that cause the manipulators to move the biopsy pipette to enter the embryo at one of the one or more entry regions. For example, the instruction engine 402 can determine coordinates that correspond to the location of one of the entry regions on the embryo. The instruction engine 402 can generate and provide to the manipulators 412 instructions 480 that cause the manipulators 412 to move the biopsy pipette so that the tip of the pipette is directed to the entry region on the embryo and enters the embryo at the entry region. The instruction engine 402 can generate and provide to the manipulators 412 instructions 480 that cause the manipulators 412 to suction biomaterial from the embryo.

[0179] An operator can adjust a stage of the embryo analysis system 400 to bring a dish positioned on the stage within a field of view of the camera 142. Once the dish is positioned within a field of view of the camera 142, the instruction engine 402 can generate and provide instructions 480 to the manipulators 412 to move a holding pipette to correctly orient the embryo into a final position for biopsy.

[0180] In some examples, the instruction engine 402 can include a machine learning model that has been trained to generate instructions 480 for the robotic components 408. For example, the machine learning model can be trained on training data generated by recording the actions of embryologists. For example, the robotic components 408 can include sensors that record the direction and magnitude of force applied to the controls for the robotic components 408 as the embryologist manipulates the controls to position the holding and biopsy pipettes. The training data can also include videos or sequences of images of the holding and biopsy pipettes as they are positioned. The training system for the instruction engine 402 can store the videos and data representing the direction and magnitude of force associated with the video in memory of the computer 144, for example.

[0181] In some examples, the instruction engine 402 can generate instructions 480 for the robotic components 408 that perform particular actions. In these examples, the instruction engine 402 can generate the instructions 480 based on the outputs of a machine learning model that has been trained to generate actions based on a virtualization of the environment of the biopsy process. In some examples, the embryo analysis system 140 can use techniques such as dynamic programming to generate the instructions 480.

[0182] The virtualization of the biopsy environment is a virtual representation of the biopsy environment. At each of multiple time points during the biopsy process, the computer 144 can generate the virtualization of the environment to represent the biopsy environment at the respective time point during the biopsy process. In some examples, the virtualization can depict segmentation masks for one or more regions of interest in an image of the biopsy environment. For example, the virtualization can depict an embryo and a biopsy pipette using segmentation masks for the embryo and the biopsy pipette. In some examples, the virtualization can depict an image of the biopsy environment and segmentation masks for one or more regions of interest in the image. For example, the virtualization can depict an embryo and a biopsy pipette using an image of the embryo and the biopsy pipette and segmentation masks for the embryo and the biopsy pipette.

[0183] FIG. 8 shows an example virtualization 800 of the biopsy environment at a particular time point during the biopsy process. For example, the particular time point is shown in FIG. 8 as 7 seconds following the initiation of the biopsy process and is prior to removing a biopsy sample from the embryo. For example, the environment includes an embryo, a biopsy pipette, and a holding pipette and the virtualization 800 depicts a representation 802 of the embryo, a representation 804 of the biopsy pipette, and a representation 806 of the holding pipette.

[0184] FIG. 8 also displays a current score, “100”, for the biopsy session that can be displayed with or overlaid with the virtualization 800, e.g., in the user interface 112. An example calculation of the score is described below with reference to FIG. 9. FIG. 8 also displays the particular time point, “0:00:07,” that can be displayed with or overlaid with the virtualization 800, e.g., in the user interface 112.

[0185] FIG. 8 also displays data representing the current state of the holding pipette and the biopsy pipette, that can be displayed with or overlaid with the virtualization 800, e.g., in the user interface 112. For example, FIG. 8 displays an indication 814 for the position and an indication 824 for the suction level for the biopsy pipette. FIG. 8 also displays an indication 816 for the position and an indication 826 for the suction level for the holding pipette.

[0186] In some examples, the computer 144 can generate the virtualization 800 from an image of the biopsy environment. For example, the computer 144 can use a segmentation machine learning model such as the machine learning model 160 described above to generate the virtualization 800. To generate the virtualization 800 from an image, the computer 144 can obtain the image of the biopsy environment. In the example of FIG. 8, the image can depict the embryo, the biopsy pipette, and the holding pipette, and the virtualization 800 generated based on the image can depict a representation 802 of the embryo, a representation 804 of the biopsy pipette, and a representation 806 of the holding pipette. In some examples, the computer 144 can receive the image used to generate the virtualization 800 from the camera 142. In some examples, such as for training, the computer 144 can obtain stored or simulated images of a biopsy environment and the virtualization 800 can be generated based on the stored or simulated images of a biopsy environment, as described further below.

[0187] In examples in which the computer 144 generates the virtualization from an image, the computer 144 can display annotations, e.g., shading, for one or more regions in the image. For example, the computer 144 can generate a feature segmentation map for the image by processing the image using a trained machine learning model (e.g., machine learning model 160 of FIG. 4) to identify one or more regions of interest in the image and generate one or more segmentation masks for each of the one or more regions of interest in the image. The regions of interest determined by the machine learning model can include, for example, the holding pipette, the biopsy pipette, biomaterial, a separation region, or an entry region. The computer 144 can use the machine learning model 160, also referred to as a segmentation machine learning model, to generate the feature segmentation map. For example, the computer 144 can provide the image as input to the segmentation machine learning model to generate the segmentation masks of the feature segmentation map.

[0188] The computer 144 can display the feature segmentation map in the virtualization 800 of the biopsy environment. For example, the computer 144 can render each segmentation mask in the virtualization and the virtualization 800 can display the regions of the feature segmentation map as annotated or shaded over the image of the biopsy environment. In the example of FIG. 8, indications for biomaterial such as an indication 810 for the inner cell mass (ICM), an indication 812 for the herniation, and an indication 813 for a biopsy-pipette exclusion zone (a region near the ICM) are displayed as shaded in different colors. In some examples, different regions are displayed as shaded in different shades of the same color. In some examples, the computer 144 can scale each segmentation mask to a standard size.

[0189] In some implementations, the image is processed by the machine learning model 160 to identify one or more portions of the image relevant to the biopsy process, and the virtualization 800 displays the relevant portions of the image identified by the machine learning model 160. For example, the machine learning model 160 can process the image to generate a feature segmentation map. The computer 144 can identify a portion of the image that includes the embryo using the feature segmentation map. In response, the computer 144 can display the identified portion in the virtualization 800 of the biopsy environment. For example, the computer 144 can transfer or copy data representing the identified portion from the image for inclusion in the virtualization 800.

[0190] In some examples, the computer 144 can generate the virtualization 800 based on data indicating a state of one or more of elements of the environment, such as the embryo, holding pipette, biopsy pipette, laser, or dish. For example, the computer 144 can maintain data representing a current state, e.g., location and orientation, of an embryo. The computer 144 can also maintain data representing a current state, e.g., location, orientation, and suction level, of the biopsy pipette and the holding pipette.

[0191] The computer 144 can obtain data representing an initial state of the embryo, biopsy pipette, and holding pipette. For example, the computer 144 can obtain the data from an image depicting the embryo, biopsy pipette, and holding pipette. During the biopsy process, as the state of the biopsy pipette and the holding pipette change, the computer 144 can use data indicating the state of the biopsy pipette and the holding pipette to generate the virtualization. For example, as the robotic components 408 or the operator manipulates the biopsy pipette and the holding pipette, the computer 144 can receive data indicating the new state of the biopsy pipette or the holding pipette. As an example, the data indicating the new state of the biopsy pipette or the holding pipette can be derived from the data representing the direction and magnitude of force applied by the robotic components 308 or the operator to the controls of the biopsy pipette or the holding pipette. The computer 144 can identify a change in the state of the biopsy pipette or the holding pipette between the new state and the current state. The computer 144 can generate the virtualization of the environment using the change in the state of the biopsy pipette or the holding pipette. For example, if the computer 144 determines that the biopsy pipette has moved, the computer 144 can generate or update the virtualization 800 to reflect the new position or orientation of the biopsy pipette within the biopsy environment. For example, for a first time point, the computer 144 can generate a virtualization. For subsequent time points, the computer 144 can update the virtualization for a preceding time point. If the computer 144 determines that the biopsy pipette has movedsuch that the biopsy pipette overlaps with the embryo at a particular location, the computer 144 can update the virtualization 800 to reflect that the biopsy pipette has entered the embryo at the particular location. If the computer 144 determines that the suction level of the biopsy pipette has increased, the computer 144 can update the virtualization 800 to reflect that the biopsy pipette contains an amount of biomaterial corresponding to the detected suction level. In some examples, the computer 144 can identify a change in the state of the embryo from an image depicting the embryo. In some examples, the computer 144 can maintain a mapping of a change in position of the biopsy or holding pipettes to a rate of change in movement for the biopsy or holding pipettes in the virtualization.

[0192] At each of the multiple time points, the computer 144 can process the virtualization of the biopsy environment using a machine learning model, also referred to as an action machine learning model, to generate a model output that includes an action to be performed. As an example, the action machine learning model be trained using reinforcement learning to generate the model output. In some implementations, the computer 144 causes an action to be performed at each time point to change the biopsy environment.

[0193] An example action machine learning model 900 is shown in FIG. 9. The action machine learning model processes a virtualization of a biopsy environment (e.g., virtualization 800) at a particular time point, for example to generate a model output for the time point. In some examples, the virtualization depicts an image of the biopsy environment and segmentation masks for regions of interest in the image. In some examples, the virtualization depicts segmentation masks, e.g., generated from an image of the biopsy environment.

[0194] The action machine learning model 900 can have any of a variety of architectures, such as a convolutional neural network. For example, the action machine learning model 900 includes one or more convolutional layers 910, fully connected layers 920, and softmax layers 930. Each of the convolutional layers such as convolutional layer 910a and convolutional layer 910b generates an output representing features of an input for the convolutional layer. For example, the input for a particular convolutional layer can include the biopsy virtualization, or the output of a preceding convolutional layer. In some examples, the action machine learning model 900 can also include one or more pooling layers that reduce the dimensions of feature maps generated by one or more convolutional layers 910. The fully connected layers 920 apply a transformation to the output from the last convolutional layer or the last pooling layer. The softmax layer 930 performs a softmaxactivation function on the output of the last fully connected layer to generate a model output. In some examples, the action machine learning model 900 can have other network layouts.

[0195] In some implementations, the model output generated by the machine learning model 900 includes a probability distribution for multiple possible actions that can be taken to modify the biopsy environment. The action machine learning model 900 can be configured to generate a model output that maximizes a total reward for the current state of the biopsy environment at a particular time point as represented by the virtual representation. Thus, the action machine learning model 900 can be trained to construct optimal policies by rewarding actions that result in better quality biopsies with higher reward scores. The biopsy quality can be tracked by reward scores that reward, for example, gentle handling of the embryo, faster biopsy completion, the proper placement of pipettes, and extraction of the target amount of material, as well as other metrics.

[0196] The total reward score assigned to a biopsy procedure analyzed by the machine learning model 900 can be determined based on one or more reward scores for one or more respective characteristics of the biopsy procedure. As an example, the total reward score assigned to a biopsy procedure can be determined based on a weighted sum of reward scores assigned to certain characteristics of the biopsy procedure. An example total reward is calculated as: score = a * (s- + a2* (s2) + a3* (s3) + a4* (s4) where a higher score indicates a higher quality biopsy.

[0197] is the time to complete the biopsy, where a smaller number indicates a higher quality biopsy. s2is a score monotonic with respect to proximity of the location that the biopsy pipette entered the embryo to the recommended entry regions of the embryo, such as a herniation. s3is the number of times the embryo was touched by the pipettes. s4is a score monotonic with respect to proximity of the holding pipette to the recommended region for holding pipette placement. The coefficients atare set to positive or negative weights such that events corresponding to an improved biopsy procedure result in higher scores. In some examples, each coefficient is a default value.

[0198] In some examples, the total reward is further based on other reward scores. For example, other reward scores can include a score that measures an amount of force used on a pipette, or the force of vacuum used in the pipettes.

[0199] The computer 144 selects the action 902 to be performed from the possible actions of the model output. As an example, the computer 144 can select the action with a highest probability from the probability distribution generated by the machine learning model 900.

[0200] In response to the machine learning model 900 determining an action 902 to be performed in the biopsy environment, the computer 144 can cause the operator of the biopsy environment and / or one or more robotic components 408 of the biopsy environment to perform the selected action 902 by providing data representing the action to the instruction engine 402. The instruction engine 402 can generate instructions 480 that instruct an operator of the biopsy environment to perform the selected action 902 and / or can provide the instructions 480 to the robotic components 408 to perform the selected action 902. For example, the action 902 determined by the machine learning model 900 can include, but is not limited to, one or more of moving the biopsy pipette, moving the holding pipette, moving the dish 122, moving a stage supporting the embryo, moving the laser 410, or moving the microscope 146.

[0201] In the example of FIG. 9, the action 902 includes adjusting the manipulator for the holding pipette and / or the biopsy pipette. For example, the action 902 can adjust the position and / or suction level of a left manipulator, e.g., for the holding pipette, and a right manipulator, e.g., for the biopsy pipette. As an example, the computer 144 can control the manipulators 412 to move the biopsy pipette to contact the embryo at a region of the embryo. As another example, the computer 144 can control the manipulators 412 to cause suction to be applied to the embryo through the biopsy pipette. As another example, the computer 144 can control the manipulators 412 to cause suction to be applied to the embryo through the holding pipette. As another example, the computer 144 can control the manipulators 412 or the laser 410 to cause the laser 410 to be directed at a region of the embryo. The computer 144 can further cause the laser 410 to be activated while the laser is directed at the region of the embryo.

[0202] In some implementations, the actions can include translating the stage and dish into an area accessible to the pipettes. For example, the agent can adjust controls such as a coarse control lever and fine control lever to move the stage and dish 122 so that the embryo is within the area accessible to the pipettes and in the field of view of the microscope.

[0203] In some implementations, the actions can include inspecting the biopsied material, for example, for quality assurance. In some implementations, the actions can include controlling the pipette so that the pipette is properly equilibrated and has proper suction for transferring and releasing the biomaterial into a sample tube.

[0204] In some examples, the computer 144 can use a different machine learning model than the action machine learning model to determine certain types of actions. As an example, the computer 144 can use a different machine learning model to determine actions such astransferring and releasing the biomaterial into a sample tube. For example, the different machine learning model can have a similar architecture as the machine learning model 900 described above, but can be configured to generate a model output that maximizes a total reward that is based on reward scores assigned to characteristics of a procedure for putting biopsied material into a tube.

[0205] The action machine learning model can be trained on a training dataset that includes training virtual environments and training model outputs. For example, a training system can generate a training virtual environment that is a virtualization generated based on stored images or simulated images. In some examples, the training system can obtain the image from videos stored in memory of the computer 144 of previous biopsy sessions conducted by an operator, the system 440, or another agent. In some examples, the virtualization for the training environment can be generated based on a simulated image. For example, the computer 144 can generate the image using a machine learning model that is configured to generate images that include at least an embryo and a biopsy pipette. As another example, the computer 144 can generate the simulated image using a simulated 3- dimensional model of an embryo. The 3 -dimensional model of the embryo can be randomized for different features of the embryo such as the location and morphology of the zona pellucida, ICM, herniation, etc. The computer 144 can generate the simulated image to include a focal plane that bisects the embryo so that the features of the embryo are visible in the resulting two-dimensional view of the embryo. In examples where the computer 144 generates the training environment virtualization based on a simulated image with a simulated model of the embryo, the computer 144 can display annotations for the feature segmentation map of the virtual representation based on data representing the simulated 3- dimensional model of the embryo.

[0206] FIG. 10 is a flow chart of an example process 1000 for controlling an agent interacting with an environment during a biopsy process. The process 1000 can be performed by the embryo analysis system 440 and the machine learning model 900 described above with reference to FIGS. 4, 8, and 9.

[0207] The method includes generating a virtualization (e.g., virtualization 800 of FIG. 8) of a biopsy environment at a particular time point during a biopsy process (1010). The virtualization depicts at least the embryo and the biopsy pipette at the respective time point. In some implementations, the system generates a virtualization of the biopsy environment in response to obtaining an image of the biopsy environment. In some implementations, the system obtains the image by receiving the image implementations of the biopsy environmentfrom a camera. In some implementations, the image is a simulated image. In some examples, the system can generate the simulated image.

[0208] In these implementations, generating the virtualization includes generating a feature segmentation map for the image that includes one or more segmentation masks. For example, a segmentation machine learning model (e.g., machine learning model 160 of FIG. 4) can be used to generate data representing the one or more segmentation masks for the image of the biopsy environment. The feature segmentation map can be displayed in the virtualization of the environment. In some of these implementations, a relevant portion of the image that includes the embryo can be identified (e.g., by a machine learning model) based on the feature segmentation map. The identified relevant portion of the image can be displayed in the virtualization of the environment.

[0209] In some implementations, the virtualization of the environment is generated from data indicating a state of one or more components of the system. For example, data indicating a state of the biopsy pipette can be received and a change in the state of the biopsy pipette can be identified based on the data. The virtualization of the environment can be generated using the change in the state of the biopsy pipette. As another example, data indicating a state of the holding pipette can be received and a change in the state of the holding pipette can be identified based on the data. The virtualization of the environment can be generated or updated using the change in the state of the holding pipette.

[0210] The virtualization of the environment is processed using a machine learning model to generate to generate a model output that includes an action to be performed by an agent of the biopsy environment (1020). For example, the virtualization of the biopsy environment can be processed using the action machine learning model described above to generate a model output that includes an action to be performed by an agent of the biopsy environment. The model output generated by the machine learning model can include a probability distribution for multiple possible actions. In some implementations, the machine learning model is configured to generate a model output that maximizes a total reward for the biopsy procedure being performed in the biopsy environment. For example, the total reward can be a weighted average of one or more reward scores for one or more respective characteristics of the biopsy procedure being performed in the biopsy environment. In some implementations, the reward score can be based on, for example, any one or more of: an amount of biomaterial contained within the biopsy pipette, a time elapsed for the biopsy process, a proximity of a sample locus to one or more identified sampling regions of theembryo, a number of times the embryo was touched by the biopsy pipette, and a proximity of a holding pipette to one or more identified holding placement regions.

[0211] Based on the model output generated by the machine learning model, an agent of the biopsy environment is caused to perform the action (1030). For example, the action is selected from the multiple possible actions of the model output. Instructions for carrying out the selected action are provided to the agent. Performing the action results in a change in the environment, e.g., to the holding pipette, the biopsy pipette, and / or the embryo. The change in the environment corresponds to an updated total reward. Causing the agent to perform the action can include, for example, controlling one or more actuators to move at least one of the biopsy pipettes, a holding pipette, a dish containing the embryo, a stage supporting the embryo, a laser, or a microscope.

[0212] As an example, causing the agent to perform the action can include controlling the one or more actuators to move the biopsy pipette to contact the embryo at a region of the embryo. As another example, causing the agent to perform the action can include controlling the one or more actuators to cause suction to be applied to the embryo through the biopsy pipette. As another example, causing the agent to perform the action can include controlling the one or more actuators to cause suction to be applied to the embryo through the holding pipette. As another example, causing the agent to perform the action can include controlling the one or more actuators to cause the laser to be directed at a region of the embryo. As another example, causing the agent to perform the action can include causing the laser to be activated while the laser is directed at the region of the embryo. As another example, causing the agent to perform the action can include providing data representing instructions, for example, through the user interface 112 of FIG. 1, to a human operator.

[0213] In some examples, the action machine learning model 900 can be an off-policy reinforcement learning model that evaluates actions that are not currently considered optimal. For example, an action can be selected that does not have the highest probability in the probability distribution. In some examples, the off-policy reinforcement learning model can explore the action space more thoroughly than an on-policy reinforcement learning model. In some examples, the off-policy reinforcement learning model can be used to evaluate the operation of an inexperienced operator or a biopsy session that results in failure.

[0214] Steps 1010-1030 can be performed at each of multiple time points throughout the course of the biopsy procedure performed in the biopsy environment. In some examples, the number of time points and / or the time elapsed between each time point is fixed. In some examples, steps 1010-1030 are repeatedly performed until the system determines that thebiopsy procedure is complete. For example, steps 1010-1030 can be repeatedly performed until the system determines that the appropriate amount of biomaterial has been removed from the embryo. In some implementations, steps 1010-1030 are repeatedly performed until the system determines that the biomaterial removed from the embryo during the biopsy procedure has been released into a sample tube.

[0215] Fig. 6 is block diagram of an example computer system 600 that can be used to perform operations described above. The system 600 includes a processor 610, a memory 620, a storage device 630, and an input / output device 640. Each of the components 610, 620, 630, and 640 can be interconnected, for example, using a system bus 650. The processor 610 is capable of processing instructions for execution within the system 600. In one implementation, the processor 610 is a single-threaded processor. In another implementation, the processor 610 is a multi -threaded processor. The processor 610 is capable of processing instructions stored in the memory 620 or on the storage device 630.

[0216] The memory 620 stores information within the system 600. In one implementation, the memory 620 is a computer-readable medium. In one implementation, the memory 620 is a volatile memory unit. In another implementation, the memory 620 is a nonvolatile memory unit.

[0217] The storage device 630 is capable of providing mass storage for the system 600. In one implementation, the storage device 630 is a computer-readable medium. In various different implementations, the storage device 630 can include, for example, a hard disk device, an optical disk device, a storage device that is shared over a network by multiple computing devices (e.g., a cloud storage device), or some other large capacity storage device.

[0218] The input / output device 640 provides input / output operations for the system 600. In one implementation, the input / output device 640 can include one or more network interface devices, e.g., an Ethernet card, a serial communication device, e.g., and RS-232 port, and / or a wireless interface device, e.g., and 802.11 card. In another implementation, the input / output device can include driver devices configured to receive input data and send output data to other input / output devices, e.g., keyboard, printer and display devices 660. Other implementations, however, can also be used, such as mobile computing devices, mobile communication devices, set-top box television client devices, etc.

[0219] Although an example processing system has been described in Fig. 6, implementations of the subject matter and the functional operations described in this specification can be implemented in other types of digital electronic circuitry, or in computersoftware, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them.

[0220] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0221] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine- readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0222] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0223] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0224] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0225] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0226] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), amobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0227] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0228] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0229] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0230] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework, or a Jax framework.

[0231] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnectedby any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0232] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0233] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any disclosure or of what may be claimed, but rather as descriptions of features that may be specific to particular examples of particular disclosures. Certain features that are described in this specification in the context of separate examples can also be implemented in combination in a single example. Conversely, various features that are described in the context of a single example can also be implemented in multiple examples separately or in any suitable subcombination. Moreover, although features may be described herein as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0234] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the examples described herein should not be understood as requiring such separation in all examples, and it should be understood that the described program components and systems can generally be integrated together in a single product or packaged into multiple products.

[0235] Particular examples of the subject matter have been described. Other examples are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular ordershown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Claims

CLAIMSWhat is claimed is:

1. A system for controlling an agent interacting with a biopsy environment during a biopsy process, the system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for controlling the agent, the operations comprising: at each of a plurality of time points: generating a virtualization of the biopsy environment, wherein the virtualization depicts an embryo and a biopsy pipette at the respective time point; processing the virtualization of the biopsy environment using a machine learning model to generate a model output that comprises an action to be performed by the agent; and causing the agent to perform the action.

2. The system of claim 1, wherein the system further comprises a camera, and wherein generating a virtualization of the biopsy environment comprises: receiving an image depicting the embryo and the biopsy pipette from the camera; and in response, generating the virtualization of the biopsy environment from the image using a segmentation machine learning model.

3. The system of claim 2, wherein generating the virtualization of the biopsy environment from the image comprises generating, using the segmentation machine learning model, a feature segmentation map for the image comprising one or more segmentation masks.

4. The system of claim 3, wherein the operations further comprise displaying the feature segmentation map in the virtualization of the biopsy environment.

5. The system of any one of claims 3-4, wherein the operations further comprise: identifying a portion of the image comprising the embryo based on the feature segmentation map; andin response, displaying the identified portion in the virtualization of the biopsy environment.

6. The system of claim 1, wherein generating a virtualization of the biopsy environment comprises: receiving data indicating a state of the biopsy pipette; identifying a change in the state of the biopsy pipette based on the data indicating the state of the biopsy pipette; and generating the virtualization of the biopsy environment based on the change in the state of the biopsy pipette.

7. The system of claim 1, wherein generating a virtualization of the biopsy environment comprises: receiving data indicating a state of a holding pipette; identifying a change in the state of the holding pipette based on the data indicating the state of a holding pipette; and generating the virtualization of the biopsy environment based on the change in the state of the holding pipette.

8. The system of any one of claims 1-7, wherein the model output comprises a probability distribution for a plurality of possible actions.

9. The system of claim 8, wherein causing the agent to perform the action comprises selecting the action to be performed by the agent from the plurality of possible actions.

10. The system of any one of claims 1-9, wherein the machine learning model is trained to generate a model output that maximizes a total reward based on one or more reward scores at each time point.

11. The system of claim 10, wherein each reward score is based on any one or more of: an amount of biomaterial contained within the biopsy pipette, a time elapsed for the biopsy process, a proximity of a sample locus to one or more identified sampling regions of the embryo, a number of times the embryo was touched by the biopsy pipette, and a proximity of a holding pipette to one or more identified holding placement regions.

12. The system of any one of claims 1-11, wherein the agent is a human operator, and wherein causing the agent to perform the action comprises providing data representing instructions to the human operator to perform the action.

13. The system of any one of claims 1-12, wherein the system further comprises a laser configured to be directed at a separation region depicted in the virtualization, the separation region indicating a region where the biopsy pipette and the embryo are in contact.

14. The system of any one of claims 1-13, wherein the system further comprises a microscope configured to be directed at the embryo.

15. The system of any one of claims 1-12, wherein the system further comprises a plurality of actuators configured to move at least one of the biopsy pipette, a holding pipette, a dish, a laser, or a microscope.

16. The system of claim 15, wherein causing the agent to perform the action comprises controlling one or more actuators of the plurality of actuators to move at least one of the biopsy pipette, the holding pipette, the dish, a stage supporting the embryo, the laser, or the microscope.

17. The system of any one of claims 1-14, wherein causing the agent to perform the action comprises controlling one or more actuators to move the biopsy pipette to contact the embryo at a region of the embryo.

18. The system of any one of claims 1-14, wherein causing the agent to perform the action comprises controlling one or more actuators to cause suction to be applied to the embryo through the biopsy pipette.

19. The system of any one of claims 1-14, wherein causing the agent to perform the action comprises controlling one or more actuators to cause suction to be applied to the embryo through a holding pipette.

20. The system of any one of claims 1-12, wherein causing the agent to perform the action comprises controlling one or more actuators to cause a laser to be directed at a region of the embryo.

21. The system of claim 20, wherein causing the agent to perform the action comprises causing the laser to be activated while the laser is directed at the region of the embryo.

22. A method performed by one or more computers for controlling an agent interacting with a biopsy environment during a biopsy process, the method comprising: at each of a plurality of time points: generating a virtualization of the biopsy environment, wherein the virtualization depicts an embryo and a biopsy pipette at the respective time point; processing the virtualization of the biopsy environment using a machine learning model to generate a model output that comprises an action to be performed by the agent; and causing the agent to perform the action.

23. The method of claim 22, wherein generating a virtualization of the biopsy environment comprises: obtaining an image depicting the embryo and the biopsy pipette; and in response, generating the virtualization of the biopsy environment from the image.

24. The method of claim 23, wherein obtaining the image comprises receiving the image from a camera directed at the embryo and the biopsy pipette.

25. The method of claim 23, wherein the image comprises a simulated image, and wherein obtaining the image comprises generating the simulated image from a model of the embryo.

26. The method of any one of claims 23-25, wherein generating the virtualization of the biopsy environment from the image comprises generating, using a segmentation machine learning model, a feature segmentation map for the image comprising one or more segmentation masks.

27. The method of claim 26, further comprising displaying the feature segmentation map in the virtualization of the biopsy environment.

28. The method of any one of claims 26-27, further comprising: identifying a portion of the image comprising the embryo based on the feature segmentation map; and in response, displaying the identified portion in the virtualization of the biopsy environment.

29. The method of claim 22, wherein generating a virtualization of the biopsy environment comprises: receiving data indicating a state of the biopsy pipette; identifying a change in the state of the biopsy pipette based on the data indicating the state of the biopsy pipette; and generating the virtualization of the biopsy environment based on the change in the state of the biopsy pipette.

30. The method of claim 22, wherein generating a virtualization of the biopsy environment comprises: receiving data indicating a state of a holding pipette; identifying a change in the state of the holding pipette based on the data indicating the state of a holding pipette; and generating the virtualization of the biopsy environment based on the change in the state of the holding pipette.

31. The method of any one of claims 22-30, wherein causing the agent to perform the action comprises controlling one or more actuators to move at least one of the biopsy pipette, a holding pipette, a dish containing the embryo, a stage supporting the embryo, a laser, or a microscope.

32. The method of any one of claims 22-30, wherein causing the agent to perform the action comprises controlling one or more actuators to move the biopsy pipette to contact the embryo at a region of the embryo.

33. The method of any one of claims 22-30, wherein causing the agent to perform the action comprises controlling one or more actuators to cause suction to be applied to the embryo through the biopsy pipette.

34. The method of any one of claims 22-30, wherein causing the agent to perform the action comprises controlling one or more actuators to cause suction to be applied to the embryo through a holding pipette.

35. The method of any one of claims 22-30, wherein causing the agent to perform the action comprises controlling one or more actuators to cause a laser to be directed at a region of the embryo.

36. The method of claim 35, wherein causing the agent to perform the action comprises causing the laser to be activated while the laser is directed at the region of the embryo.

37. The method of any one of claims 22-36, wherein the model output comprises a probability distribution for a plurality of possible actions.

38. The method of claim 37, wherein causing the agent to perform the action comprises selecting the action to be performed by the agent from the plurality of possible actions.

39. The method of any one of claims 22-38, wherein the machine learning model is trained to generate a model output that maximizes a total reward based on one or more reward scores at each time point.

40. The method of claim 39, wherein each reward score is based on any one or more of: an amount of biomaterial contained within the biopsy pipette, a time elapsed for the biopsy process, a proximity of a sample locus to one or more identified sampling regions of the embryo, a number of times the embryo was touched by the biopsy pipette, and a proximity of a holding pipette to one or more identified holding placement regions.

41. The method of any one of claims 22-30, and 37-40, wherein the agent is a human operator, and wherein causing the agent to perform the action comprises providing data representing instructions to the human operator to perform the action.

42. A method performed by one or more computers for analyzing an embryo during a biopsy process, the method comprising: receiving, from a camera in real-time during the biopsy process, one or more images that each depict an embryo and a biopsy pipette; processing, in real-time using a machine learning model, the one or more images to determine an amount of biomaterial contained within the biopsy pipette; and causing, in real-time, a user interface to display an indication of the amount of biomaterial within the biopsy pipette.

43. The method of claim 42, wherein processing the one or more images comprises identifying one or more regions of interest in each image, wherein the one or more regions of interest can include at least one of a holding pipette, the biopsy pipette, or biomaterial.

44. The method of claim 43, further comprising determining, in real-time, a segmentation mask for each region.

45. The method of claim 44, further comprising causing, in real-time, the user interface to display an indication of each segmentation mask.

46. The method of any one of claims 42-45, wherein biomaterial comprises at least one of an embryo, an inner cell mass, a zona pellucida, an inner boundary of the zona pellucida, trophectoderm cells within the zona pellucida, trophectoderm cells within the biopsy pipette, or a herniation.

47. The method of any one of claims 42-46, further comprising: based on processing the one or more images, determining, in real-time, that the amount of biomaterial contained within the biopsy pipette is below a predetermined range; andin response, causing, in real-time, the user interface to display an indication that the amount of biomaterial contained within the biopsy pipette is below the predetermined range.

48. The method of any one of claims 42-46, further comprising: based on processing the one or more images, determining, in real-time, that the amount of biomaterial contained within the biopsy pipette is above a predetermined range; and in response, causing, in real-time, the user interface to display an indication that the amount of biomaterial contained within the biopsy pipette is above the predetermined range.

49. The method of any one of claims 42-46, further comprising: based on processing the one or more images, determining, in real-time, that the amount of biomaterial contained within the biopsy pipette is within a predetermined range; and in response, causing, in real-time, the user interface to display an indication that the amount of biomaterial contained within the biopsy pipette is within the predetermined range.

50. The method of claim 49, wherein the predetermined range is between 5 and 10 cells.

51. The method of claim 49, wherein the predetermined range is between 67% and 133% of an adequate target amount.

52. The method of any one of claims 49-51, further comprising: based on processing the one or more images, determining, in real-time, a separation region where the biopsy pipette and the embryo are in contact; and in response, causing, in real-time, the user interface to display an indication of the separation region.

53. The method of claim 52, further comprising causing, in real-time, the user interface to display a laser guide at the separation region.

54. The method of any one of claims 52-53, further comprising: causing a laser to be directed at the separation region; and controlling the laser to be activated while the laser is directed at the separation region.

55. The method of any one of claims 42-54, wherein processing the one or more images to determine an amount of biomaterial contained within the biopsy pipette comprises processing, in real-time using the machine learning model, the one or more images to determine a volume of the biomaterial.

56. The method of any one of claims 42-54, wherein processing the one or more images to determine an amount of biomaterial contained within the biopsy pipette comprises processing, in real-time using the machine learning model, the one or more images to determine a number of cells contained within the biopsy pipette.

57. The method of any one of claims 42-56, further comprising: processing, in real-time using the machine learning model, the one or more images, to determine one or more entry regions of the embryo for taking the biopsy.

58. The method of claim 57, wherein the one or more entry regions comprise at least one of a herniation of the embryo, or a region away from an inner cell mass of the embryo.

59. The method of any one of claims 57-58, further comprising: causing, in real-time, the user interface to display an indication of the one or more entry regions.

60. The method of any one of claims 57-59, further comprising: causing the biopsy pipette to be enter the embryo at one of the one or more entry regions; and causing the biopsy pipette to suction biomaterial from the embryo.

61. The method of any one of claims 42-60, wherein: receiving one or more images comprises receiving, in real-time from the camera, a sequence of images that depicts the embryo and the biopsy pipette; and processing the one or more images comprises: selecting two or more images from the sequence of images; identifying one or more images in the sequence as non-selected images; andprocessing, in real-time using a machine learning model, the selected images to determine an amount of biomaterial contained within the biopsy pipette.

62. The method of claim 61, wherein processing, in real-time using a machine learning model, the selected images to determine an amount of biomaterial contained within the biopsy pipette comprises determining a segmentation mask for each of one or more regions in each of the selected images, wherein each region depicts at least one of a holding pipette, the biopsy pipette, or biomaterial.

63. The method of claim 62, wherein processing the one or more images comprises: for each of the one or more non-selected images in the sequence of images: estimating a segmentation mask for each of the one or more regions in the non-selected image.

64. The method of claim 63, wherein estimating a segmentation mask for each of the one or more regions in each of the one or more non-selected images comprises: for each of the one or more regions, estimating a respective location for each of a plurality of points that define the segmentation mask based on the respective location for each of the plurality of points in one or more prior images in the sequence of images.

65. The method of any one of claims 63-64, further comprising, for each of the one or more images: causing, in real-time, the user interface to display an indication of each segmentation mask or each estimated segmentation mask.

66. The method of any one of claims 42-65, further comprising: determining that the biopsy pipette has separated from the embryo; determining that the amount of biomaterial within the biopsy pipette is non-zero; and in response, recording the amount of biomaterial within the biopsy pipette and an identifier for the embryo.

67. The method of any one of claims 42-66, further comprising training the machine learning model, wherein training comprises: receiving a set of training data, wherein the training data comprises:(i) a plurality of training outputs, each training output comprising a labeled image, wherein the labeled image includes an image depicting an embryo and a biopsy pipette and includes one or more labels corresponding to one or more regions in the labeled image,(ii) a plurality of training inputs, each training input corresponding to a training output, and each training input comprising an unlabeled image, wherein the unlabeled image includes the image of the corresponding training output; and training the machine learning model on the set of training data.

68. The method of claim 67, wherein training the machine learning model on the set of training data comprises training the machine learning model to, for each training input, process the training input to generate a model output that matches the corresponding training output.

69. The method of any one of claims 67-68, wherein training the machine learning model on the set of training data comprises training the machine learning model to detect one or more regions in an image.

70. The method of any one of claims 67-69, wherein the one or more labels correspond to a region that depicts trophectoderm cells within the biopsy pipette and represent an amount of trophectoderm cells within the biopsy pipette.

71. The method of claim 70, wherein training the machine learning model on the set of training data comprises training the machine learning model to predict an amount of trophectoderm cells within the biopsy pipette in an image.

72. A system for analyzing an embryo during a biopsy process, the system comprising: a camera; one or more computers; and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for analyzing an embryo during a biopsy process, the operations comprising:receiving, from the camera in real-time during the biopsy process, one or more images that each depict an embryo and a biopsy pipette; processing, in real-time using a machine learning model, the one or more images to determine an amount of biomaterial contained within the biopsy pipette; and causing, in real-time, a user interface to display an indication of the amount of biomaterial within the biopsy pipette.

73. The system of claim 72, wherein the system further comprises a laser configured to be directed at a separation region depicted in the one or more images.

74. The system of any one of claims 72-73, wherein the system further comprises a microscope configured to be directed at the embryo.

75. The system of any one of claims 72-74, wherein the system further comprises a plurality of actuators configured to move at least one of the biopsy pipette, a holding pipette, a dish, a laser, or a microscope.

Images