Methods, computer programs, and systems for automated microinjection

Abstract

Methods, computer programs, and systems for automated microinjection Provided herein is a method, computer program, and system for automated microinjection, such as automated intracytoplasmic sperm injection (ICSI).The method described herein includes: creating a first data set of oocyte and holding device and a second data set of injection pipette by a processing unit; detecting oocyte and holding device in the first data set and detecting injection pipette in the second data set; selecting images of the first data set and the second data set in which the equatorial plane of oocyte / holding device and injection pipette has improved focus parameters; selecting images of the first data set and the second data set and labeling pixels related to oocyte and injection pipette; detecting the tip of injection pipette; detecting different morphological structures of oocyte using artificial intelligence computer vision algorithms in the first data set; using the detected morphological structures to create an injection trajectory of injection pipette to perform ICSI; detecting when oocyte bursts and when sperm is released from injection pipette into oocyte cytoplasm.

Description

Detailed Description of the Invention

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of U.S. Provisional Application No. 63 / 342,793, filed May 17, 2022, which is incorporated by reference herein in its entirety. [Background technology] Microinjection is a method of introducing foreign substances into cells using a fine-tipped needle such as a micropipette. Compared with other introduction methods such as electroporation and viral vectors, microinjection has a higher success rate and can better preserve the biological activity of cells. Traditional manual microinjection techniques are time-consuming and prone to errors. Therefore, automated microinjection systems and methods can increase the efficiency of the process and the success rate of injection.

[0002] Provided herein are methods, computer programs, and systems for automated microinjection, such as automated intracytoplasmic sperm injection (ICSI).The methods, computer programs, and systems can include a set of computer vision (CV) detection algorithms and their training to perform the microinjection procedure.ICSI is a procedure in which a single sperm is directly injected into the cytoplasm of an oocyte using a micromanipulator, a micropipette, and a biochip.

[0003] The methods described herein include performing an ICSI procedure using CV. In some embodiments, the methods described herein do not include using CV in combination with electrical resistance or pressure sensing.

[0004] In some embodiments, provided herein is an automated method of performing ICSI, comprising one or more of the following steps: a) receiving, by a processing unit, a first set of images of an oocyte and a holding device (e.g. a holding pipette or a biochip) used to immobilize or hold the oocyte, resulting in a first data set, the oocyte being attached to the holding device, the first set of images being acquired by means of moving the oocyte using a motor in an axis perpendicular to an optical sensor plane, and each image of the first set of images being associated with a given oocyte motor position; b) receiving, by the processing unit, a second set of images of the injection pipette (e.g., a microinjection pipette) resulting in a second data set, the second set of images being acquired by moving the injection pipette in an axis perpendicular to the optical sensor plane, and each image of the second image set being associated with a given pipette motor position; c) detecting, by the processing unit, the oocyte and / or the holding pipette in the first data set and the injection pipette in the second data set using one or more CV detection algorithms; d) using artificial intelligence techniques, by the processing unit, selecting from the images of the first data set an image in which the oocyte is best in focus compared to other images of the first image set, the image corresponding to the equatorial plane of the oocyte; e) positioning, by the motor, the oocyte at a motor position associated with an image of the first data set in which the oocyte is most in focus compared to other images of the first image set; f) selecting, by the processing unit, a number of images of the first data set and labelling pixels belonging to the holding device and the oocyte and selecting a number of images of the second data set and labelling pixels belonging to the injection pipette; g) detecting, by the processing unit, the tip of the injection pipette by running one or more CV algorithms on the second data set using the labeled pixels of the pipette tip; h) detecting, by the processing unit, distinct morphological structures of the oocyte by performing one or more CV algorithms on the first data set, the distinct morphological structures including the zona pellucida, polar bodies, pericellular spaces, and cytoplasm; i) A processing unit generates an injection pipette trajectory for ICSI using the detected oocyte morphological structure; j) execute the trajectory of i) including the movement of the injection pipette (to the motor position); k) perforating the outer layer of the oocyte (zona pellucida) by laser or piezoelectric and expelling the mass of zona pellucida to the outside of the oocyte; l) detecting, by a processing unit, the ejection of a mass of zona pellucida (cork) using one or more CV algorithms; m) performing oolemma perforation using a piezoelectric injection pipette and detecting oolemma rupture using one or more CV algorithms; n) depositing sperm into the oocyte with an injection pipette; and o) detecting, by a processing unit, the ejection of sperm using one or more CV algorithms.

[0005] After step e), the processing unit uses artificial intelligence techniques to select the image from the second data set in which the injection pipette tip is most in focus, and uses the motor position associated with the most in focus image to move the injection pipette to a predetermined position that results in alignment of the injection pipette tip with the oocyte equatorial plane.

[0006] In some embodiments, the image or set of images is acquired by an imaging device, such as an optical instrument, an optical sensor, a microscope, a camera, or any device with an optical system capable of forming an image and capable of forming a digital image.

[0007] In some embodiments, in step a), the first set of images and the second set of images are acquired separately. overview

[0008] In some embodiments, in step a), the first set of images and the second set of images are acquired from the bottom towards the top of the oocyte. In some embodiments, step a) further comprises randomly selecting a plurality of images of the first and second data sets and labeling the oocyte and / or the holding pipette and the injection pipette in the randomly selected images using an image detection algorithm, e.g. a region of interest (ROI) algorithm.

[0009] In some embodiments, in step f) at least one artificial neural network (ANN) and / or at least one CV algorithm is implemented on the images of the first data set that have in particular a maximum value of a focus parameter, such as the variance of the Laplacian.

[0010] In some embodiments, step f) further comprises detecting oocyte background. In some embodiments, the trajectory is created by calculating the center of the cell morphology in a given image, using the calculated center to calculate where the trajectory intersects the zona pellucida and how far into the cytoplasm it must penetrate, and checking if the trajectory intersects the polar body.

[0011] In some embodiments, ICSI can be performed by high frequency vibration actuation (using a piezoelectric actuator) of the injection pipette, which achieves a puncturing effect on the zona pellucida and punctures the oolemma as the injection pipette follows a calculated trajectory.

[0012] Oocyte shell perforation (e.g., zona pellucida perforation) can be achieved by thermally disrupting the oocyte membrane using a laser or by applying high-frequency vibrations to the oocyte membrane with an injection pipette (Piezo method), i.e., piezoelectric-assisted ICSI (Piezo-ICSI).

[0013] In some embodiments, the piezoelectric is stopped once it crosses the periplasmic space, and can then be turned on again when the injection pipette has pushed the oolemma sufficiently into the cytoplasm to successfully puncture the oolemma.

[0014] In some embodiments, the method further comprises acquiring a third set of images of the performed ICSI, thereby creating a third data set, and labeling the images into the following two classes: a) The first class has a ruptured or loose egg cell membrane; and b) The second class, in which the egg cell membrane is neither ruptured nor relaxed.

[0015] Classification can be performed by a CV algorithm from a sequence of images, using these as input to train the classification algorithm. In some embodiments, the method further includes detecting when the sperm are ejected from the injection pipette by acquiring a fourth image set of the performed ICSI using an optical sensor, thereby creating a fourth data set, labeling the sperm using an image detection algorithm, labeling the injection pipette using an image detection algorithm, predicting where the sperm are by training a detection CV algorithm using the fourth data set, and predicting where the injection pipette is by training a detection CV algorithm using the fourth data set. The image detection algorithm can be a ROI or a semantic segmentation algorithm. Each image of the fourth image set, independent of each other, includes sperm during the performance of the ICSI operation, including when the pipette is removed from the oocyte.

[0016] Further provided herein is a system for automation of ICSI procedures, comprising an optical sensor, a holding device adapted to receive an oocyte, an injection pipette, and a processing device including at least one memory and one or more processors, the one or more processors being configured to: a) receiving a first set of images of the oocyte and holding pipette, resulting in a first data set, the first image set being acquired by moving the oocyte / holding pipette in an axis perpendicular to an optical sensor plane, each image of the first image set being associated with a given oocyte / holding pipette motor position; receiving a second set of images of the injection pipette, resulting in a second data set, the second image set being acquired by moving the pipette in an axis perpendicular to the sensor plane, each image of the second image set being associated with a given pipette motor position; b) detecting oocytes and / or holding pipettes in the first data set and injection pipettes in the second data set using one or more CV detection algorithms; c) selecting images from the first and second data sets in which the equatorial plane of the oocyte and / or the holding pipette / apparatus and the injection pipette tip are best in focus, and using the positions of the oocyte and the motor of the holding pipette / apparatus associated with the selected images to align the oocyte and the injection pipette using artificial intelligence techniques; d) selecting a plurality of images of the first data set and labeling pixels belonging to the holding pipette and the oocyte, and selecting a plurality of images of the second data set and labeling pixels belonging to the injection pipette; e) Detect the tip of the injection pipette by running a semantic segmentation algorithm on the labeled pixels; f) detecting distinct morphological structures of the oocyte by executing at least one artificial neural network and / or at least one CV algorithm on the first data set, the distinct morphological structures including: the zona pellucida, the polar body, the pericellular space, and the cytoplasm; and g) Using the detected oocyte morphological structure, calculate where the tip of the pipette will reach to release the sperm into the oocyte and create the injection pipette trajectory for ICSI.

[0017] Other embodiments disclosed herein also include software programs for performing the methods and operations summarized above and disclosed in detail below. For example, provided herein is a computer program product having a computer-readable medium including computer program instructions encoded thereon that, when executed on at least one processor in a computer system, cause the processor to perform the operations described herein. [Brief description of the drawings]

[0018] [Figure 1] Figure 1 is an explanatory diagram of stacked images of an oocyte and a holding pipette. The horizontal black lines indicate different focal planes. FIG. 2 is an explanatory diagram showing an example of ROI labeling of an oocyte and a holding pipette. [Figure 3] Figure 3 shows bottom (Panel A) and side (Panel B) views of the ICSI operation. Arrows indicate the direction of movement of the holding pipette / device or the injection pipette. FIG. 4 is a flowchart illustrating an embodiment of the proposed method. [Detailed explanation]

[0019] Provided herein are methods, systems, and computer programs for automated microinjection, such as automated ICSI. In some embodiments, the automated ICSI is performed using only the CV strategy.

[0020] Metaphase II (MII) is the stage of oocyte maturation where the first polar body is extruded. MII oocytes contain three major components: the zona pellucida (ZP), a protective glycoprotein space that surrounds the mother cell; the ooplasmic region, which is the oocyte cytoplasm; and the pericellular space (PVS), a thick layer between the ooplasm and the ZP. The morphological structure of the oocyte is an important indicator of the likelihood of embryo implantation and healthy development after ICSI. The morphological structures include the zona pellucida, polar body, oocyte, pericellular space, and cytoplasm. The morphological features include oocyte area, oocyte shape, ooplasmic area, ooplasmic translucency, zona pellucida thickness, and pericellular space width.

[0021] In some embodiments, the CV algorithm can be used as follows: - ANN and CV algorithms are used to align the injection pipette 300 (or the injection pipette) with the oocyte 100 in the axis perpendicular to the optical sensor plane. -ANN and CV algorithms are used to detect the images formed by the optical instrument, the injection pipette 300 and the oocyte 100. -Detecting the different morphological structures 101-105 of the oocyte 100, such as the zona pellucida 105, the polar body 104, the oolemma 103, the periplasmic space 102, and the cytoplasm 101 (see Figure 3).

[0022] An injection trajectory 301 is created for the injection pipette 300 so that the injection pipette 300 does not damage the polar body 104 during injection, and further the zona pellucida 105 and the oolemma 103 are penetrated using a piezoelectric actuator. Various aspects of the methods described herein can be embodied in programming. The program aspects of the technology can be, for example, a product or article of manufacture in the form of executable code and / or associated data carried on or embodied in a type of machine-readable medium. The tangible non-transitory storage medium can include semiconductor memory, tape drives, disk drives, etc., memory or other storage for a computer, processor, etc., or associated modules thereof, and can provide storage for software programming.

[0023] All or part of the software can be communicated over a network, such as the Internet or various other communication networks. Such communication allows, for example, loading of the software from one computer or processor to another, such as from a management server or host computer of a scheduling system, to a hardware platform(s) of a computing environment or other system implementing similar functionality in connection with image processing. Thus, other types of media that can carry software elements can include optical, electrical, and electromagnetic waves, such as those used across physical interfaces between local devices, over wired and optical fixed line networks, and over various wireless links. Physical elements that transmit such waves, such as wired or wireless links, optical links, etc., can also be considered media carrying the software. A computer or machine-readable medium may refer to a medium that participates in providing instructions to a processor for execution.

[0024] Machine-readable media can take many forms, including but not limited to tangible storage media, carrier wave media, or physical transmission media. Non-volatile storage media include, for example, optical or magnetic disks, such as the storage devices of any computer(s). These systems or any components thereof can be implemented. Volatile storage media can include dynamic memory, such as the main memory of such a computer platform. Tangible transmission media can include coaxial cables, copper wire and optical fibers, including the wires that form a bus within a computer system. Carrier wave transmission media can take the form of electric or electromagnetic signals, or acoustic or light waves such as those generated during radio frequency (RF) or infrared (IR) data communications. Common forms of computer readable media may include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, DVDs or DVD-ROMs, any other optical media, punch cards paper tapes, any other physical storage media with patterns of holes, RAMs, PROMs and EPROMs, FLASH-EPROMs, NAND flash, SSDs, any other memory chips or cartridges, carrier waves transmitting data or instructions, cables or links transmitting such carrier waves, or any other medium from which a computer can read programming code and / or data. These forms of computer readable media may be involved in carrying one or more sequences of one or more instructions to a physical processor for execution.

[0025] The execution of the various components described herein may be embodied in a hardware device, but may also be implemented as a software-only solution, for example installed on an existing server. Furthermore, image processing may be implemented as firmware, hardware, software, e.g., application programs, or a combination thereof.

[0026] [Usage example] Example 1: Automated ICSI method and system for performing it The CV algorithm described herein provides automation of the ICSI procedure. In this example, the AI ​​computer vision algorithm is called ALGX_AI. The classical computer vision algorithm is called ALGX_CV.

[0027] Oocyte-holding pipette detector (ALG1_AI) 1 shows an example of a method for acquiring images of an oocyte and holding pipette. An oocyte 100 is attached to a holding pipette 200. A stack (or series) of N images of the oocyte 100 and holding pipette 200 (or a first pipette) within the optical field of view 40 is created. For each oocyte 100, a stack of images is collected from the underside (bottom) of the oocyte 100 to the top (top) of the oocyte 100, thereby generating a first dataset, i.e. the oocyte and holding pipette dataset (dataset_oocyte_and_holding_pipette). Then, N images per stack are randomly selected and the pixels of the oocyte 100 and holding pipette 200 are labeled with ROIs. Figure 2 shows an example of how the images are acquired.

[0028] The labeled data is used to train a detection algorithm. For example, the data is split into three groups: 80% training, 10% validation, and 10% test. The algorithm is trained until the validation loss stabilizes. To evaluate the performance of this process, a test set was used. The test set was randomly selected from the dataset. To evaluate the accuracy, we can use the intersection over union (IoU) of 75% or more. Resulting accuracy: 100% (training set size = 80, validation set size = 10, test set size = 10).

[0029] Oocyte Plane Selector (ALG2_CV) For each stack, the oocyte 100 is cut at a fixed dimension. ALG1_AI can be used to identify the equatorial plane of the oocyte 100. In some cases, this step can be done manually.

[0030] For each image in the stack, a Gaussian blur with a kernel of K × K pixels is applied and then the focus parameters (e.g. the variance of the Laplacian) are calculated. The image of the stack that is most equatorially focused is identified by selecting the image of the stack for which the focus parameter is greatest. The performance of this automated process was evaluated by comparing the output results with the manual judgment of an embryologist. A senior embryologist manually identified the equatorial plane in 40 sets of stacks. The embryologist's output was then compared with the output of ALG2_CV. The experiment was successful if both outputs were the same. Results: 100% accuracy over 40 experiments.

[0031] Injection pipette detector (ALG3_AI) 3 shows an example of how images of an injection pipette are acquired. A stack of N images of the injection pipette 300 is created. For each injection pipette 300, a stack of images is collected from the bottom to the top of the oocyte 100, thereby generating an injection dataset, i.e. the injection pipette dataset (dataset_injection_pipette).

[0032] N pictures per stack are randomly selected and pixels belonging to the injection pipette 300 are labeled with an ROI.

[0033] The labeled data is used to train a semantic segmentation algorithm. For example, the data is split into three groups: 80% training, 10% validation, and 10% testing. Then, data augmentation is performed on the training set to increase the diversity of the data. Finally, the algorithm is trained until the validation loss stabilizes. To evaluate the performance of this process, we use a test set. To evaluate the accuracy, we can use the f1 score on images of 768x768 pixels. Results: f1 score pipette = 0.06 (training set size = 300, validation set size = 30, test set size = 30).

[0034] Injection Pipette Plane Selector and Tip Detection (ALG4_CV) The tip of the injection pipette can be used to align the injection pipette with the oocyte. Since the entire pipette is not parallel to the light sensor, it is advantageous to focus only on the injection pipette tip. To align the injection pipette with the oocyte, only the tip of the injection pipette 300 needs to be in the same plane as the equatorial plane of the oocyte 100, as shown in FIG. 3.

[0035] For each stack, crop the image to size M x M at the center of the pipette tip. ALG3_AI can be used to identify the tip of the injection pipette 300. In some cases, this step can also be performed manually.

[0036] For each image in the stack, a Gaussian blur with a K × K kernel is applied and then the focus parameters (e.g. the variance of the Laplacian) are calculated. The image of the stack in which the tip of the injection pipette 300 is best in focus is identified by selecting the image of the stack in which the focus parameter is greatest. ALG3_AI is then used to detect the position of the tip of the injection pipette 300. The performance of this automated process was also evaluated by comparing the output results with those determined manually by an embryologist. A senior embryologist manually labeled the image in which the oocyte was most in focus and identified the tip of the injection pipette in a stack of 40 sets. The output of the embryologist was compared with the output of ALG4_CV. An experiment was considered successful if both outputs were the same or the error in the tip of the pipette was less than 5 pixels. Note: 5 pixels in our system (standard microscope) corresponds to a distance of less than 1 μm. Results 100% accuracy over 40 experiments.

[0037] Morphological detector (ALG5_AI) From the initial data set, images of the oocyte 100 are cropped and in each stack the image with the maximum focus parameter is selected. An expert embryologist labels all pixels that belong to the polar bodies 104, the pericellular space 102, the cytoplasm 101, and the zona pellucida 105. All other pixels are labeled as background.

[0038] The labeled data is used to train a semantic segmentation algorithm. First, data augmentation is performed, then the data is split into three groups: 80% training, 10% validation, and 10% testing. The algorithm is trained until the validation loss stabilizes. A test set was used to evaluate the performance of this process. To evaluate the accuracy, f1 score on 768x768 pixel images was used. Results: f1 score pb = 0.4, f1 score PVS = 0.2, f1 score Cell = 0.01 and f1 score ZP = 0.05 (training set size = 80, validation set size = 10, test set size = 10).

[0039] Morphological post-processing (ALG6_CV) The output of ALG5_AI is used to group all pixels that belong to the same class to form blobs. Once all blobs of the same class are created, the following steps are performed:

[0040] Of the blobs associated with the pericellular space 102, the cytoplasm 101, and the zona pellucida 105, only the largest blob from each is retained, the remaining blobs are assigned as background.

[0041] Of the blobs associated with the polar body 104, only the largest blob is kept and the area of ​​the blob (in pixels) is calculated. If the area is greater than a threshold, the blob is labeled as a polar body 104. Otherwise, the mass is labeled as pericellular space 102 / cytoplasm 101.

[0042] Using the blob from the previous step as input, an injection trajectory 301 is created. aY axis: The cytoplasmic blob is centered on the Y axis. bX-axis: Using the Y-axis from the previous step, the injection trajectory 301 of the injection pipette 300 crossing the zona pellucida 105 and the penetration distance into the cytoplasmic blob is calculated. It is determined whether the trajectory 301 crosses the polar body 104 or gets too close to it. This information can be used to stop the injection or to continue the procedure. c. Piezoelectric Actuator: First, the piezoelectric actuator is activated when the injection pipette 300 crosses the zona pellucida 105, and then it is deactivated when the piezoelectric actuator crosses the pericellular space 102. At this point, the tip of the injection pipette 300 contacts and pushes against the oolemma 103. Finally, the piezoelectric actuator is activated again when the injection pipette 300 penetrates deep enough to rupture the oolemma 103.

[0043] The detection rate of polar bodies is 90%, and the detection rate of other morphological features is 100%, with an error of less than 1% on a test set of 10 photos. A test set is used to evaluate the performance of the process. To evaluate the accuracy of blob construction, the experiment is considered successful when the intersection over union (IoU) of blobs and labeling is 97% or higher.

[0044] Trajectory accuracy was assessed by comparing the output results with those determined manually by an embryologist. A senior embryologist created injection trajectories manually for 40 oocytes. The test was considered successful if the difference between the two trajectories was less than 5 pixels. Resulting blob accuracy: zona pellucida 100%, pericellular space 100%, cytoplasm 100%, polar body 90%. Trajectory accuracy: 97.5%. The more labels, the better the results.

[0045] Egg cell membrane relaxation detector (ALG7_AI) N movies are collected from the full injection to generate a third dataset, the dataset for pipette penetration (dataset_pipette_penetration). An embryologist classifies frames from the dataset and labels them into two classes: a. Class 0: The egg cell membrane 103 has ruptured and retracted. b. Class 1: If the conditions of Class 0 are not true. The optical flow from successive frames is computed and the images are labelled using the previous labelling as follows: a. Class 0: When one of the frames of the oolemma 103 is labeled 0 (oolemma has ruptured and retracted). b. Class 1: If the conditions of Class 0 are not true. In some cases, the oocyst membrane may not rupture sufficiently during ICSI. Determining whether the oocyst membrane is ruptured may be necessary to determine when to stop the perforation device (e.g., laser or piezoelectric) and start the release of sperm into the oocyte. The optical flow can be determined by one or more AI or CV algorithms, for example the Gunnar-Farneback algorithm.

[0046] A classification algorithm is then trained using the labeled computed optical flow. Data augmentation is then performed. The data can then be split into three groups: 80% training, 10% validation, and 10% testing. The algorithm is trained until the validation loss stabilizes.

[0047] A test set is used to evaluate the performance of the process. The experiment is considered successful if the AI ​​and the embryologist's output are the same or the AI ​​detects the puncture of the oolemma within 5 frames. Result Accuracy = 90% (Training set size = 80, Validation set size = 10, Test set size = 10).

[0048] Sperm and pipette tracking (ALG8_AI) From the third dataset, similar to ALG5_AI, images of the injection pipette 300, zona pellucida 105, polar body 104, and cytoplasm 101 can be cropped and labeled at pixel level. Spermatozoon / sperm 310 can also be labeled using ROIs. Both algorithms are trained as ALG5_AI and ALG1_AI. Once the sperm 310 pass the tip of the injection pipette 300, the system detects that injection has occurred.

[0049] To evaluate the performance of this process, videos that were not used for training are taken into account. The test is considered successful if the AI ​​and the embryologist's outputs are the same or if the AI ​​detects the injection of the sperm 310 in the next two frames.

[0050] FIG. 4 summarizes the method illustrated in FIG. 3 herein. The process begins when an oocyte 100 is secured to a holding pipette 200 and within the field of view of the optical sensor 40, and sperm 310 are loaded into the injection pipette 300 (see FIG. 3). In step 401, the system's processor receives a first image set (or a first image stack) of the oocyte 100 and holding pipette 200. In step 402, the processing unit receives a second image set (or a second image stack) of the injection pipette 300. The first and second image sets are acquired by the optical sensor 40 moving in an axis perpendicular to the optical sensor plane, and each of the first and second image sets is associated with a predetermined optical sensor motor position. In step 403, the oocyte 100 and / or holding pipette 200 are detected in the first data set, and the injection pipette 300 is detected in the second data set. This detection can be performed by using one or more CV detection algorithms, such as for example the above mentioned (ALG1_AI) and (ALG2_CV). In step 404, the processing unit selects the image of the first data set in which the equatorial plane of the oocyte 100 and / or holding pipette 200 has the best focus parameters and the image of the second data set in which the equatorial plane of the injection pipette 300 has the best focus parameters. The position of the sample / oocyte motor associated with each selected image can be used for the alignment of the oocyte 100 and / or holding pipette 200 and injection pipette 300, respectively.

[0051] In step 405, pixels related to the oocyte 100 and / or the holding pipette 200 are labeled, and pixels related to the injection pipette 300 are labeled. In step 406, the processing unit detects the tip of the injection pipette 300 by running a semantic segmentation algorithm on the labeled pixels. In step 407, the processing unit detects the different morphological structures 101-105 of the oocyte 100 by running an artificial intelligence and / or CV algorithm on the first data set. In step 408, an injection trajectory 301 of the injection pipette 300 is created to perform ICSI using the detected morphological structures 101-105 of the oocyte 100. In step 409, ICSI is performed based on the injection trajectory 301. In step 410, the processing unit detects oolemma rupture in the data set by classifying the images into two groups: ruptured / relaxed oolemma or non-ruptured / relaxed oolemma. In step 411, sperm are released into the oocyte.

[0052] In some embodiments, a first image set of the oocyte 100 is collected and the equatorial plane of the oocyte 100 is identified using the ALG1_AI and ALG2_CV algorithms described above, and a second image set of the injection pipette 300 is collected and the most focused image of the second image set and the tip of the injection pipette are identified using the ALG3_AI and ALG4_CV algorithms, respectively. The ALG6_CV algorithm can then be used to create an IN injection trajectory 301. The ALG8_AI algorithm can be used to maintain the sperm 310 at the tip of the injection pipette 300 during the execution of the injection trajectory 301. When executing the injection trajectory 301, the ALG7_AI and ALG8_AI algorithms can be used to deliver the sperm 310 immediately after the oolemma 103 is punctured, for example.

[0053] [Embodiment] The following non-limiting embodiments provide illustrations of the devices, systems, and methods disclosed herein, but do not limit the scope of the disclosure. EMBODIMENT 1 a) receiving a first image set, each image of the first image set including an oocyte immobilized by a holding device independently of each other, the oocyte in each image of the first image set being the same oocyte, and the holding device for each image of the first image set being the same holding device; b) labeling, by an image detection algorithm, a number of pixels associated with the oocyte in each image of the first image set; and c) using artificial intelligence to determine an image in which the oocyte is most in focus compared to other images in the first image set based on a plurality of labeled pixels associated with the oocyte. EMBODIMENT 2 d) receiving a second set of images, each image of the second set of images, independent of one another, including an injection pipette, the injection pipette in each image of the second set of images being the same injection pipette, and in each image of the second set of images, the injection pipette is positioned for injection of sperm into an oocyte; e) labeling, by an image detection algorithm, a number of pixels associated with the injection pipette in each image of the second image set; and 2. The method of claim 1, further comprising: f) using artificial intelligence to determine, based on the labeled plurality of pixels associated with the injection pipette, an image in which the injection pipette is most in focus compared to other images of the second image set.

[0054] EMBODIMENT 3 3. The method of embodiment 1 or 2, wherein each image of the first image set is acquired by an imaging device, each image of the first image set has a visual plane, the visual planes of each image of the first image set are parallel, the oocyte moves in an axis perpendicular to an optical sensor plane, each position along the axis perpendicular to the sensor plane is associated with a predetermined oocyte position independently of each other, the sensor plane is parallel to the visual plane of each image of the first image set, each image of the first image set is independently associated with an oocyte position, one oocyte position is most effective, and the most effective oocyte position is the position associated with the image of the first image set in which the oocyte is most in focus compared to other images of the first image set.

[0055] EMBODIMENT 4 The method of embodiment 2 or 3, wherein each image of the second image set is acquired by the imaging device along an axis perpendicular to the sensor plane, each position along the axis perpendicular to the sensor plane is associated independently of one another with a predetermined injection pipette position, each image of the second image set is associated independently of one another with an injection pipette position, one injection pipette position is most effective, and the most effective injection pipette position is the position associated with the image of the second image set in which the injection pipette is most in focus compared to other images of the second image set.

[0056] EMBODIMENT 5 The method according to any one of embodiments 2 to 4, further comprising the step of aligning the oocyte and the injection pipette based on (i) an image of the first image set in which the oocyte is most in focus compared to other images of the first image set, and (ii) an image of the second image set in which the injection pipette is most in focus compared to other images of the second image set.

[0057] EMBODIMENT 6 6. The method according to any one of the preceding embodiments, further comprising identifying a morphological structure of the oocyte based on the plurality of labeled pixels associated with the oocyte.

[0058] EMBODIMENT 7 7. The method of embodiment 6, wherein identifying the morphological structure of the oocyte based on the labeled plurality of pixels is performed by an artificial neural network.

[0059] EMBODIMENT 8 8. The method according to embodiment 6 or 7, wherein the step of identifying the morphological structure of the oocyte based on the labeled plurality of pixels is performed by a computer vision algorithm.

[0060] EMBODIMENT 9 9. The method according to any one of the preceding embodiments, further comprising detecting the background of the oocyte in each image of the first image set. EMBODIMENT 10 10. The method of any one of embodiments 2 to 9, further comprising identifying the tip of the injection pipette based on a plurality of labeled pixels associated with the injection pipette.

[0061] EMBODIMENT 11 The method according to any one of embodiments 2 to 10, wherein each image of the first image set and each image of the second image set are acquired from the underside of the oocyte towards the top of the oocyte.

[0062] EMBODIMENT 12 The method according to any one of embodiments 6 to 11, further comprising a step of determining an injection trajectory of the injection pipette into the oocyte based on the identified morphological structure of the oocyte and the identified tip of the injection pipette.

[0063] EMBODIMENT 13 The injection trajectory is Identifying the center of the morphological structure of the oocyte; Determine where the injection trajectory intersects the zona pellucida of the oocyte; Using the identified centers of morphological structures to determine the distance that the injection trajectory must penetrate into the oocyte cytoplasm to be effective; and 13. The method of embodiment 12, wherein the injection trajectory is determined by determining whether the injection trajectory intersects with the polar body of the oocyte.

[0064] EMBODIMENT 14 14. The method of embodiment 13, wherein the morphological structure is a zona pellucida. EMBODIMENT 15 14. The method of embodiment 13, wherein the morphological structure is a polar body. EMBODIMENT 16 14. The method of embodiment 13, wherein the morphological structure is a pericellular space. EMBODIMENT 17 14. The method of embodiment 13, wherein the morphological structure is a cytoplasm.

[0065] EMBODIMENT 18 The method according to any one of embodiments 12 to 17, further comprising a step of performing intracytoplasmic sperm injection (ICSI) on the oocyte at the injection trajectory by the injection pipette, wherein the sperm are injected from the injection pipette into the oocyte.

[0066] EMBODIMENT 19 20. The method of embodiment 18, further comprising the step of actuating the injection pipette to penetrate the zona pellucida of the oocyte as the injection pipette intersects the zona pellucida of the oocyte.

[0067] EMBODIMENT 20 20. The method of embodiment 19, further comprising the step of stopping the injection pipette when it crosses the pericellular space of the oocyte. EMBODIMENT 21 21. The method of embodiment 20, further comprising the step of reactivating the injection pipette to puncture the oocyte, thereby releasing the sperm within the oocyte.

[0068] EMBODIMENT 22 receiving a third set of images, each image of the third set of images, independently of one another, including an oocyte undergoing ICSI, the oocyte including an oocyte membrane; labeling, by an image detection algorithm, a number of pixels in each image of the third image set that are associated with rupture or loosening of the egg cell membrane; and 22. The method of any one of embodiments 2 to 21, further comprising labeling, in each image of the third image set, a plurality of pixels associated with unruptured or unrelaxed egg cell membranes using an image detection algorithm.

[0069] EMBODIMENT 23 23. The method of embodiment 22, further comprising the step of calculating optical flow between successive images of the third image set.

[0070] EMBODIMENT 24 24. The method of embodiment 22 or 23, further comprising training a classification algorithm using a plurality of labeled pixels associated with ruptured or relaxed egg cell membranes and a plurality of labeled pixels associated with unruptured or unrelaxed egg cell membranes to classify the image into two classes.

[0071] EMBODIMENT 25 receiving a fourth image set, each image of the fourth image set, independent of each other, including sperm performing ICSI on an oocyte at an injection trajectory, the sperm in each image of the fourth image set being the same sperm, and the injection pipette in each image of the fourth image set being the same injection pipette; labeling, with an image detection algorithm, a number of pixels associated with sperm in each image of the fourth image set; predicting the location of the sperm by training a detection algorithm using a fourth set of images; and By using the fourth image set to train a detection algorithm to predict the location of the tip of the injection pipette, The method according to any one of embodiments 18 to 24, further comprising detecting the release of sperm from the injection pipette.

[0072] EMBODIMENT 26 A system comprising a processing unit including at least one memory and one or more processors configured to execute the method according to any one of embodiments 1 to 25.

[0073] EMBODIMENT 27 A computer program product comprising a non-transitory computer-readable medium having computer executable code encoded therein, the computer executable code adapted to be executed to implement the method of any one of embodiments 1 to 25. [Brief description of the drawings]

[0074] [Figure 1] Figure 1 is an illustration of stacked images of an oocyte and a holding pipette. The horizontal black lines indicate the different focal planes. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of ROI labeling of an oocyte and a holding pipette. [Diagram 3] Figure 3 shows bottom (panel A) and side (panel B) views of the ICSI operation. Arrows indicate the direction of movement of the holding pipette / device or injection pipette. [Figure 4] FIG. 4 is a flow chart illustrating one embodiment of the proposed method.

Claims

1. a) receiving a first image set, each image of the first image set independently of one another including an oocyte immobilized by a holding device, the oocyte in each image of the first image set being the same oocyte, and the holding device for each image of the first image set being the same holding device; b) labeling, by an image detection algorithm, a number of pixels associated with the oocyte in each image of the first image set; and c) using artificial intelligence to determine, based on the labeled plurality of pixels associated with the oocyte, an image in which the oocyte is most in focus compared to other images of the first image set.

2. d) receiving a second set of images, each image of the second set of images independently of one another including an injection pipette, the injection pipette in each image of the second set of images being the same injection pipette, and in each image of the second set of images, the injection pipette being positioned for injection of sperm into the oocyte; e) labeling, by an image detection algorithm, a number of pixels associated with the injection pipette in each image of the second image set; and 2. The method of claim 1, further comprising: f) using artificial intelligence to determine, based on a plurality of labeled pixels associated with the injection pipette, an image in which the injection pipette is most in focus compared to other images of the second image set.

3. 2. The method of claim 1, wherein each image of the first image set is acquired by an imaging device, each image of the first image set has a visual plane, the visual planes of each image of the first image set are parallel, the oocyte moves in an axis perpendicular to an optical sensor plane, each position along the axis perpendicular to the sensor plane is associated with a predefined oocyte position independently of each other, the sensor plane is parallel to the visual plane of each image of the first image set, each image of the first image set is associated with an oocyte position independently of each other, one oocyte position is most effective, and the most effective oocyte position is the position associated with the image of the first image set in which the oocyte is most in focus compared to other images of the first image set.

4. 3. The method of claim 2, wherein each image of the second image set is acquired by an imaging device along an axis perpendicular to the sensor plane, each position along the axis perpendicular to the sensor plane is associated with a predetermined injection pipette position independently of each other, each image of the second image set is associated with the injection pipette position independently of each other, one injection pipette position is most effective, and the most effective injection pipette position is the position associated with an image of the second image set at which the injection pipette is most in focus compared to other images of the second image set.

5. 3. The method of claim 2, further comprising aligning the oocyte and the injection pipette based on: (i) the image of the first image set in which the oocyte is most in focus compared to other images of the first image set; and (ii) the image of the second image set in which the injection pipette is most in focus compared to other images of the second image set.

6. The method of claim 1 , further comprising identifying a morphological structure of the oocyte based on a plurality of labeled pixels associated with the oocyte.

7. The method of claim 6 , wherein identifying a morphological structure of the oocyte based on the labeled pixels is performed by an artificial neural network.

8. The method of claim 6 , wherein identifying the oocyte morphological structure based on the labeled pixels is performed by a computer vision algorithm.

9. The method of claim 1 , further comprising detecting a background of the oocyte in each image of the first image set.

10. The method of claim 2 , further comprising identifying a tip of the injection pipette based on a plurality of labeled pixels associated with the injection pipette.

11. The method of claim 2 , wherein each image of the first image set and each image of the second image set are acquired from the bottom to the top of the oocyte.

12. 7. The method of claim 6, further comprising determining an injection trajectory of the injection pipette into the oocyte based on the identified morphological structure of the oocyte and the identified tip of the injection pipette.

13. The injection trajectory is identifying a center of a morphological structure of the oocyte; determining where the injection trajectory intersects the zona pellucida of the oocyte; Using the identified center of the morphological structure to determine the distance that the injection trajectory must penetrate into the oocyte cytoplasm to be effective; and The method of claim 12, determined by determining whether the injection trajectory intersects with the polar body of the oocyte.

14. The method of claim 13, wherein the morphological structure is the zona pellucida.

15. The method of claim 13, wherein the morphological structure is the polar body.

16. The method of claim 13 , wherein the morphological structure is a pericellular space.

17. The method of claim 13, wherein the morphological structure is a cytoplasm.

18. 13. The method of claim 12, further comprising the step of performing intracytoplasmic sperm injection (ICSI) on the oocyte at the injection trajectory with the injection pipette, wherein the sperm are injected from the injection pipette into the oocyte.

19. 20. The method of claim 18, further comprising the step of actuating the injection pipette to penetrate the zona pellucida of the oocyte as the injection pipette intersects the zona pellucida of the oocyte.

20. 20. The method of claim 19, further comprising the step of stopping the injection pipette when it intersects the pericellular space of the oocyte.

21. 21. The method of claim 20, further comprising re-actuating the injection pipette to puncture the egg cell membrane, thereby releasing the sperm within the oocyte.

22. receiving a third set of images, each image of the third set of images, independently of one another, including an oocyte undergoing the ICSI, the oocyte including an oolemma; labeling, in each image of the third image set, a number of pixels associated with rupture or loosening of the egg cell membrane by an image detection algorithm; and 20. The method of claim 18, further comprising labeling, by the image detection algorithm, a plurality of pixels associated with unruptured or unrelaxed egg cell membrane in each image of the third image set.

23. The method of claim 22 , further comprising the step of calculating optical flow between successive images of the third image set.

24. 23. The method of claim 22, further comprising training a classification algorithm using a plurality of labeled pixels associated with ruptured or relaxed egg membranes and a plurality of labeled pixels associated with the non-ruptured or relaxed egg membranes to classify the images into two classes.

25. receiving a fourth image set, each image of the fourth image set independently of one another including the sperm during the performance of ICSI on the oocyte at the injection trajectory, the sperm in each image of the fourth image set being the same sperm, and the injection pipette in each image of the fourth image set being the same injection pipette; labeling, with an image detection algorithm, a number of pixels associated with the sperm in each image of the fourth image set; training a detection algorithm using the fourth set of images to predict the location of the sperm; and by training a detection algorithm using the fourth image set to predict the location of the tip of the injection pipette; 20. The method of claim 18, further comprising detecting the release of the sperm from the injection pipette.

26. A system comprising a processing unit including at least one memory and one or more processors configured to perform the method according to any one of claims 1 to 25.

27. 26. A computer program product comprising a non-transitory computer readable medium having encoded thereon computer executable code, the computer executable code adapted to be executed to perform a method according to any one of claims 1 to 25.

Images