Determining a location in a germ cell structure

Abstract

Systems and methods for determining a clinical parameter representing a location of interest within a germ cell structure are provided. An image of a germ cell structure is obtained and provided to a neural network to generate a clinical parameter. The clinical parameter is stored on a non-transitory computer readable medium.

Description

[0001] Related Applications

[0002] This application claims priority to each of U.S. Provisional Patent Application No. 63 / 077,405, filed September 11, 2020, entitled "Artificial Intelligence System for Assisting Preimplantation Embryo Aneuploidy Screening" and U.S. Provisional Patent Application No. 63 / 077,398, filed September 11, 2020, entitled "Artificial Intelligence Support System for Oocyte and Preimplantation Embryo Alignment in Intracytoplasmic Sperm Injection (ICSI) and Assisted Hatching (AH) Procedures". The entire contents of each of these applications are incorporated herein by reference in their entirety for all purposes.

[0003] Government Rights Statement

[0004] This invention was made with government support from one or more of the National Institutes of Health (NIH) grant numbers R01AI118502, R01AI138800, and R21HD092828. The government has certain rights to this invention. Technical Field

[0005] This invention generally relates to the field of assisted reproduction, and more specifically, to determining the location of germ cell structures. Background Technology

[0006] Infertility is an undervalued healthcare problem affecting more than 48 million couples worldwide and a cause of suffering, depression, and stigma. While assisted reproductive technologies (ART) such as in vitro fertilization (IVF) have alleviated the burden of infertility to some extent, their efficiency is low, with an average success rate of approximately 26% reported in the United States in 2015. IVF remains an expensive solution, costing between $7,000 and $20,000 per ART cycle in the United States, which is often not covered by insurance. Furthermore, many patients require multiple IVF cycles to achieve pregnancy. Embryos typically transfer to the patient's uterus during the cleavage or blastocyst stage of development. Embryos are described as being in the cleavage stage two or three days after fertilization. Embryos reach the blastocyst stage five or six days after fertilization. The blastocyst has a fluid-filled cavity and two distinguishable cell types, the trophectoderm and the inner cell mass (ICM). Summary of the Invention

[0007] According to one aspect of the invention, a system is provided. The system includes a processor and a non-transitory computer-readable medium storing machine-executable instructions for assigning values ​​representing locations of interest within germ cell structures. The machine-executable instructions include an imager interface that receives images of the germ cell structures from an associated imager. A neural network determines clinical parameters representing the locations of interest within the germ cell structures from the images. The clinical parameters are stored in the non-transitory computer-readable medium.

[0008] According to another aspect of the invention, a method is provided for determining clinical parameters representing locations of interest within germ cell structures. An image of the germ cell structure is obtained and fed into a neural network to generate the clinical parameters. The clinical parameters are stored on a non-transitory computer-readable medium.

[0009] According to another aspect of the invention, a method for planning assisted reproductive processes is provided. An image of a germ cell structure is obtained and fed into a neural network to generate clinical parameters representing locations of interest within the germ cell structure. The germ cell structure is then processed at the locations determined based on the clinical parameters. Attached Figure Description

[0010] The above and other features of the present invention will become apparent to those skilled in the art upon reading the following description with reference to the accompanying drawings, wherein:

[0011] Figure 1 An example of a system for determining locations of interest on germ cell structures is shown;

[0012] Figure 2 Another example of a system for determining locations of interest on germ cell structures is shown;

[0013] Figure 3 A method for determining clinical parameters representing locations of interest within germ cell structures is shown;

[0014] Figure 4 Methods for planning assisted reproductive processes are shown; and

[0015] Figure 5 This is a schematic block diagram of an exemplary system showing hardware components capable of implementing the systems and methods disclosed herein. Detailed Implementation

[0016] Two of the most widely used and technically challenging micromanipulation procedures performed in clinical embryology laboratories are intracytoplasmic sperm injection (ICSI) and assisted hatching (AH). Both procedures are performed manually by highly trained embryologists. ICSI involves arranging metaphase (MII) oocytes, selecting and fixing sperm, and injecting sperm at a precise location that does not interfere with the mitotic spindle. The spindle is located near the extrusion polar body and cannot be visualized using a bright-field microscope. Assisted hatching is a procedure designed to allow the embryo to escape from the zona pellucida (ZP). Studies have shown that AH can increase pregnancy chances in older women with recurrent IVF failures and in frozen embryo transfer cycles. This procedure is widely used for cleavage-stage embryos to facilitate herniation and biopsy of trophectoderm cells for preimplantation genetic testing. If AH is performed too close to healthy cells, blastomeres can be easily damaged.

[0017] The term "germ cell structure" as used in this article refers to single-celled or multi-celled structures involved in assisted reproductive technologies in mammalian individuals. Germ cell structures can include gametes (such as oocytes) and preimplantation fertilized embryos.

[0018] The term "static observation" as used in this article refers to an image or group of images representing a single point in the development of germ cell structures. When multiple images are used in static observation, no significant changes in the structure and appearance of the germ cell structures occur between images.

[0019] Figure 1 An example of a system 100 for determining locations of interest on germ cell structures is shown. System 100 includes a processor 102 and a non-transitory computer-readable medium 110 storing machine-executable instructions for assigning values ​​representing locations of interest within the germ cell structures. The machine-executable instructions include an imager interface 112 that receives images of the germ cell structures from an associated imager. For example, the imager interface 112 may receive images from the imager via a bus or network connection and modulate the images for analysis at a neural network 114. In one example, the neural network 114 may be implemented on a cloud computing system, where images are transmitted via a network interface (not shown) to a server containing the neural network 114.

[0020] Neural network 114 determines clinical parameters representing locations of interest within a germ cell structure from an image of the germ cell structure. In one embodiment, each possible value of the clinical parameter represents a specified location in a representation of the germ cell structure. In one example, the representation may be generally circular, and the various values ​​for the clinical parameter may represent individual sectors of the circle. In one embodiment, the representation is divided into twelve 30-degree sectors. In another example, the representation is divided into an array of tiled polygons, and each polygon in the array is represented by one of the values ​​for the clinical parameter. In one embodiment, the germ cell structure is an oocyte, and the clinical parameter represents the location of the polar body within the oocyte. In another embodiment, the germ cell structure is an embryo, and the clinical parameter represents the location on the zona pellucida furthest from a healthy blastomere.

[0021] The neural network 114 comprises multiple interconnected nodes. Values ​​from an image (e.g., brightness and / or chromaticity values ​​associated with individual pixels) are provided to multiple input nodes. Each input node provides these input values ​​to a layer of one or more intermediate nodes. A given intermediate node receives one or more output values ​​from previous nodes. The received values ​​are weighted according to a set of weights established during the training of the classifier. The intermediate node transforms its received values ​​into a single output according to the activation function at the node. For example, an intermediate node may sum the received values ​​and perform the following operations on the sum: recognition function, step function, sigmoid function, hyperbolic tangent, rectified linear unit, leaky rectified linear unit, parametric rectified linear unit, Gaussian error linear unit, softplus function, exponential linear unit, scaled exponential linear unit, Gaussian function, sigmoid linear unit, growing cosine unit, heaviside function, and mish function. The final layer of the nodes provides confidence values ​​for the ANN's output classes, where each node has a correlation value representing the confidence of one of the classifier's relevant output classes.

[0022] Many ANN classifiers are fully connected and feedforward. However, convolutional neural networks (CNNs) consist of convolutional layers where nodes from the previous layer are connected only to a subset of nodes in the convolutional layer. Recurrent neural networks (RNNs) are a class of neural networks where connections between nodes form a directed graph along a time series. Unlike feedforward networks, RNNs can incorporate feedback from states induced by previous inputs, allowing the output of a RNN for a given input to be not only a function of the input but also a function of one or more previous inputs. For example, Long Short-Term Memory (LSTM) networks are a modified version of RNNs that make it easier to remember historical data in memory.

[0023] A neural network 114 is trained on multiple labeled images of appropriate germ cell structures. A "labeled image" is an image in which the location of the location of interest is known, for example, through expert annotation, and during training, clinical parameters associated with the location of the location of interest are provided to the neural network along with the images. During training, the weights associated with the interconnections between nodes in the neural network 114 are iteratively changed until, once the network changes, when a new, unlabeled image is presented, the network output provides clinical parameters representing the location of the location of interest within the new image. These clinical parameters, or a representation of them, may be stored on a non-transitory computer-readable medium 110 and / or provided to a user via an associated output device.

[0024] Figure 2 Another example of a system 200 for determining locations of interest on germ cell structures is shown. Specifically, system 200 generates categorical clinical parameters representing locations on germ cell structures. System 200 includes an imager 202 that acquires images of germ cell structures during at least one day of development. For example, imager 202 may include one or more cameras capable of producing images in the visible or infrared range and paired with appropriate optics to provide images of germ cell structures. In one embodiment, imager 202 may be implemented as part of a time-lapse embryo imaging system for capturing images of embryos at multiple days of development. In another embodiment, imager 202 may be configured to generate static observations of germ cell structures as a set of one or more images. In one embodiment, imager 202 includes an accessory for a mobile device that operates in conjunction with the mobile device's camera to provide images of germ cell structures. The housing for the accessory may be 3D printed using polylactic acid with dimensions of 82*34*48 mm. An acrylic lens may be included within the housing to provide appropriate magnification of the embryo images.

[0025] In another embodiment, the imager 202 can be implemented as a stand-alone system with an optical housing 3D-printed from polylactic acid and dimensions of 62*92*175mm. This housing contains electronic circuitry with white light-emitting diodes, a 3-volt battery, and a single-pole double-throw switch. Embryo samples are transmitted through a 10* plan-achromatic objective lens for image magnification and a complementary metal-oxide-semiconductor (CMOS) image sensor for embryo image data acquisition. The CMOS sensor can be connected to a single-board computer to process the captured images. The imager 202 can be connected to a mobile device via a wireless connection (e.g., Wi-Fi, Bluetooth, or similar) for data processing and visualization.

[0026] One or more images acquired at imager 202 are provided to analysis system 210, which includes processor 212, output device 214, and non-transitory computer-readable medium 220 storing instructions executable by the processor. These instructions can be executed to provide imager interface 222, which receives single or multiple images of germ cell structures. Imager interface 222 may apply one or more imaging conditioning techniques (e.g., cropping and filtering) to better prepare the images for analysis. The images are then provided to neural network 224, which provides classification clinical parameters representing the desired location.

[0027] In one implementation, neural network 224 can be a convolutional neural network, which is a feedforward artificial neural network that includes convolutional layers. These convolutional layers effectively apply convolution to the values ​​of the previous layer of the network to emphasize various feature sets within the image. In the convolutional layers, each neuron is connected only to an appropriate subset of neurons in the previous layer, referred to as the neuron's receptive field. In one example, the Xception architecture is used to implement the convolutional neural network. In one implementation, at least one chromaticity value associated with each pixel (e.g., a value for an RGB color channel, a YCrCb color channel, or a grayscale value) is provided as the initial input to the convolutional neural network.

[0028] In another implementation, neural network 224 can be implemented as a recurrent neural network. In a recurrent neural network, connections between nodes in the network are selected to form a directed graph along a sequence, allowing it to exhibit dynamic temporal behavior. In another implementation, neural network 224 is implemented and trained as a discriminative network in a generative adversarial model, where the generative and discriminative networks provide feedback to each other, causing the generative network to generate increasingly complex samples for the discriminative network to attempt classification. Regardless of the structure of neural network 224, some or all layers of the neural network can be trained via transfer learning from another system, where only some layers are trained on training images of germ cell structures. The last layer of neural network 224 can be implemented as a softmax layer to provide classification results.

[0029] In response to a new image, neural network 224 generates clinical parameters representing portions of the image containing the location of interest. These clinical parameters can be provided to a user at output device 214 via user interface 226. For example, user interface 226 may include appropriate software instructions for receiving the output of neural network 224 and presenting it at output device 214. In one embodiment, output device 214 may include a mobile device that wirelessly communicates with analysis system 210. In one example, the clinical parameters may be provided to the user as a representation of a cellular reproductive structure, with portions of the cellular reproductive structure highlighted within that representation.

[0030] In view of the above structural and functional characteristics, referring to Figure 3 and Figure 4 This will allow for a better understanding of the methods according to various aspects of the present invention. Although for the purpose of simplifying the description, Figure 3 and Figure 4 The methods are shown and described as being performed sequentially; however, it should be understood and recognized that the invention is not limited to the order shown, as some aspects may occur in a different order than those shown and described herein, and / or other aspects may occur simultaneously. Furthermore, not all features shown are required to implement a method according to one aspect of the invention.

[0031] Figure 3 A method 300 for determining clinical parameters representing locations of interest within germ cell structures is illustrated. In one example, method 300 can be used to determine the location of a polar body within an oocyte, where the clinical parameter represents the portion of the oocyte containing the polar body. In another example, the method can be used to determine the location on the zona pellucida furthest from a healthy blastomere, where the clinical parameter represents the appropriate location. At 302, an image of the germ cell structure is obtained. For example, the image can be captured by a suitable imager and provided to a computer system for image processing.

[0032] At 304, an image of the germ cell structure is fed to a neural network to generate clinical parameters. Each possible value of the clinical parameter represents a specified location in the representation of the germ cell structure, such that the clinical parameter indicates a location of interest in the image. The neural network can be implemented as any of a convolutional neural network, a recurrent neural network, or a discriminative classifier trained as part of a generative adversarial network. At 306, the clinical parameters are stored on a non-transitory computer-readable medium.

[0033] Figure 4 A method 400 for planning assisted reproductive processes is illustrated. At 402, an image of the germ cell structure is obtained. For example, the image can be captured at a suitable imager and fed to a computer system for image processing. At 404, the image of the germ cell structure is fed to a neural network to generate clinical parameters representing locations of interest within the germ cell structure. Each possible value of the clinical parameter represents a specified location in the representation of the germ cell structure, such that the clinical parameter indicates the location of interest in the image. The neural network can be implemented as any of a convolutional neural network, a recurrent neural network, or a discriminative classifier trained as part of a generative adversarial network.

[0034] At 406, the germ cell structure can be processed at a location determined according to clinical parameters. In one example, method 400 can be used to determine the location of a polar body within an oocyte, where the clinical parameters represent the portion of the oocyte containing the polar body. The location of the polar body can be used to determine an appropriate location for intracytoplasmic sperm injection (ICSI). For example, ICSI can be performed at a location 90 degrees away from the polar body. In another example, method 400 can be used to determine the location on the zona pellucida that is furthest from a healthy blastomere to avoid damage to healthy blastomeres during laser-assisted hatching. In one implementation, this processing can be performed by an automated robotic system based on the location represented by the clinical parameters.

[0035] Figure 5 This demonstrates that it is possible to achieve Figure 1-4 A schematic block diagram of an exemplary system 500 of hardware components, as disclosed in the systems and methods, such as... Figure 1 and Figure 2 The system shown is shown in the figure. System 500 may include various systems and subsystems. System 500 may be any of a personal computer, laptop, workstation, computer system, device, application-specific integrated circuit (ASIC), server, server blade center, or server cluster.

[0036] System 500 may include a system bus 502, a processing unit 504, a system memory 506, memory devices 508 and 510, a communication interface 512 (e.g., a network interface), a communication link 514, a display 516 (e.g., a video screen), and an input device 518 (e.g., a keyboard and / or mouse). System bus 502 can communicate with processing unit 504 and system memory 506. Additional memory devices 508 and 510 (e.g., hard disk drives, servers, standalone databases, or other non-volatile memory) can also communicate with system bus 502. System bus 502 interconnects processing unit 504, memory devices 506-510, communication interface 512, display 516, and input device 518. In some examples, system bus 502 also interconnects additional ports (not shown), such as Universal Serial Bus (USB) ports.

[0037] System 500 can be implemented in a computing cloud. In this scenario, the features of system 500 (e.g., processing unit 504, communication interface 512, and memory devices 508 and 510) can represent a single instance of hardware or multiple instances of hardware, where applications execute (i.e., distributed) on multiple instances of hardware (e.g., a computer, router, memory, processor, or a combination thereof). Alternatively, system 500 can be implemented on a single dedicated server.

[0038] Processing unit 504 may be a computing device and may include an application-specific integrated circuit (ASIC). Processing unit 504 executes a set of instructions to implement the operations of the examples disclosed herein. The processing unit may include a processing core.

[0039] Additional memory devices 506, 508, and 510 can store data, programs, instructions, database queries, and any other information that may be needed to operate the computer, in text or compiled form. Memory devices 506, 508, and 510 can be implemented as computer-readable media (integrated or removable), such as memory cards, disk drives, optical discs (CDs), or servers accessed via a network. In some examples, memory devices 506, 508, and 510 may include text, images, video, and / or audio, portions of which may be available in a human-understandable format.

[0040] Alternatively or alternatively, system 500 can access external data sources or query sources through communication interface 512, which can communicate with system bus 502 and communication link 514.

[0041] In operation, system 500 can be used to implement one or more portions of the embryo evaluation system according to the present invention. According to some examples, computer-executable logic for implementing the composite application testing system resides on one or more of system memory 506 and memory devices 508, 510. Processing unit 504 executes one or more computer-executable instructions derived from system memory 506 and memory devices 508 and 510. It should be understood that computer-readable media may include multiple computer-readable media operatively connected to the processing unit.

[0042] Specific details are set forth in the foregoing description to provide a thorough understanding of the embodiments. However, it should be understood that the embodiments may be practiced without these specific details. For example, circuits may be shown in block diagrams so as not to obscure the embodiments with unnecessary details. In other instances, well-known circuits, data processing, algorithms, structures, and techniques may be shown without unnecessary details in order to avoid obscuring the embodiments.

[0043] The above-described techniques, blocks, steps, and methods can be implemented in various ways. For example, these techniques, blocks, steps, and methods can be implemented in hardware, software, or a combination thereof. For hardware implementation, the processing unit can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the above functions, and / or combinations thereof.

[0044] Furthermore, it should be noted that embodiments can be described as processes, which may be flowcharts, diagrams, data flow diagrams, structural diagrams, or block diagrams. Although flowcharts can describe operations as sequential processes, many operations can be performed in parallel or concurrently. Moreover, the order of operations can be rearranged. A process terminates when its operations are completed, but there may be other steps not included in the diagram. A process can correspond to a method, procedure, program, subroutine, subroutine, etc. When a process corresponds to a function, its termination corresponds to the function returning to call the function or the main function.

[0045] Furthermore, embodiments can be implemented using hardware, software, scripting languages, firmware, middleware, microcode, hardware description languages, and / or any combination thereof. When implemented in software, firmware, middleware, scripting languages, and / or microcode, program code or code segments that perform the necessary tasks can be stored in a machine-readable medium (e.g., storage medium). Code segments or machine-executable instructions can represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, scripts, classes, or instructions, data structures, and / or program statements. Code segments can be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, and / or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted through any suitable means, including memory sharing, messaging, ticketing, network transmission, etc.

[0046] For firmware and / or software implementations, the methods can be implemented using modules (e.g., processes, functions, etc.) that perform the functions described herein. Any machine-readable medium that tangibly embodies the instructions can be used to implement the methods described herein. For example, software code can be stored in memory. Memory can be implemented within a processor or outside a processor. As used herein, the term "memory" means any type of long-term, short-term, volatile, non-volatile, or other storage medium, and is not limited to any particular type of memory or the number of memories, or the type of medium storing memory.

[0047] Furthermore, as disclosed herein, the term "storage medium" can refer to one or more memories used for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, disk storage media, optical storage media, flash memory devices, and / or other machine-readable media used for storing information. The terms "computer-readable medium" and "machine-readable medium" include, but are not limited to, portable or fixed storage devices, optical storage devices, wireless channels, and / or various other storage media capable of storing instructions and / or data. It should be understood that "computer-readable medium" or "machine-readable medium" can include multiple media, each operatively connected to the processing unit. In this context, when it is stated that data is stored on a computer-readable medium, it can refer to any interconnection medium within the system.

[0048] Although the principles of this disclosure have been described above in conjunction with specific devices and methods, it should be clearly understood that the description is by way of example only and not as limiting the scope of this disclosure.

Claims

1. A system for determining clinical parameters representing a location of interest within a germ cell structure, comprising: processor; and A non-transitory computer-readable medium storing machine-executable instructions for allocating values ​​representing locations of interest within germ cell structures, the machine-executable instructions including: An imager interface that receives images of the germ cell structure from an associated imager; and A neural network that determines clinical parameters representing a location of interest within the germ cell structure from the image of the germ cell structure, trained on a set of images, each image labeled with one of a plurality of classes, wherein each of the plurality of classes represents one of a plurality of sectors, the plurality of sectors including a circular representation of the germ cell structure, the circular representation of the germ cell structure containing the location of interest; and The clinical parameter is a classification parameter representing one of the plurality of classes, and the classification parameter is stored in the non-transitory computer-readable medium.

2. The system according to claim 1, wherein, The germ cell structure is an oocyte, and the clinical parameters indicate the location of the polar body within the oocyte.

3. The system according to claim 1, wherein, The germ cell structure is an embryo, and the clinical parameter represents the position on the zona pellucida furthest from a healthy blastomere.

4. The system according to claim 1, further comprising: An imager, the imager comprising: White light-emitting diode; Complementary metal-oxide-semiconductor CMOS image sensors; and An objective lens, which is connected to the CMOS image sensor.

5. A method for determining clinical parameters representing a location of interest within a germ cell structure, the method comprising: Obtain an image of the germ cell structure; as well as The image of the germ cell structure is fed to a neural network to generate the clinical parameters. The neural network is trained on a set of images, each image labeled with one of a plurality of classes, wherein each of the plurality of classes represents one of a plurality of sectors, the plurality of sectors including a circular representation of the germ cell structure, the circular representation of the germ cell structure containing the location of interest; and The clinical parameters are stored on a non-transitory computer-readable medium, wherein the clinical parameters are classification parameters representing one of the plurality of classes.

6. The method according to claim 5, wherein, Providing an image of the embryo to the neural network includes providing the image of the embryo to a discriminative classifier trained as part of a generative adversarial network.

7. The method according to claim 5, wherein, Providing an image of the embryo to the neural network includes: providing an image of the embryo to a recurrent neural network.

Images