Mitotic identification using generative adversarial networks

Abstract

A machine learning model may be trained to add a tag to a representation of a nucleus that is experiencing mitotic on the tissue image. This may include performing a set of training data creation steps and a set of model training steps. The training data creation step may include receiving a tissue image including a set of representations of a nucleus being subjected to mitotic and creating a target image corresponding to the tissue image. The set of model training steps may include providing the tissue image to a generator model, generating an output image based on the tissue image using the generator model, and training the generator model to create an improved output image based on a difference between the output image and the target image.

Description

Technical Field

[0001] The present disclosure relates to processing histological images using deep learning techniques to identify cells undergoing mitosis. Background Art

[0002] One of the most important topics in microscopy imaging is the classification of cells, typically stained with hematoxylin and eosin (H&E) dye. A particularly challenging task in this field is the detection of mitotic figures—cells undergoing division—in tumor tissue. Mitotic figures are histologically defined by the presence of hair-like projections of chromosomes (nuclear material) in the absence of a nuclear membrane. A common quantitative method is mitotic counting (MC), which involves counting mitotic figures within a standard-sized area of ​​the tumor where the highest mitotic density is assumed. The number of mitotic figures is widely considered one of the most powerful predictors of the biological behavior of many tumor types in humans and animals. However, there are numerous difficulties associated with this type of mitotic counting. For example, even when histological images are annotated by pathologists, studies have found that significant disagreement between human annotators can range from approximately 17-34% in distinguishing individual mitotic figures from other cellular structures. Furthermore, requiring pathologists to provide annotated images can be a significant barrier to obtaining data for training algorithms. Therefore, there is a need for improved techniques for automatically identifying mitotic figures in histological images. Summary of the Invention

[0003] The technology disclosed herein is susceptible to implementation in various ways. For example, it can be implemented as a method for training a machine learning model to add labels to representations of nuclei undergoing mitosis on a tissue image. This method can include performing a set of training data creation steps and a set of model training steps. When performing these steps, the training data creation step can include receiving a tissue image comprising a set of representations of nuclei undergoing mitosis and creating a target image corresponding to the tissue image. In this case, creating the target image can include, for each representation of a nucleus undergoing mitosis in the set of representations of nuclei undergoing mitosis, receiving an identification of the centroid of the representation and adding a label to the tissue image at the centroid of the representation. In this case, the label added to the tissue image at the centroid of the representation can have a uniform size and shape with all other labels added to the tissue image during the execution of the set of training data creation steps. Similarly, in a method comprising performing a set of model training steps, the model training step can include providing the tissue image to a generator model, generating an output image based on the tissue image using the generator model, and training the generator model to create an improved output image based on the difference between the output image and the target image.

[0004] Other types of implementations, including systems and computer-readable media for performing the described methods, are also possible and will be apparent to those skilled in the art based on this disclosure. Therefore, the example methods provided in this disclosure should be understood to be illustrative only and should not be considered as limiting. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] While the specification concludes with claims which particularly point out and distinctly claim the invention, it is believed the invention will be better understood from the following description of certain examples taken in conjunction with the accompanying drawings, wherein like reference numerals represent like elements, and wherein:

[0006] Figure 1 Describes a method that can be used to create input for generative adversarial network training based on a collection of annotated histology images;

[0007] Figure 2 shows a cropped region of the image that can be considered as the received centroid annotation;

[0008] Figure 3 shows the potential result of repeatedly adding markers at the centroid locations,

[0009] Figure 4 An example image pair is shown;

[0010] Figure 5 Provides an example of a conditional generative adversarial network (cGAN) architecture that can be used to train such machine learning models;

[0011] Figures 6A-6B We show a training method that can be used for conditional generative adversarial networks.

[0012] Figure 7 is a block diagram of a computing device that can be used to implement aspects of the disclosed technology.

[0013] The drawings are not intended to be limiting in any way, and it is contemplated that various embodiments of the invention may be carried out in a variety of other ways, including those not necessarily depicted in the drawings. The accompanying drawings, which are incorporated in and form a part of this specification, illustrate several aspects of the invention and, together with the description, serve to explain the principles of the invention; it should be understood, however, that the invention is not limited to the precise arrangements shown. DETAILED DESCRIPTION

[0014] The present disclosure relates to apparatus, systems, and methods for labeling nuclei undergoing mitosis in histological images, and for training machine learning models to automatically create such labels. As described herein, a generative adversarial network can be used to train a deep learning model to label cells undergoing mitosis based on training data that requires less complex annotations than used in many previous methods. By using this type of approach, systems and methods based on the present disclosure can provide benefits such as reducing the workload required to annotate training data, and / or reducing prediction time relative to using object detection to identify mitosis, and / or improving detection performance relative to previous methods, including, for example, those described in "Weakly supervised mitosis detection in breast histopathology images using concentric loss" by Li et al., Medical Image Analysis, 53, 165-178 (April 2019), or "Dual Segmentation of Mitoses and Nuclei Using Conditional GANs on Multi-center Breast H&F, Images" by Razavi et al., Journal of Pathology Informatics 13 (2022), each of which is incorporated herein by reference in its entirety.

[0015] Now go to Figure 1 , which illustrates a method that can be used to create input for generative adversarial network training based on a collection of annotated histology images. As shown in the figure, the method can start by receiving 101 a collection of images with centroid annotations. For example, this can be achieved by receiving one or more H&E stained tissue images (in which a pathologist has identified nuclei undergoing mitosis and identified a centroid for each of these nuclei), dividing each of these images into a set of uniformly sized cropped regions, and then treating the cropped region including one or more identified centroids as the image for which the centroid annotations were received 101. Figure 2 An illustration of a cropped region of an image that may be viewed as receiving a centroid annotation 101 using this approach is provided in , with crosshairs added to indicate the identity of the centroid for ease of reference.

[0016] exist Figure 1In the method, once an image with centroid annotations is received 101, it can be processed to create a labeled image that can be used for training a generative adversarial network. As shown in the figure, the processing can include modifying one of the images received 101 by adding 102 a label to one of the identified centroids of the image received 101. This can be achieved by adding a shape having a size determined by the expected size of a nucleus undergoing mitosis to the received image 101. For example, where the received image is a cropped region of an H&E stained tissue image, the addition 102 can be achieved by adding a circle centered on the centroid and having a diameter that is 1-2 times the expected diameter of a cell undergoing mitosis, taking into account the magnification of the H&E stained image. This can then be repeated on a centroid-by-centroid basis, with the method moving on to the next centroid in the image 103 until a label has been added 102 for each centroid. Figure 3 An example of the potential results of repeatedly adding 102 markers at the centroid locations is provided in , which shows how to modify the Figure 2 The centroid of the image.

[0017] Once labels have been added 102 to all centroids in the image, a check 104 can be performed to check whether there are other images with centroid annotations that should have labels added. If so, the method can proceed 105 to the next image, and the above process can be repeated for that image. Otherwise, the method can end 106 and provide a collection of image pairs as output, where each pair will include an unlabeled H&E stained tissue image and a modified version of that image with the labels added as described above. Figure 4 An example of this type of image pair is provided in , which shows an original H&E stained tissue image 401 and a labeled image 402 to which labels have been added to identify nuclei undergoing mitosis.

[0018] Use as Figure 1 The training data generated by the method described herein can be used to train a machine learning model to automatically add labels to descriptions of nuclei undergoing mitosis in tissue images. These labeled tissue images can then be used for disease diagnosis or treatment, such as by generating mitotic counts. This use can take advantage of the fact that the labels are easier to identify than the underlying nuclei and / or that the original image can be subtracted from the modified image to create a mask consisting of the labels added to the mitotic nuclei. Figure 5 An example of a conditional generative adversarial network (cGAN) architecture that can be used to train such a machine learning model is provided in , and Figures 6A-6B The training methods that can be performed using this type of conditional generative adversarial network architecture are shown in Figure 2, and both figures are discussed below.

[0019] like Figure 5As shown, cGAN 500 may include a generator 501, which may be implemented using an encoder-decoder such as a U-Net (described by Ronneberger et al. in "U-net: Convolutional networks for biomedical image segmentation", arXiv:1505.04597v1, the disclosure of which is incorporated herein by reference in its entirety) and trained to generate an output image 504 (e.g., an image of tissue without markers indicating mitosis) based on an input image 503 (e.g., an image of tissue without markers indicating mitosis). The cGAN will also include a discriminator 502, which may be implemented using a classifier (e.g., the PatchGAN convolutional classifier described by Isola et al. in "Image-to-Image Translation with Conditional Adversarial Networks", arXiv:1611.07004v3, the disclosure of which is incorporated herein by reference in its entirety), which will attempt to distinguish between the output image generated by the generator 501 and an image generated using a neural network such as a neural network. Figure 1 As discussed in more detail below, the generator 501 can be trained based on the difference between the output image 504 and the ground truth image 505 corresponding to the input image used to create the output image, and based on the discriminator's ability to accurately determine that the output image generated by the generator is not the ground truth image. The discriminator 502 can also be trained based on its ability to accurately distinguish between the output image from the generator and the ground truth image 505, such as that generated using the method shown in FIG. Figure 1 The illustrated method exploits the ability to train on ground truth images created by the method, thereby training a model (i.e., generator 501) to automatically generate images that are highly similar to images that would otherwise require significant human input (e.g., identification of centroids by a skilled human pathologist).

[0020] like Figure 6A As shown, during training, an input image (e.g., a raw H&E stained tissue image 401) can be provided 601 as input to the generator, and the generator can generate 602 an output image (e.g., a labeled image 402, which is likely to be largely random, if not completely random, before training the generator). The input and output images can then be provided 603 as a pair to the discriminator, which can (in the case of a discriminator such as PatchGAN, which can provide a real / generated classification for portions of the output image) classify 604 the output image as real (i.e., a ground-truth image) or generated (i.e., created by the generator).

[0021] How the true / generated classification is applied may differ depending on whether the component being trained is the generator 501 or the discriminator 502. For example, in some cases, training may alternate between training the generator and the discriminator, with the first component being trained on a batch of images and the second component being locked during that training, and then the second component being trained on a batch of images (which may or may not be the same batch of images as the batch used to train the first component) while the first component is locked. In this case, by alternating training of the generator and the discriminator, if the generator is being trained, the output target loss may be calculated 605. This calculation 605 may be implemented by calculating the L1 distance between the ground truth labeled image corresponding to the applicable input image and the generated image corresponding to the applicable input image using the following (e.g., using Equation 1):

[0022] L L1 (G)=E x,y,z [||yG(x,y)||1]

[0023] Equation 1

[0024] In the above equation, L L1 (G) is the L1 distance, x and y are the vector representations of the input image and the output image generated by the generator based on the input image, respectively, z is a random noise vector that can be provided to the generator along with the input image to introduce randomness, and G(x,y) is the vector representation of the output image created by the generator based on x and z.

[0025] Once the output target loss is calculated 605, the overall loss for the generator can be calculated 606 using the output target loss. This overall loss can be used to adjust the generator to create images that are closer to the target and more likely to fool the discriminator by combining the loss based on whether the discriminator can correctly classify the output image as created by the generator (rather than incorrectly classifying it as a true value image) with the output target loss. In some implementations, this calculation 606 of the generator loss can also include a weighting value to adjust the relative influence of the output target loss and the loss based on the discriminator's ability to correctly classify the output image. An example of an equation that can be used for this calculation is provided below in Equation 2, where Loss Final Is the generator loss calculated in 606, Loss GAN (G, D) is the loss based on the discriminator’s ability to correctly classify the generator’s output. L1 (G) is the L1 distance, which can be calculated using Equation 1, for example, and λ is a weighting factor that controls the relative contribution of different losses to the overall loss of the generator:

[0026] Loss Final =Loss GAN (G,D)+λLossL1 (G)

[0027] Equation 2

[0028] After the loss for the generator is calculated 606, the loss can then be used to update 607 the generator, for example by backpropagating the loss through the nodes that make up the underlying network, adjusting the values ​​at those nodes to reduce the expected future loss on future computations. After updating 607 the generator, if there are more images (e.g., if the generator was updated based on a batch of images, and there are more images in the batch that have not yet been processed), the training process can move on 608 to the next image and repeat the process using that image. Alternatively, if there are no more images, the process can end. For example, if you are using a training set such as Figure 6A The training process shown utilizes a batch of images to train the generator. Once all images in the batch have been used to train the generator, the generator training can end 609 (for that batch) and the training of the discriminator can then proceed, as shown below. Figure 6A described in the context of .

[0029] Now go to Figure 6B As shown in the figure, when training the discriminator, after the generator generates 602 an output image, the discriminator's ability to correctly classify the input can be used to calculate 610 an output discriminator loss, e.g., a cross entropy loss of the portion of the generated output image that was incorrectly identified as a portion of the true value image. This output discriminator loss can then be used to update 611 the discriminator, e.g., using back propagation in a similar manner as described above in the context of updating 607 the generator. A similar training procedure can then be applied to train the discriminator using the true value images. Specifically, the discriminator can be provided 612 with an input image and its corresponding true value image (e.g., using a method such as Figure 1 The process shown generates a labeled image from an input image). The ground truth image is then classified 613, a loss is calculated 614 based on the accuracy of the discriminator's classification, and the discriminator is then updated 615 as appropriate based on the loss. After this update 615, if there are more images to train the discriminator, the process can move on 608 to the next image and repeat, or if training on the input image (or, depending on the context, input images from a particular training batch) is complete, the process can end 609 (which may result in moving on to the next batch of images).

[0030] Of course, it should be understood that the above Figure 5 and Figure 6A and 6B The architectures and training methods discussed in the context of are illustrative only, and systems and methods different from these descriptions can be implemented by those skilled in the art based on this disclosure without undue experimentation. For example, although Figure 6B The discussion of describes a scenario in which the discriminator is trained with ground truth images and output images corresponding to a single input image, but it is also possible that instead of using pairs of images as described, the discriminator can be trained on a batch of images that includes output and / or ground truth images that are uncorrelated with the other images included in the batch. Similarly, while the above discussion describes that a random vector can be provided as input to the generator along with the input image, in some implementations such a random vector can be omitted and other measures can be taken to introduce randomness (e.g., designating one or more layers in the generator as dropout layers, which are layers configured to randomly set their inputs to 0). Other variations are also possible and will be immediately apparent to those skilled in the art in light of this disclosure. Thus, the above examples, such as Figure 5 、 6A The discussion of 6B, as well as other examples and figures in this document, should not be construed as implying limitations on the protections provided for this document or any related document.

[0031] In such a Figure 5 、 6A After training as described in the context of and 6B, a computer can be configured with the trained generator to take a tissue image as input and automatically generate an output image with labels regarding nuclei undergoing mitosis. Figure 7 An example of a computer system that can be configured in this manner is provided in Figure 7 is a block diagram illustrating an example computing device 700 that can be used in conjunction with various embodiments described herein. Computing device 700 can be a server or any conventional personal computer, or any other processor-enabled device capable of wired or wireless data communications. Other computing devices, systems, and / or architectures can also be used, including devices that are not capable of wired or wireless data communications, as will be apparent to those skilled in the art.

[0032] The computing device 700 preferably includes one or more processors, such as a processor 710. The processor 710 can be, for example, a CPU, a GPU, a TPU, or an array or a combination thereof, such as a CPU and TPU combination or a CPU and GPU combination. Additional processors can be provided, such as auxiliary processors for managing input / output, auxiliary processors for performing floating-point mathematical operations (e.g., TPUs), specialized microprocessors with architectures suitable for rapidly executing signal processing algorithms (e.g., digital signal processors, image processors), slave processors (e.g., back-end processors) subordinate to the main processing system, additional microprocessors or controllers for dual-processor systems or multi-processor systems, or coprocessors. Such auxiliary processors can be discrete processors or can be integrated with the processor 710. Examples of CPUs that can be used with the computing device 700 are Pentium processors, Core i7 processors, and Xeon processors, all of which are available from Intel Corporation of Santa Clara, California. An example GPU that can be used with the computing device 700 is the Tesla K80 GPU from Nvidia Corporation of Santa Clara, California.

[0033] The processor 710 is connected to the communication bus 705. The communication bus 705 may include a data channel for facilitating information transfer between the memory of the computing device 700 and other peripheral components. The communication bus 705 may also provide a set of signals for communicating with the processor 710, including a data bus, an address bus, and a control bus (not shown). The communication bus 705 may include any standard or non-standard bus architecture, such as a bus architecture that conforms to the Industry Standard Architecture (ISA), the Extended Industry Standard Architecture (EISA), the Micro Channel Architecture (MCA), the Peripheral Component Interconnect (PCI) local bus, or a bus architecture that conforms to a standard promulgated by the Institute of Electrical and Electronics Engineers (IEEE), including IEEE 488 general-purpose interface bus (GPIB), IEEE 696 / S-100, etc.

[0034] The computing device 700 preferably includes a main memory 715 and may also include a secondary memory 720. The main memory 715 provides storage of instructions and data for programs executed on the processor 710 (e.g., one or more functions and / or modules discussed above). It should be understood that the computer-readable program instructions stored in the memory and executed by the processor 710 can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written and / or compiled in any combination of one or more programming languages ​​(including but not limited to Smalltalk, C / C++, Java, JavaScript, Perl, Visual Basic, .NET, etc.). The main memory 715 is typically a semiconductor-based memory, such as dynamic random access memory (DRAM) and / or static random access memory (SRAM). Other semiconductor-based memory types include, for example, synchronous dynamic random access memory (SDRAM), Rambus dynamic random access memory (RDRAM), ferroelectric random access memory (FRAM), etc., including read-only memory (ROM).

[0035] The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0036] The secondary storage 720 may optionally include an internal memory 725 and / or a removable medium 730. The removable medium 730 is read and / or written in any known manner. The removable storage medium 730 may be, for example, a tape drive, a compact disc (CD) drive, a digital versatile disc (DVD) drive, other optical disc drives, a flash drive, etc.

[0037] The removable storage medium 730 is a non-transitory computer-readable medium having computer executable code (ie, software) and / or data stored thereon. The computer software or data stored on the removable storage medium 730 is read into the computing device 700 for execution by the processor 710.

[0038] Secondary memory 720 may include other similar elements for allowing computer programs or other data or instructions to be loaded into computing device 700. Such devices may include, for example, external storage media 745 and a communication interface 740 that allows software and data to be transferred from external storage media 745 to computing device 700. Examples of external storage media 745 may include an external hard drive, an external optical drive, an external magneto-optical drive, etc. Other examples of secondary memory 720 may include semiconductor-based memory, such as programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable read-only memory (EEPROM), or flash memory (a block-oriented memory similar to EEPROM).

[0039] As described above, the computing device 700 may include a communications interface 740. The communications interface 740 allows software and data to be transferred between the computing device 700 and external devices (e.g., a printer), a network, or other information sources. For example, computer software or executable code may be transferred from a network server to the computing device 700 via the communications interface 740. Examples of the communications interface 740 include a built-in network adapter, a network interface card (NIC), a Personal Computer Memory Card International Association (PCMCIA) network card, a cardbus network adapter, a wireless network adapter, a Universal Serial Bus (USB) network adapter, a modem, a network interface card (NIC), a wireless data card, a communications port, an infrared interface, an IEEE 1394 FireWire, or any other device capable of interfacing the system 700 with a network or another computing device. The communication interface 740 preferably implements industry-promulgated protocol standards, such as Ethernet IEEE 802 standards, fiber channel, digital subscriber line (DSL), asynchronous digital subscriber line (ADS), frame relay, asynchronous transfer mode (ATM), integrated digital services network (ISDN), personal communications service (PCS), transmission control protocol / Internet protocol (TCP / IP), serial line Internet protocol / point-to-point protocol (SLIP / PPP), etc., but customized or non-standard interface protocols may also be implemented.

[0040] The software and data transmitted via the communication interface 740 typically take the form of electrical communication signals 755. These signals 755 may be provided to the communication interface 740 via a communication channel 750. In embodiments, the communication channel 750 may be a wired or wireless network, or any other type of communication link. The communication channel 750 carries the signals 755 and may be implemented using a variety of wired or wireless communication means, including wire or cable, fiber optics, a traditional telephone line, a cellular telephone link, a wireless data communication link, a radio frequency (RF) link, or an infrared link, to name a few.

[0041] Computer executable code (e.g., computer programs or software) is stored in the main memory 715 and / or the secondary memory 720. The computer program may also be received via the communication interface 740 and stored in the main memory 715 and / or the secondary memory 720. Such a computer program, when executed, enables the computing device 700 to perform the various functions of the disclosed embodiments described elsewhere herein.

[0042] In this document, the term "computer-readable medium" is used to refer to any non-transitory computer-readable storage medium for providing computer-executable code (e.g., software and computer programs) to the computing device 700. Examples of such media include main memory 715, secondary memory 720 (including internal memory 725, removable media 730, and external storage media 745), and any peripheral devices communicatively coupled to the communication interface 740 (including a network information server or other network device). These non-transitory computer-readable media are means for providing executable code, programming instructions, and software to the computing device 500.

[90] In embodiments implemented using software, the software may be stored on a computer-readable medium and loaded into the computing device 700 via the removable media 730, the I / O interface 735, or the communication interface 740. In such embodiments, the software is loaded into the computing device 700 in the form of an electrical communication signal 755. When executed by the processor 710, the software preferably causes the processor 710 to perform the features and functions described elsewhere herein.

[0043] The I / O interface 735 provides an interface between one or more components of the computing device 700 and one or more input and / or output devices. Example input devices include, but are not limited to, a keyboard, a touch screen or other touch-sensitive device, a biometric sensing device, a computer mouse, a trackball, a pen-based pointing device, and the like. Examples of output devices include, but are not limited to, a cathode ray tube (CRT), a plasma display, a light-emitting diode (LED) display, a liquid crystal display (LCD), a printer, a vacuum fluorescent display (VFD), a surface-conduction electron-emitter display (SED), a field emission display (FED), and the like.

[0044] Computing device 700 also includes optional wireless communication components that facilitate wireless communication over voice and / or data networks. The wireless communication components include antenna system 770, radio system 765, and baseband system 760. In computing device 700, radio frequency (RF) signals are sent and received over the air through antenna system 770 under the management of radio system 765.

[0045] The antenna system 770 may include one or more antennas and one or more multiplexers (not shown) that perform switching functions to provide transmit and receive signal paths for the antenna system 770. In the receive path, a received RF signal may be coupled from the multiplexer to a low noise amplifier (not shown) that amplifies the received RF signal and transmits the amplified signal to the radio system 765.

[0046] The radio system 765 may include one or more radios configured to communicate on various frequencies. In an embodiment, the radio system 765 may combine a demodulator (not shown) and a modulator (not shown) in a single integrated circuit (IC). The demodulator and modulator may also be separate components. In the incoming path, the demodulator strips off the RF carrier signal, leaving a baseband receive audio signal, which is sent from the radio system 765 to the baseband system 760.

[0047] If the received signal contains audio information, the baseband system 760 decodes the signal and converts it into an analog signal. The signal is then amplified and sent to a speaker. The baseband system 760 also receives analog audio signals from a microphone. These analog audio signals are converted into digital signals and encoded by the baseband system 760. The baseband system 760 also encodes and decodes the digital signals for transmission and generates a baseband transmit audio signal, which is routed to the modulator section of the radio system 765. The modulator mixes the baseband transmit audio signal with an RF carrier signal to generate an RF transmit signal, which is routed to the antenna system 770 and can pass through a power amplifier (not shown). The power amplifier amplifies the RF transmit signal and routes it to the antenna system 770, where the signal is switched to the antenna port for transmission.

[0048] The baseband system 760 is also communicatively coupled to the processor 710, which may be a central processing unit (CPU). The processor 710 may access data storage areas 715 and 720. The processor 710 is preferably configured to execute instructions (i.e., computer programs or software) that may be stored in the main memory 715 or the secondary memory 720. Computer programs may also be received from the baseband processor 760 and stored in the main memory 710 or the secondary memory 720, or executed upon receipt. Such computer programs, when executed, enable the computing device 700 to perform the various functions of the disclosed embodiments. For example, the data storage areas 715 or 720 may include various software modules.

[0049] The computing device also includes a display 775 attached directly to the communications bus 705 , which may be provided instead of or in addition to any display connected to the I / O interface 735 described above.

[0050] Various embodiments may also be implemented primarily in hardware using components such as application specific integrated circuits (ASICs), programmable logic arrays (PLAs), or field programmable gate arrays (FPGAs). Implementation of a hardware state machine capable of performing the functions described herein will also be readily apparent to those skilled in the relevant art. Various embodiments may also be implemented using a combination of hardware and software.

[0051] In addition, it will be understood by those skilled in the art that the various illustrative logic blocks, modules, circuits and method steps described in conjunction with the above-mentioned figures and the embodiments disclosed herein can generally be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above with respect to their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Technicians can implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing departure from the scope of the present invention. In addition, the grouping of functions within modules, blocks, circuits or steps is for ease of description. Without departing from the present invention, specific functions or steps can be moved from one module, block or circuit to another.

[0052] As further illustration of potential implementations and applications of the disclosed technology, the following examples are provided in a non-exhaustive manner in which the teachings herein may be combined or applied. It should be understood that the following examples are not intended to limit the scope of coverage of any claims that may be filed at any time in this application or in a subsequent filing of this application. No disclaimers are intended. The following examples are provided for illustrative purposes only and for no other purpose. It is contemplated that the various teachings herein may be arranged and applied in a variety of other ways. It is also contemplated that some variations may omit certain features mentioned in the following examples. Therefore, unless expressly indicated otherwise by the inventor or the inventor's successor in interest at a later date, no aspect or feature mentioned below should be considered critical. If any claims are made in this application or in a subsequent filing related to this application that include additional features beyond those mentioned below, it shall not be presumed that these additional features were added for any reason related to patentability.

[0053] Example 1

[0054] A method comprises training a machine learning model to add labels to representations of nuclei undergoing mitosis on a tissue image by: a) performing a set of training data creation steps comprising: i) receiving a tissue image comprising a set of representations of nuclei undergoing mitosis; and ii) for each representation of a nucleus undergoing mitosis in the set of representations of nuclei undergoing mitosis, creating a target image corresponding to the tissue image by: A) receiving an identification of a centroid of the representation; B) adding a label to the tissue image at the centroid of the representation, wherein the label added to the tissue image at the centroid of the representation has a uniform size and shape with all other labels added to the tissue image during the performance of the set of training data creation steps; and b) performing a set of model training steps comprising: i) providing the tissue image to a generator model; ii) generating an output image based on the tissue image using the generator model; and iii) training the generator model to create an improved output image based on the difference between: A) an output image based on the tissue image; and B) a target image corresponding to the tissue image.

[0055] Example 2

[0056] A method according to Example 1, wherein: a) the set of model training steps comprises: i) providing a first paired input comprising a tissue image and an output image based on the tissue image to a discriminator model, ii) using the discriminator model to classify the output image based on the tissue image as belonging to a category selected from: A) a generated image, wherein the generated image is a category for all output images generated by the generator model; and B) a target image, wherein the target image is a category for all target images created in performing the set of training data creation steps; and b) training the generator model to create improved output images is also based on the discriminator model classifying the output image based on the tissue image as a generated image.

[0057] Example 3

[0058] The method of Example 2, wherein: a) the set of model training steps includes: i) providing a second paired input to a discriminator model comprising a tissue image and a target image corresponding to the tissue image; ii) using the discriminator model to classify an output based on the tissue image as belonging to a category selected from the group consisting of: A) a generated image; and B) a target image; and b) training the discriminator model to create an improved classification based on the discriminator model classifying the target image corresponding to the tissue image as a generated image.

[0059] Example 4

[0060] The method of Example 3, wherein the set of model training steps includes alternately training a generator model and a discriminator model on batches of tissue images, wherein the generator model is locked while training the discriminator model, and wherein the discriminator model is locked while training the generator model.

[0061] Example 5

[0062] The method of Example 2, wherein: a) the generator model is an encoder-decoder, and b) the discriminator model is a classifier configured to provide a classification of a generated image or a target image to each of the plurality of blocks for each paired input.

[0063] Example 6

[0064] The method of Example 5, wherein the generator model is a U-Net encoder-decoder and the discriminator model is a PatchGAN classifier.

[0065] Example 7

[0066] The method of Example 2, wherein after performing the set of model training steps, the method includes configuring the computer to add a label to the representation of nuclei undergoing mitosis in the tissue image by configuring the computer with the generator model without configuring the computer with the discriminator model.

[0067] Example 8

[0068] A method according to Example 1, wherein after performing the set of model training steps, the method includes providing a generator model as a machine learning model to add labels to representations of nuclei undergoing mitosis on tissue images, wherein the generator model provided as a machine learning model to add labels to representations of nuclei undergoing mitosis on tissue images includes one or more dropout layers.

[0069] Example 9

[0070] A system for training a machine learning model to add labels to representations of cell nuclei undergoing mitosis on a tissue image, the system comprising a processor configured with a set of computer-executable instructions operable to, when executed, perform a method comprising: a) performing a set of training data creation steps, comprising: i) receiving a tissue image comprising a set of representations of nuclei undergoing mitosis; and ii) for each representation of a nucleus undergoing mitosis in the set of representations of nuclei undergoing mitosis, creating a target image corresponding to the tissue image by: a) receiving the set of training data creation steps; and ii) for each representation of a nucleus undergoing mitosis in the set of representations of nuclei undergoing mitosis, creating a target image corresponding to the tissue image by: The method comprises the steps of: i) providing the tissue image to a generator model, ii) generating an output image based on the tissue image using the generator model, and iii) training the generator model to create an improved output image based on the difference between: A) an output image based on the tissue image, and B) a target image corresponding to the tissue image.

[0071] Example 10

[0072] A system according to Example 9, wherein: a) the set of model training steps comprises: i) providing a first paired input comprising a tissue image and an output image based on the tissue image to a discriminator model; ii) classifying, using the discriminator model, the output image based on the tissue image as belonging to a category selected from: A) a generated image, wherein the generated image is a category for all output images generated by the generator model; and B) a target image, wherein the target image is a category for all target images created in performing the set of training data creation steps; and b) training the generator model to create improved output images is also based on the discriminator model classifying the output image based on the tissue image as a generated image.

[0073] Example 11

[0074] A system according to example 10, wherein: a) the set of model training steps comprises: i) providing a second paired input to a discriminator model comprising a tissue image and a target image corresponding to the tissue image; ii) using the discriminator model to classify an output based on the tissue image as belonging to a category selected from the group consisting of: A) a generated image; and B) a target image; and b) training the discriminator model to create an improved classification based on the discriminator model classifying the target image corresponding to the tissue image as a generated image.

[0075] Example 12

[0076] The system of example 11, wherein the set of model training steps comprises alternatingly training a generator model and a discriminator model on batches of tissue images, wherein the generator model is locked while training the discriminator model, and wherein the discriminator model is locked while training the generator model.

[0077] Example 13

[0078] The system of example 10, wherein: a) the generator model is an encoder-decoder; and b) the discriminator model is a classifier configured to provide a classification of a generated image or a target image to each of the plurality of blocks for each paired input.

[0079] Example 14

[0080] The system of example 13, wherein the generator model is a U-Net encoder-decoder and the discriminator model is a PatchGAN classifier.

[0081] Example 15

[0082] A non-transitory computer-readable medium storing computer-executable instructions operable to program a computer including a processor to perform a method for training a machine learning model to add labels to representations of cell nuclei undergoing mitosis on a tissue image, the method comprising: a) performing a set of training data creation steps including: i) receiving a tissue image including a set of representations of nuclei undergoing mitosis; and ii) for each representation of a nucleus undergoing mitosis in the set of representations of nuclei undergoing mitosis, creating a target image corresponding to the tissue image by: a) receiving a target image of the representation; a) providing the tissue image to a generator model, ii) generating an output image based on the tissue image using the generator model; and iii) training the generator model to create an improved output image based on the difference between: A) an output image based on the tissue image, and B) a target image corresponding to the tissue image.

[0083] Example 16

[0084] Computer-readable medium according to example 15, wherein: a) the set of model training steps comprises: i) providing a first paired input comprising a tissue image and an output image based on the tissue image to a discriminator model; ii) classifying, using the discriminator model, the output image based on the tissue image as belonging to a category selected from: A) a generated image, wherein the generated image is a category for all output images generated by the generator model; and B) a target image, wherein the target image is a category for all target images created in performing the set of training data creation steps; and b) training the generator model to create improved output images is also based on the discriminator model classifying the output image based on the tissue image as a generated image.

[0085] Example 17

[0086] Computer-readable medium according to example 16, wherein: a) the set of model training steps comprises: i) providing a second paired input to a discriminator model comprising a tissue image and a target image corresponding to the tissue image; ii) using the discriminator model to classify an output based on the tissue image as belonging to a category selected from the group consisting of: A) a generated image; and B) a target image; and b) training the discriminator model to create an improved classification based on the discriminator model classifying the target image corresponding to the tissue image as a generated image.

[0087] Example 18

[0088] Computer-readable medium according to Example 17, wherein the set of model training steps includes alternatingly training a generator model and a discriminator model on batches of tissue images, wherein the generator model is locked while training the discriminator model, and wherein the discriminator model is locked while training the generator model.

[0089] Example 19

[0090] The computer-readable medium of example 16, wherein: a) the generator model is an encoder-decoder; and b) the discriminator model is a classifier configured to provide a classification of a generated image or a target image to each of the plurality of blocks for each paired input.

[0091] Example 20

[0092] The computer-readable medium of example 19, wherein the generator model is a U-Net encoder-decoder and the discriminator model is a PatchGAN classifier.

[0093] Each calculation or operation described herein can be performed using a computer or other processor with hardware, software and / or firmware. Various method steps can be performed by modules, and modules can include any of various digital and / or analog data processing hardware and / or software arranged to perform the method steps described herein. Modules optionally include data processing hardware, which is suitable for performing one or more of these steps by having an applicable machine programming code associated therewith, and modules for two or more steps (or a part of two or more steps) are integrated into a single processor board, or are separated into different processor boards in any one of various integrated and / or distributed processing architectures. These methods and systems will generally adopt a tangible medium comprising machine-readable code, with instructions for performing the above-mentioned method steps. Suitable tangible media can include memory (including volatile memory and / or non-volatile memory), storage media (such as magnetic recording on floppy disks, hard disks, magnetic tapes, etc.; optical storage such as CD, CD-R / W, CD-ROM, DVD; or any other digital or analog storage media), etc.

[0094] All patents, patent publications, patent applications, journal articles, books, technical references, etc. discussed in this disclosure are incorporated herein by reference in their entirety for all purposes.

[0095] Different arrangements of components depicted in the drawings or described above, as well as components and steps not shown or described, are possible. Similarly, some features and subcombinations are useful and can be employed without reference to other features and subcombinations. The embodiments of the present invention have been described for illustrative purposes, not limiting, and alternative embodiments will become apparent to readers of this patent. In some cases, method steps or operations may be performed in a different order, or operations may be added, deleted, or modified. It is understood that in certain aspects of the present invention, a single component may be replaced by multiple components, and multiple components may be replaced by a single component to provide an element or structure or perform a given function or functions. Unless such substitution is inappropriate for practicing certain embodiments of the present invention, such substitution is considered within the scope of the present invention. Therefore, the claims should not be construed as limited to the examples, drawings, embodiments, and illustrations provided above, but should be construed as having the scope provided when their terms are given the broadest reasonable interpretation provided by a commonly used dictionary. Except where a term or phrase is indicated under the heading "Defined" as having a specific meaning, it should be understood to have that meaning when used in the claims.

[0096] Clear definition

[0097] It should be understood that in the above examples and claims, the statement that something is "based on" another thing should be understood to mean that the thing is at least partially determined by the thing indicated as being based on. In order to indicate that something must be completely determined based on another thing, it is described as "completely based on" anything that must be completely determined by it.

[0098] It should be understood that in the above examples and claims, the term "set" should be understood as one or more things grouped together. The terms "subset" and "superset" should be understood as synonyms of "set," where the use of "set," "subset," or "superset" is for ease of reference and not intended to convey a substantive distinction.

Claims

1. A method comprising training a machine learning model to add labels to representations of nuclei undergoing mitosis on tissue images by: a) Execute a set of training data creation steps, including: i) receiving a tissue image comprising a collection of representations of nuclei undergoing mitosis; as well as ii) for each representation of a nucleus undergoing mitosis in the set of representations of nuclei undergoing mitosis, creating a target image corresponding to the tissue image by the following steps: A) receiving an identification of a centroid of said representation, B) adding a marker to the tissue image at the centroid of the representation, wherein the marker added to the tissue image at the centroid of the representation has a uniform size and shape with all other markers added to the tissue image during performance of the set of training data creation steps, as well as b) Execute a set of model training steps, including: i) providing said tissue image to a generator model, ii) generating an output image based on the tissue image using the generator model; and iii) training the generator model to create improved output images based on the difference between: A) said output image based on said tissue image, and B) The target image corresponding to the tissue image.

2. The method according to claim 1, wherein: a) The model training step set includes: i) providing a first paired input comprising said tissue image and said output image based on said tissue image to a discriminator model, ii) classifying, using the discriminator model, the output image based on the tissue image as belonging to a class selected from: A) a generated image, wherein the generated image is a category for all output images generated by the generator model; and B) target images, wherein the target images are categories for all target images created in performing the set of training data creation steps; as well as b) training the generator model to create an improved output image is further based on the discriminator model classifying the output image based on the tissue image as a generated image.

3. The method according to claim 2, wherein: a) The model training step set includes: i) providing a second paired input comprising the tissue image and the target image corresponding to the tissue image to the discriminator model; ii) classifying, using the discriminator model, an output based on the tissue image as belonging to a class selected from the group consisting of: A) the generated image; and B) target image; as well as b) training the discriminator model to create an improved classification based on the discriminator model classifying the target image corresponding to the tissue image as a generated image.

4. The method according to claim 3, wherein: The set of model training steps includes alternately training the generator model and the discriminator model on batches of tissue images, wherein the generator model is locked while training the discriminator model, and wherein the discriminator model is locked while training the generator model.

5. The method according to claim 2, wherein: a) the generator model is an encoder-decoder, and b) The discriminator model is a classifier configured to provide a classification of a generated image or a target image to each of the plurality of blocks for each paired input.

6. The method of claim 5, wherein the generator model is a U-Net encoder-decoder and the discriminator model is a PatchGAN classifier.

7. The method of claim 2, wherein after performing the set of model training steps, the method comprises: The computer is configured to add labels to representations of nuclei undergoing mitosis in a tissue image by configuring the computer with the generator model and without configuring the computer with the discriminator model.

8. The method of claim 1, wherein after performing the set of model training steps, the method comprises: The generator model is provided as the machine learning model to add labels to the representation of nuclei undergoing mitosis on the tissue image, wherein the generator model provided as the machine learning model to add labels to the representation of nuclei undergoing mitosis on the tissue image includes one or more dropout layers.

9. A system for training a machine learning model to add labels to representations of cell nuclei undergoing mitosis on tissue images, the system comprising a processor configured with a set of computer-executable instructions operable, when executed, to perform a method comprising: a) Execute a set of training data creation steps, including: i) receiving a tissue image comprising a collection of representations of nuclei undergoing mitosis; and ii) for each representation of a nucleus undergoing mitosis in the set of representations of nuclei undergoing mitosis, creating a target image corresponding to the tissue image by the following steps: A) receiving an identification of a centroid of the representation; B) adding a marker to the tissue image at the centroid of the representation, wherein the marker added to the tissue image at the centroid of the representation has a uniform size and shape with all other markers added to the tissue image during performance of the set of training data creation steps; as well as b) Execute a set of model training steps, including: i) providing said tissue image to a generator model, ii) generating an output image based on the tissue image using the generator model; and iii) training the generator model to create improved output images based on the difference between: A) said output image based on said tissue image, and B) The target image corresponding to the tissue image.

10. The system of claim 9, wherein: a) The model training step set includes: i) providing a first paired input comprising said tissue image and said output image based on said tissue image to a discriminator model; ii) classifying, using the discriminator model, the output image based on the tissue image as belonging to a class selected from: A) a generated image, wherein the generated image is a category for all output images generated by the generator model; and B) target images, wherein the target images are categories for all target images created in performing the set of training data creation steps; as well as b) training the generator model to create an improved output image is further based on the discriminator model classifying the output image based on the tissue image as a generated image.

11. The system of claim 10, wherein: a) The model training step set includes: i) providing a second paired input comprising the tissue image and the target image corresponding to the tissue image to the discriminator model; ii) classifying, using the discriminator model, an output based on the tissue image as belonging to a class selected from the group consisting of: A) the generated image; and B) target image; as well as b) training the discriminator model to create an improved classification based on the discriminator model classifying the target image corresponding to the tissue image as a generated image.

12. The system of claim 11, wherein the set of model training steps comprises alternately training the generator model and the discriminator model on batches of tissue images, wherein the generator model is locked while training the discriminator model, and wherein the discriminator model is locked while training the generator model.

13. The system of claim 10, wherein: a) the generator model is an encoder-decoder; and b) The discriminator model is a classifier configured to provide a classification of a generated image or a target image to each of the plurality of blocks for each paired input.

14. The system of claim 13, wherein the generator model is a U-Net encoder-decoder and the discriminator model is a PatchGAN classifier.

15. A non-transitory computer-readable medium storing computer-executable instructions operable to program a computer comprising a processor to perform a method for training a machine learning model to add labels to representations of cell nuclei undergoing mitosis on a tissue image, the method comprising: a) Execute a set of training data creation steps, including: i) receiving a tissue image comprising a collection of representations of nuclei undergoing mitosis; and ii) for each representation of a nucleus undergoing mitosis in the set of representations of nuclei undergoing mitosis, creating a target image corresponding to the tissue image by the following steps: A) receiving an identification of a centroid of the representation; B) adding a marker to the tissue image at the centroid of the representation, wherein the marker added to the tissue image at the centroid of the representation has a uniform size and shape with all other markers added to the tissue image during performance of the set of training data creation steps; as well as b) Execute a set of model training steps, including: i) providing said tissue image to a generator model, ii) generating an output image based on the tissue image using the generator model; and iii) training the generator model to create improved output images based on the difference between: A) said output image based on said tissue image, and B) The target image corresponding to the tissue image.

16. The computer-readable medium of claim 15, wherein: a) The model training step set includes: i) providing a first paired input comprising said tissue image and said output image based on said tissue image to a discriminator model; ii) classifying, using the discriminator model, the output image based on the tissue image as belonging to a class selected from: A) a generated image, wherein the generated image is a category for all output images generated by the generator model; and B) target images, wherein the target images are categories for all target images created in performing the set of training data creation steps; as well as b) training the generator model to create an improved output image is further based on the discriminator model classifying the output image based on the tissue image as a generated image.

17. The computer-readable medium of claim 16, wherein: a) The model training step set includes: i) providing a second paired input comprising the tissue image and the target image corresponding to the tissue image to the discriminator model; ii) classifying, using the discriminator model, an output based on the tissue image as belonging to a class selected from the group consisting of: A) the generated image; and B) target image; as well as b) training the discriminator model to create an improved classification based on the discriminator model classifying the target image corresponding to the tissue image as a generated image.

18. The computer-readable medium of claim 17, wherein the set of model training steps comprises alternately training the generator model and the discriminator model on batches of tissue images, wherein the generator model is locked while training the discriminator model, and wherein the discriminator model is locked while training the generator model.

19. The computer-readable medium of claim 16, wherein: a) the generator model is an encoder-decoder; and b) The discriminator model is a classifier configured to provide a classification of a generated image or a target image to each of the plurality of blocks for each paired input.

20. The computer-readable medium of claim 19, wherein the generator model is a U-Net encoder-decoder and the discriminator model is a PatchGAN classifier.

Images