Deep Learning-Based Instance Segmentation Training via Regression Layers

Abstract

New tools and techniques are provided for implementing digital microscopy imaging using deep learning-based segmentation and / or instance segmentation based on part annotation. In various embodiments, a computing system can receive a first and a second image, the first image comprising a field of view of a biological sample and the second image comprising a label of an object of interest in the biological sample. The computing system can encode the second image using an encoder to generate third and fourth encoded images (different from each other) comprising a proximity score or proximity map. The computing system can train an AI system to predict the object of interest based at least in part on the third and fourth encoded images. The computing system can generate (using regression) and decode (using a decoder) two or more images based on a new image of the biological sample to predict the label of the object in the new image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application serial number 62 / 832,880, filed on April 12, 2019, by Elad Arbel et al., entitled “DL Based Segmentation via Regression Layers” (the “'880 Application”) (Attorney Docket No. 20190110-01), and U.S. patent application serial number 62 / 832,877, filed on April 11, 2019, by Elad Arbel et al., entitled “Nuclei Segmentation Using Partial Annotation” (the “'877 Application”) (Attorney Docket No. 20190111-01), the disclosure of each of which is incorporated herein by reference in its entirety for all purposes.

[0003] This application may be related to U.S. patent application serial number __ / ___ (the “’__ application”) filed by Elad Arbel et al. on April 10, 2020, entitled “User Interface Configured to Facilitate User Annotation for Instance Segmentation Within Biological Samples” (attorney docket number 20190111-02), which claims priority to the ’880 and ’877 applications, the disclosure of which is incorporated herein by reference in its entirety for all purposes.

[0004] The disclosures of each of these applications / patents (collectively referred to herein as the "Related Applications") are incorporated herein by reference in their entirety for all purposes.

[0005] Copyright Notice

[0006] Portions of the disclosure of this patent document contain material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. Technical Field

[0007] The present disclosure generally relates to methods, systems, and apparatus for implementing digital microscopy (e.g., digital pathology or live cell imaging, etc.), and more particularly, to methods, systems, and apparatus for implementing digital microscopy using deep learning-based segmentation, implementing instance segmentation based on part annotation, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within a biological sample. Background Art

[0008] Digital pathology has become increasingly popular in recent years as many stained tissue slides are digitally scanned at high resolution (e.g., 40X) and viewed as whole slide images ("WSIs") using digital devices (e.g., PCs, tablets, etc.) rather than standard microscopes. Having this information in a digital format enables digital analysis that can be applied to the WSI to facilitate diagnosis.

[0009] Given an image containing many instances of a particular type of object, instance segmentation is the problem of identifying and delineating the different instances in the image (e.g., which cells may be touching or partially overlapping other cells). An example of such a task is nucleus segmentation in microscopy images, where all nuclei need to be segmented. This task is an important step in many digital pathology analyses, such as nucleus classification and various cancer grading tasks. Developing robust nucleus segmentation methods is particularly challenging due to the huge diversity in nucleus shape, color, orientation, and density across different tissues and staining types (e.g., for multi-organ nucleus segmentation or MoNuSeg).

[0010] The performance of cell nucleus segmentation algorithms depends on the size and quality of the available ground truth data that can be used to train the model. For a field of view I of size NxMx3, the ground truth data for cell nucleus segmentation can be specified by an integer-valued label mat (Label Mat) L of size NxM, where all pixels belonging to a specific cell nucleus are assigned a unique positive ID and all background pixels are assigned a zero value. Collecting this ground truth data is very challenging because the exact boundaries of each cell nucleus in the FOV need to be specified. This tedious annotation task is performed by domain experts for thousands of cell nuclei. Therefore, the current difficulty in obtaining large amounts of training data is a factor that limits the broader applicability of deep learning models for cell nucleus segmentation. In addition, while deep learning ("DL")-based models may tend to perform very well for the specific tasks for which they were developed (e.g., cell nucleus segmentation in a specific tissue type, or a specific staining protocol, etc.), they often perform poorly when naively applied to different tissue types, also requiring non-trivial additional annotation work in transfer learning scenarios.

[0011] In many cases, WSI analysis pipelines require robust nucleus segmentation as a fundamental building block. High-performance nucleus segmentation involves a training phase that utilizes high-quality training data annotated by domain experts (e.g., pathologists) to process multiple nucleus outlines. This annotation task is difficult, time-consuming, and tedious to perform, limiting the applicability of deep learning models for nucleus segmentation. Deep learning-based models often perform poorly when naively applied to different segmentation methods.

[0012] Therefore, there is a need for more robust and scalable solutions for implementing digital microscopy, and more specifically, there is a need for methods, systems, and apparatus for implementing digital microscopy using deep learning-based segmentation, implementing instance segmentation based on part annotation, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within biological samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] A further understanding of the nature and advantages of particular embodiments may be achieved by reference to the remainder of the specification and drawings, in which like reference numerals are used to refer to like components. In some cases, a sub-label is associated with a reference numeral to denote one of multiple similar components. When a reference numeral is referenced without specifying an existing sub-label, it is intended to refer to all such multiple similar components.

[0014] Figure 1 is a schematic diagram illustrating a system for implementing digital microscopy using deep learning-based segmentation, implementing instance segmentation based on part annotation, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within a biological sample, according to various embodiments.

[0015] Figure 2A-2C is a system flow diagram illustrating various systems for implementing digital microscopy using deep learning based segmentation, implementing instance segmentation based on part annotation, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within a biological sample, according to various embodiments.

[0016] Figure 3A-Figure 3E is a schematic diagram illustrating various embodiments of a user interface for facilitating user annotation of instance segmentation within a biological sample, according to various embodiments.

[0017] Figure 4 Depicted are examples of various images showing annotations of an object of interest in a raw image of a first biological sample and showing predictions of the object of interest by an artificial intelligence ("AI") system, in accordance with various embodiments.

[0018] Figure 5 Depicted are examples of various images showing elastic enhancement of an original image of a first biological sample and elastic enhancement of an annotated image of the original image, according to various embodiments.

[0019] Figure 6 Depicted are examples of various images showing color enhancement of an original image of a first biological sample, according to various embodiments.

[0020] Figure 7 Depicted are examples of various images showing efficacy of predictions based on fully and partially segmented objects of interest, in accordance with various embodiments.

[0021] Figures 8A-8D is a flowchart illustrating a method for implementing digital microscopy using deep learning-based segmentation and / or instance segmentation based on part annotation, according to various embodiments.

[0022] Figures 9A-9D is a flowchart illustrating a method for implementing digital microscopy using deep learning-based segmentation, implementing instance segmentation based on part annotation, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within a biological sample, according to various embodiments.

[0023] Figure 10 is a block diagram illustrating an exemplary computer or system hardware architecture according to various embodiments.

[0024] Figure 11 is a block diagram of a networked system illustrating a computer, computing system, or system hardware architecture that can be used in accordance with various embodiments. DETAILED DESCRIPTION

[0025] Overview

[0026] Various embodiments provide tools and techniques for implementing digital microscopy (e.g., digital pathology or live cell imaging, etc.), and more specifically, provide methods, systems, and apparatus for implementing digital microscopy using deep learning-based segmentation, implementing instance segmentation based on partial annotation, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within a biological sample.

[0027] In various embodiments, a computing system may receive a first image and a second image, the first image including a field of view (“FOV”) of a first biological sample, and the second image including a marking of an instance of an object of interest in the first biological sample. The computing system may encode the second image using an encoder to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image.

[0028] In some embodiments, the first biological sample may include, but is not limited to, one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like, wherein the object of interest may include, but is not limited to, at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, an organ structure, and / or the like. In some cases, the labeling of the instance of the object of interest in the second image may include, but is not limited to, at least one of a complete annotation of the first instance of the object of interest identifying the center of mass and edges of the first instance of the object of interest, or a partial annotation of the second instance of the object of interest identifying only the center of mass of the second instance of the object of interest, and / or the like.

[0029] In some embodiments, encoding the second image to generate a third encoded image may include computing, with the computing system, a center of mass of each labeled instance of an object of interest in the second image; and generating, with the computing system, the third encoded image, the third encoded image including a highlighting of the center of mass of each labeled instance of the object of interest. In some cases, encoding the second image to generate a fourth encoded image may include computing, with the computing system, an edge or boundary of each labeled instance of the object of interest in the second image; and generating, with the computing system, a fourth encoded image, the fourth encoded image including a highlighting of the edge or boundary of each labeled instance of the object of interest.

[0030] According to some embodiments, the computing system may train an AI system to generate or update an AI model to predict an instance of an object of interest based at least in part on the third encoded image and the fourth encoded image. The computing system may use a regression layer of the AI ​​system or the (updated) AI model to generate a fifth image and a sixth image, where the sixth image is different from the fifth image.

[0031] The computing system may decode the fifth image and the sixth image using a decoder to generate a seventh image, the seventh image including predicted markers of instances of an object of interest in the first biological sample, in some cases by applying at least one of one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or one or more machine learning operations that directly decode the fifth and sixth images to generate the seventh image. In some cases, applying at least one of the one or more morphological operations or the one or more machine learning operations may include applying one or more morphological operations, wherein after decoding the fifth and sixth images by applying the one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images, the computing system may apply a watershed algorithm to generate the seventh image. In some cases, the one or more morphological operations may include, but are not limited to, at least one of a reconstruction open transform or a regional H-minimum transform, and / or the like.

[0032] According to some embodiments, the computing system may compare the seventh image with the second image to generate an instance segmentation evaluation result. In some cases, generating the instance segmentation evaluation result may include evaluating the instance segmentation performance using one or more indicators, which may include but are not limited to at least one of the aggregated Jaccard index ("AJI") indicator, the F1 indicator, the dice indicator, the average dice indicator, or the joint dice indicator and / or the like. In some cases, the instance segmentation evaluation result may include but is not limited to at least one of the instance segmentation evaluation indicator, one or more numerical instance segmentation evaluation scores, or instance segmentation classification (including but not limited to true positive ("TP"), true negative ("TN"), false positive ("FP"), false negative ("FN"), over-segmentation or under-segmentation, etc.) and / or the like. The computing system may display the generated instance segmentation evaluation result on a display screen. In some cases, the seventh image may be generated by a marker-controlled watershed algorithm using a regression layer (which may include an edge surface regression layer, etc.). In some cases, the parameters for the morphological operation may be set using the instance segmentation evaluation result (e.g., AJI score, etc.) as the objective function after applying Bayesian optimization.

[0033] In some cases, training the AI ​​system to generate or update the AI ​​model to predict an instance of the object of interest based at least in part on multiple sets of at least two images generated from the second image may include at least encoding the second image to generate a third encoded image and a fourth encoded image, training the AI ​​system to generate or update the AI ​​model to predict an instance of the object of interest based at least in part on the third encoded image and the fourth encoded image, generating a fifth image and a sixth image, decoding the fifth image and the sixth image to generate a seventh image, and comparing the seventh image to the second image, etc. Although the AI ​​system is trained using two images (in this case, the third encoded image and the fourth encoded image), various embodiments are not limited thereto, and more than two images (or surfaces) may be used.

[0034] According to some embodiments, a computing system may receive an eighth image, the eighth image including a field of view (FOV) of a second biological sample that is different from the first biological sample; generate two or more images based on the eighth image using an AI model generated or updated by a trained AI system, the two or more images being different from each other; and decode the two or more images using a decoder to generate a ninth image, the ninth image including predicted markers of instances of an object of interest in the second biological sample. Similar to decoding the fifth and sixth images, decoding the two or more images to generate the ninth image may include decoding the two or more images using the computing system and the decoder by applying at least one of one or more morphological operations that identify foreground and background markers in each of the two or more images before generating the ninth image, or one or more machine learning operations that directly decode the two or more images to generate the ninth image. In the case of applying the one or more morphological operations, after decoding the two or more images by applying the one or more morphological operations that identify foreground and background markers in each of the two or more images, the computing system may apply a watershed algorithm to generate the ninth image. In this way, a trained AI system and / or AI model can be used to predict labels for instances of objects of interest in new biological samples—in some cases where there are no ground truth images (or previously user-annotated images) corresponding to the new biological sample.

[0035] Alternatively or additionally, the computing system may generate a user interface configured to collect training data using at least one of complete or partial annotations of objects of interest within an image of a biological sample, and may display a first image of a FOV including a first biological sample within a display portion of the user interface. The computing system may receive a first user input from a user (e.g., a pathologist, clinician, doctor, nurse, or laboratory technician, etc.) via the user interface, the first user input indicating the presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface. The computing system may generate a boundary around each of the first plurality of objects of interest based at least in part on the location of each of the first plurality of objects within the first image identified by the first user input and at least in part on analyzing pixels in or around the corresponding location using an algorithm (which may include, but is not limited to, an object detection algorithm, a pixel recognition algorithm, an edge detection algorithm, and / or the like).

[0036] In some cases, the computing system may receive a second user input from a user via a user interface indicating a movement of a point within one of the first plurality of objects of interest from a previous location within the first image to a new location, and may generate a new boundary around the one of the first plurality of objects of interest contained within the first image displayed in the display portion of the user interface based at least in part on the new location of the point within the one of the first plurality of objects of interest within the first image indicated by the second user input and based at least in part on analyzing pixels in or around the new location of the point within the one of the first plurality of objects of interest using an algorithm, the new boundary replacing a boundary previously generated around the one of the first plurality of objects of interest. In some cases, the computing system may receive a third user input from a user via the user interface indicating a partial annotation of one of the second plurality of objects of interest contained within the first image displayed in the display portion of the user interface, and may generate a partial annotation symbol in the first image that identifies a location of a centroid of the one of the second plurality of objects of interest without a boundary based at least in part on the location of the third user input within the first image. In some cases, the computing system may receive a fourth user input from the user via the user interface, indicating that one of the third plurality of objects of interest is unknown or that the instance class of one of the third plurality of objects of interest should be switched to another instance class (e.g., cancer, benign, etc.), and may generate an unknown annotation symbol (i.e., a symbol or annotation, etc. representing an unknown instance or object) in the first image that identifies the location of the unknown object represented by the fourth user input based at least in part on the location of the fourth user input within the first image, or may switch the instance class of the selected one of the third plurality of objects of interest to another instance class selected by the fourth user input (e.g., switching between cancer and benign, switching between complete annotation and partial annotation, switching between partial annotation and unknown annotation, switching between complete annotation and unknown annotation, etc.).

[0037] According to some embodiments, the first user input may include, but is not limited to, one of a click input or a boundary area input. In some cases, the click input may define the location of the center of mass of a first object in the first plurality of objects of interest identified by the click input, while the boundary area input may define an area within the first image that marks the outer limits of the boundary of a second object in the first plurality of objects of interest identified by the boundary area input. In some cases, the boundary area input may include, but is not limited to, one of a rectangular boundary area input, a circular boundary area input, a polygon placement input, a line placement input, and / or the like. In some embodiments, the second user input may include, but is not limited to, a click and drag input. In some cases, the third user input may include, but is not limited to, a double-click input, wherein the third user input selects or deselects a boundary around one of the second plurality of objects of interest. In some cases, the fourth user input may include, but is not limited to, one of a shift-mouse click input or a key-press-mouse click input, wherein the fourth user input may include, but is not limited to, one of switching between a complete annotation and an unknown annotation or switching between instance classes in an instance class list. However, various embodiments are not limited to these specific inputs, and these inputs may be any suitable input for indicating a complete annotation, a partial annotation, and / or an unknown annotation, etc.

[0038] The computing system may generate at least one of a second image or an annotation dataset based on the first image, the second image including data regarding the location of each of the first plurality of objects of interest within the first image based on the received first user input and a generated boundary around each of the first plurality of objects of interest identified by the received first user input, and the annotation dataset including at least one of pixel location data or coordinate data for each of the first plurality of objects within the first image based on the first user input and a generated boundary around each of the first plurality of objects of interest identified by the received first user input.

[0039] In this way, the system provides a fast and efficient UI that allows users (or annotators) to generate annotations in an efficient manner. In particular, users do not need to open any menus or follow a set of complex operations to interact with the UI of the annotation system. Through a single operation (i.e., using click input or boundary area input, etc.), a complete annotation can be generated (i.e., a boundary is generated around the location marked by the click input or boundary area input, etc.). To change the automatically generated boundary, the user only needs to use a single operation (i.e., using click-drag input, etc.) to move a point within the instance or object, thereby causing the system to redraw or regenerate a new boundary around the instance or object. Therefore, the user does not need to waste time manually drawing around the edge or boundary of the instance or object to obtain a complete annotation. Similarly, through a single operation (i.e., shift plus mouse click input, key plus mouse click input, or mouse / keyboard combination, etc.), a complete annotation can be changed to a partial annotation, or the class of the instance or object can be changed. The operation is not bound to a specific mouse / keyboard operation; instead, any combination can be used or customized as appropriate or as needed.

[0040] These and other aspects of implementing digital microscopy (e.g., digital pathology or live cell imaging, etc.) using deep learning-based segmentation (in some cases, through multiple regression layers or other machine learning or deep learning architectures, etc.), implementing instance segmentation based on partial annotations, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within biological samples are described in greater detail with respect to the accompanying drawings. While the focus is on biological samples as described in the following figures, various embodiments are not limited thereto, and instance segmentation, training a system to generate or update an AI model to predict instance segmentation, and / or a user interface configured to facilitate user annotation of instance segmentation may be adapted for application to non-biological samples, including but not limited to chemical samples, humans, animals, plants, insects, tools, vehicles, structures, landmarks, planets, stars, specific animate objects, or specific inanimate objects, and / or the like.

[0041] The following detailed description shows several exemplary embodiments in more detail to enable those skilled in the art to practice such embodiments. The described embodiments are provided for the purpose of illustration and are not intended to limit the scope of the present invention.

[0042] In the following description, for the purpose of explanation, many specific details are set forth to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that other embodiments of the present invention may be practiced without some of these specific details. In other cases, certain structures and devices are shown in block diagram form. Several embodiments are described herein, and although various features are attributed to different embodiments, it will be understood that features described with respect to one embodiment may also be combined with other embodiments. However, for the same reason, a single feature or multiple features of any described embodiment should not be considered essential to each embodiment of the present invention, as other embodiments of the present invention may omit such features.

[0043] Unless otherwise indicated, all numbers used herein to express quantities, dimensions, and the like should be understood as being modified in all instances by the term "about." In this application, unless otherwise indicated, the use of the singular includes the plural, and the use of the terms "and" and "or" means "and / or" unless otherwise indicated. Furthermore, the use of the term "include" and other forms such as "include" and "component" should be considered non-exclusive. Furthermore, unless specifically indicated otherwise, terms such as "element" or "component" encompass elements and components comprising one unit as well as elements and components comprising more than one unit.

[0044] The various embodiments described herein, while embodying (in some cases) software products, computer-implemented methods, and / or computer systems, represent tangible, concrete improvements over the prior art, including, but not limited to, digital pathology technology, live cell imaging technology, digital microscopy technology, instance segmentation technology, cell nucleus segmentation technology, user interface technology, and / or the like. In other aspects, certain embodiments can improve the functionality of a user device or the system itself (e.g., a digital pathology system, a live cell imaging system, a digital microscopy imaging system, an instance segmentation system, a cell nucleus segmentation system, a user interface system, etc.), for example by receiving a first image with a computing system, the first image comprising a field of view (“FOV”) of a first biological sample; receiving a second image with the computing system, the second image comprising a labeling of an instance of an object of interest in the first biological sample; encoding the second image with the computing system and using an encoder to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image; and training an artificial intelligence (“AI”) system to generate or update an AI model to predict an instance of the object of interest based at least in part on the third encoded image and the fourth encoded image; generating, using a regression layer of the AI ​​system, a fifth image and a sixth image based on the first image and based on the training, the sixth image being different from the fifth image; decoding the fifth image and the sixth image with the computing system and using a decoder to generate a seventh image, the seventh image comprising a predicted labeling of the instance of the object of interest in the first biological sample; and comparing the seventh image with the second image with the computing system to generate an instance segmentation evaluation result; and / or the like.

[0045] Alternatively or additionally, certain embodiments may improve functionality of a user device or system itself (e.g., a digital pathology system, a live cell imaging system, a digital microscopy imaging system, an instance segmentation system, a cell nucleus segmentation system, a user interface system, etc.), for example, by generating, with a computing system, a user interface configured to collect training data using at least one of complete annotation or partial annotation of objects of interest within an image of a biological sample; displaying, with the computing system, and within a display portion of the user interface, a first image including a field of view (“FOV”) of a first biological sample; receiving, with the computing system and via the user interface, a first user input from a user, the first user input indicating the presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface; and identifying, with the computing system, based at least in part on the first user input, each of the first plurality of objects of interest within the first image. The method comprises: determining a position of each of a first plurality of objects of interest within the first image and generating a boundary around each of the first plurality of objects of interest based at least in part on analyzing pixels in or around the corresponding position using an algorithm; generating, with a computing system, at least one of a second image or an annotation dataset based on the first image, the second image including data regarding the position of each of the first plurality of objects of interest within the first image based on the received first user input and the generated boundary around each of the first plurality of objects of interest identified by the received first user input, the annotation dataset including at least one of pixel position data or coordinate data for each of the first plurality of objects within the first image based on the first user input and the generated boundary around each of the first plurality of objects of interest identified by the received first user input.

[0046] In particular, to the extent any abstract concepts are present in the various embodiments, such concepts may be implemented as described herein by devices, software, systems, and methods involving specific novel functionality (e.g., steps or operations), such as providing a user interface configured to receive user input indicating at least one of a complete annotation of a first instance of an object of interest identifying a centroid and edges of the first instance of the object of interest, a partial annotation of a second instance of the object of interest identifying only a centroid of the second instance of the object of interest, or an annotation of an unknown instance of the object of interest; and / or training an AI system (including but not limited to at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network (“CNN”), or a fully convolutional network (“FCN”), and / or the like) to predict instances of an object of interest in an image of a biological sample based on complete and / or partial annotations; and / or the like, to name a few examples, that go beyond merely conventional computer processing operations. These functions can produce tangible results outside of the implementation of a computer system, including, by way of example only, optimized presentation and tracking of user input indicating complete annotations, partial annotations, and / or annotations of unknown objects, and / or optimized training of an AI system to generate or update an AI model to predict instances of objects of interest in biological sample images based on complete and / or partial annotations, and / or the like, at least some of which can be observed or measured by a user (including but not limited to a pathologist, clinician, doctor, nurse, or laboratory technician, etc.).

[0047] In one aspect, a method may include receiving, with a computing system, a first image comprising a field of view (“FOV”) of a first biological sample; receiving, with the computing system, a second image comprising labels of instances of an object of interest in the first biological sample; and training an artificial intelligence (“AI”) system to generate or update an AI model to predict instances of the object of interest based at least in part on multiple sets of at least two images generated from the second image, each of at least two images in the multiple sets of at least two images being different from each other.

[0048] In some embodiments, the computing system may include a computing system disposed in a working environment, a remote computing system disposed outside the working environment and accessible via a network, a network server, a network browser, or a cloud computing system, and / or the like. The working environment may include at least one of a laboratory, a clinic, a medical facility, a research facility, a healthcare facility, or a room, and / or the like. In some cases, the AI ​​system may include at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network ("CNN"), or a fully convolutional network ("FCN"), and / or the like. In some cases, the first biological sample may include one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like. The object of interest may include at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, or an organ structure, and / or the like.

[0049] According to some embodiments, training an AI system to generate or update an AI model to predict instances of an object of interest based at least in part on a plurality of sets of at least two images generated from a second image may include encoding the second image with a computing system using an encoder to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image; training the AI ​​system to generate or update the AI ​​model to predict instances of the object of interest based at least in part on the third encoded image and the fourth encoded image; generating a fifth image and a sixth image based on the first image using the AI ​​model generated or updated by the AI ​​system, the sixth image being different from the fifth image; and decoding the fifth image and the sixth image with a decoder using the computing system to generate a seventh image, the seventh image including a predicted label for an instance of the object of interest in the first biological sample. In some cases, training the AI ​​system to generate or update the AI ​​model to predict instances of the object of interest based at least in part on a plurality of sets of at least two images generated from the second image may further include comparing the seventh image with the second image with the computing system to generate an instance segmentation evaluation result.

[0050] In some embodiments, encoding the second image to generate a third encoded image may include: computing, with the computing system, a center of mass of each labeled instance of an object of interest in the second image; and generating, with the computing system, the third encoded image, the third encoded image including a highlighting of the center of mass of each labeled instance of the object of interest. In some embodiments, encoding the second image to generate a fourth encoded image may include: computing, with the computing system, an edge or boundary of each labeled instance of the object of interest in the second image; and generating, with the computing system, the fourth encoded image including a highlighting of the edge or boundary of each labeled instance of the object of interest.

[0051] By way of example only, in some cases, encoding the second image to generate a third encoded image may further include calculating, with the computing system, a first distance metric between each pixel in the third encoded image and each centroid of each labeled instance of the object of interest; and calculating, with the computing system, a first function to generate a first proximity map, the first function being a function of the first distance metric, the third encoded image including the first proximity map. Similarly, encoding the second image to generate a fourth encoded image may further include calculating, with the computing system, a second distance metric between each pixel in the fourth encoded image and a nearest edge pixel of an edge or boundary of each labeled instance of the object of interest; and calculating, with the computing system, a second function to generate a second proximity map, the second function being a function of the second distance metric, the fourth encoded image including the second proximity map.

[0052] According to some embodiments, the method may further include assigning, with the computing system, a first weighted pixel value to each pixel in the third encoded image based at least in part on at least one of a first distance metric, a first function, or a first proximity map calculated for each pixel; and assigning, with the computing system, a second weighted pixel value to each pixel in the fourth encoded image based at least in part on at least one of a second distance metric, a second function, or a second proximity map calculated for each pixel.

[0053] In some embodiments, the method may further include determining, by the computing system, a first pixel loss value between each pixel in the third encoded image and a corresponding pixel in the fifth image; determining, by the computing system, a second pixel loss value between each pixel in the fourth encoded image and a corresponding pixel in the sixth image; calculating, by the computing system, a loss value based on a first weighted pixel value for each pixel in the third encoded image multiplied by the product of the first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image, and a second weighted pixel value for each pixel in the fourth encoded image multiplied by the product of the second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image; and updating, by the AI ​​system, the AI ​​model by updating one or more parameters of the AI ​​model based on the calculated loss value. In some cases, the loss function may include one of a mean squared error loss function, a mean squared logarithmic error loss function, a mean absolute error loss function, a Huber loss function, a weighted sum of squared differences loss function, and / or the like. In this case, generating the fifth and sixth images may include generating the fifth and sixth images based on the first image using the updated AI model.

[0054] According to some embodiments, the labeling of the instance of the object of interest in the second image includes at least one of a complete annotation of the first instance of the object of interest identifying a centroid and an edge of the first instance of the object of interest, a partial annotation of the second instance of the object of interest identifying only a centroid of the second instance of the object of interest, or an unknown annotation of the third instance of the object of interest identifying neither a centroid nor an edge, and / or the like. In some cases, the method may further include, before calculating the loss value, masking, with the computing system, the second instance of the object of interest using the partial annotations in corresponding pixels in the fourth encoded image and the sixth image, but not masking the second instance of the object of interest using the partial annotations in the third encoded image or the fifth image; and, before calculating the loss value, masking, with the computing system, the third instance of the object of interest using the unknown annotations in corresponding pixels in the third encoded image and the fifth image and corresponding pixels in the fourth encoded image and the sixth image.

[0055] In some embodiments, decoding the fifth and sixth images to generate the seventh image may include decoding the fifth and sixth images to generate the seventh image using a computing system and a decoder by applying at least one of one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or one or more machine learning operations that directly decode the fifth and sixth images to generate the seventh image. In some cases, applying at least one of the one or more morphological operations or the one or more machine learning operations may include applying one or more morphological operations, and the method may further include applying a watershed algorithm to generate the seventh image after decoding the fifth and sixth images by applying the one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images. In some cases, the one or more morphological operations may include at least one of a reconstruction open transform, a regional H-minimum transform, and / or the like.

[0056] According to some embodiments, the method may further include receiving, with a computing system, an eighth image, the eighth image including a FOV of a second biological sample different from the first biological sample; generating, based on the eighth image, two or more images using an AI model generated or updated by a trained AI system, the two or more images being different from one another; and decoding, with the computing system and using a decoder, the two or more images to generate a ninth image, the ninth image including predicted markers of instances of an object of interest in the second biological sample.

[0057] In some cases, the first image and the second image can be data enhanced before being received by the computing system, wherein the data enhancement of the first image and the second image can include at least one of elastic enhancement or color enhancement and / or the like configured to facilitate instance segmentation. In some cases, the at least two images include at least a centroid layer image, a boundary layer image, and a semantic segmentation layer image, the centroid layer image highlighting the centroid of each labeled instance of the object of interest in the second image, the boundary layer image highlighting the edge or boundary of each labeled instance of the object of interest in the second image, and the semantic segmentation layer image including semantic segmentation data for each labeled instance of the object of interest in the second image.

[0058] In another aspect, a system may include a computing system that may include at least one first processor and a first non-transitory computer-readable medium communicatively coupled to the at least one first processor. The first non-transitory computer-readable medium may store thereon computer software including a first set of instructions that, when executed by the at least one first processor, cause the computing system to: receive a first image, the first image including a field of view (“FOV”) of a first biological sample; receive a second image, the second image including a labeling of an instance of an object of interest in the first biological sample; and train an artificial intelligence (“AI”) system to generate or update an AI model to predict instances of the object of interest based at least in part on a plurality of sets of at least two images generated from the second image, each of at least two images in the plurality of sets of at least two images being different from each other.

[0059] In yet another aspect, a method may include receiving, with a computing system, a first image comprising a field of view (“FOV”) of a first biological sample; generating, based on the first image, two or more images using an artificial intelligence (“AI”) model generated or updated by a trained AI system, each of the two or more images being different from one another, wherein training the AI ​​system comprises training the AI ​​system to generate or update the AI ​​model to predict instances of an object of interest based, at least in part, on multiple sets of at least two images generated from user-annotated images, each of at least two images in the multiple sets of at least two images being different from one another; and decoding, with the computing system and using a decoder, the two or more images to generate a second image comprising predicted markers of instances of the object of interest in the first biological sample.

[0060] In one aspect, a method may include generating, with a computing system, a user interface configured to collect training data using at least one of complete annotation or partial annotation of objects of interest within an image of a biological sample; displaying, with the computing system, and within a display portion of the user interface, a first image including a field of view (“FOV”) of a first biological sample; receiving, with the computing system and via the user interface, a first user input from a user, the first user input indicating a presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface; and generating, with the computing system, a user interface configured to collect training data using at least one of complete annotation or partial annotation of objects of interest within an image of a biological sample; displaying, with the computing system, a first image including a field of view (“FOV”) of a first biological sample within a display portion of the user interface; receiving, with the computing system and via the user interface, a first user input indicating a presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface; and generating, with the computing system, a user interface configured to generate ... and generating, with a computing system, at least one of a second image or an annotation dataset based on the first image, the second image including data regarding the location of each of the first plurality of objects of interest within the first image based on the received first user input and the generated boundary around each of the first plurality of objects of interest identified by the received first user input, the annotation dataset including at least one of pixel location data or coordinate data for each of the first plurality of objects of interest within the first image based on the received first user input and the generated boundary around each of the first plurality of objects of interest identified by the received first user input.

[0061] In some embodiments, the computing system may include a computing system disposed in a work environment, a remote computing system disposed outside the work environment and accessible via a network, a network server, a network browser, a cloud computing system, and / or the like. In some cases, the work environment may include at least one of a laboratory, a clinic, a medical facility, a research facility, a healthcare facility, or a room, and / or the like. In some cases, the first biological sample may include a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like. In some cases, the object of interest may include at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, or an organ structure, and / or the like.

[0062] In some embodiments, the method may further include receiving, with the computing system and via the user interface, a second user input from a user, the second user input indicating that a point within one of the first plurality of objects of interest moves from a previous position within the first image to a new position; and generating, with the computing system, a new boundary around the one of the first plurality of objects of interest contained within the first image displayed in a display portion of the user interface based at least in part on the new position of the point within the one of the first plurality of objects of interest within the first image indicated by the second user input, and at least in part on analyzing pixels in or around the new position of the point within the one of the first plurality of objects of interest using an algorithm, the new boundary replacing a boundary previously generated around the one of the first plurality of objects of interest.

[0063] According to some embodiments, the method may further include receiving, with the computing system and via the user interface, a third user input from a user, the third user input indicating a partial annotation of one of the second plurality of objects of interest contained within the first image displayed in the display portion of the user interface; and generating, with the computing system, a partial annotation symbol in the first image based at least in part on a location of the third user input within the first image, the partial annotation symbol identifying a location of an unbounded centroid of one of the second plurality of objects of interest.

[0064] In some embodiments, the method may further include receiving, with the computing system and via the user interface, a fourth user input from the user, the fourth user input indicating that one of the third plurality of objects of interest is unknown or that the instance class of one of the third plurality of objects of interest should be switched to another instance class; and generating, with the computing system, an unknown annotation symbol in the first image identifying the location of the unknown object represented by the fourth user input based at least in part on the location of the fourth user input within the first image, or switching, with the computing system, the instance class of a selected one of the third plurality of objects of interest to another instance class selected by the fourth user input.

[0065] By way of example only, in some cases, the first user input may include one of a click input or a boundary area input, wherein the click input defines the location of the center of mass of each of at least one first object in the first plurality of objects of interest identified by the click input, wherein the boundary area input defines an area within the first image that marks the outer limits of the boundary of a second object in the first plurality of objects of interest identified by the boundary area input, wherein the boundary area input may include one of a rectangular boundary area input, a circular boundary area input, a polygon placement input, a line placement input, and / or the like. The second user input may include a click and drag input. The third user input may include a double-click input, wherein the third user input may include selecting or deselecting a boundary around one of the second plurality of objects of interest. The fourth user input may include one of a shift plus mouse click input or a key plus mouse click input, and / or the like. The fourth user input may include one of switching between a complete annotation and an unknown annotation or switching between instance classes from a list of instance classes.

[0066] According to some embodiments, the method may further include training an artificial intelligence ("AI") system to generate or update an AI model to predict an instance of an object of interest in the first biological sample based at least in part on a plurality of sets of at least two images generated from at least one of the second image or the annotation dataset, each of at least two images in the plurality of sets of at least two images being different from one another. In some cases, training the AI ​​system to generate or update the AI ​​model to predict an instance of an object of interest based at least in part on the plurality of sets of at least two images may include: encoding, with a computing system and using an encoder, at least one of the second image or the annotation dataset to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image; training the AI ​​system to generate or update the AI ​​model to predict an instance of the object of interest based at least in part on the third encoded image and the fourth encoded image; generating, using the AI ​​model generated or updated by the AI ​​system, a fifth image and a sixth image based on the first image, the sixth image being different from the fifth image; and decoding, with the computing system and using a decoder, the fifth image and the sixth image to generate a seventh image, the seventh image including a predicted indicia for the instance of the object of interest in the first biological sample. In some cases, the AI ​​system may include at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network (“CNN”), a fully convolutional network (“FCN”), and / or the like. In some cases, training the AI ​​system to generate or update the AI ​​model to predict instances of the object of interest based at least in part on the multiple sets of at least two images may also include comparing, with the computing system, the seventh image to the second image to generate an instance segmentation evaluation result.

[0067] In some embodiments, the third encoded image may include a centroid of each of the first plurality of objects of interest based on the first user input, wherein the fourth encoded image may include a generated boundary of each of the first plurality of objects of interest. According to some embodiments, encoding the second image to generate the third encoded image may include: calculating, with the computing system, a first distance metric between each pixel in the third encoded image and each centroid of each of the first plurality of objects of interest; and calculating, with the computing system, a first function to generate a first proximity map, the first function being a function of the first distance metric, the third encoded image including the first proximity map. Similarly, encoding the second image to generate the fourth encoded image may include: calculating, with the computing system, a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of an edge or boundary of each of the first plurality of images of interest; and calculating, with the computing system, a second function to generate a second proximity map, the second function being a function of the second distance metric, the fourth encoded image including the second proximity map.

[0068] According to some embodiments, the method may also include assigning, with the computing system, a first weighted pixel value to each pixel in the third encoded image based at least in part on at least one of a first distance metric, a first function, or a first proximity map calculated for each pixel; and assigning, with the computing system, a second weighted pixel value to each pixel in the fourth encoded image based at least in part on at least one of a second distance metric, a second function, or a second proximity map calculated for each pixel.

[0069] In some embodiments, the method may further include determining, with the computing system, a first pixel loss value between each pixel in the third encoded image and a corresponding pixel in the fifth image; determining, with the computing system, a second pixel loss value between each pixel in the fourth encoded image and a corresponding pixel in the sixth image; calculating, with the computing system, a loss value based on a first weighted pixel value for each pixel in the third encoded image multiplied by the product of the first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image, and a second weighted pixel value for each pixel in the fourth encoded image multiplied by the product of the second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image; and updating, with the AI ​​system, the AI ​​model by updating one or more parameters of the AI ​​model based on the calculated loss value. In some cases, the loss function may include one of a mean squared error loss function, a mean squared logarithmic error loss function, a mean absolute error loss function, a Huber loss function, a weighted sum of squared differences loss function, and / or the like. In this case, generating the fifth and sixth images may include generating the fifth and sixth images based on the first image using the updated AI model.

[0070] According to some embodiments, decoding the fifth and sixth images to generate the seventh image may include decoding, with a computing system and using a decoder, the fifth and sixth images to generate the seventh image by applying at least one of one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or one or more machine learning operations that directly decode the fifth and sixth images to generate the seventh image. In some cases, applying at least one of the one or more morphological operations or the one or more machine learning operations may include applying one or more morphological operations, wherein the method may further include applying a watershed algorithm to generate the seventh image after decoding the fifth and sixth images by applying the one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images.

[0071] In another aspect, a system may include a computing system that may include at least one first processor and a first non-transitory computer-readable medium communicatively coupled to the at least one first processor. The first non-transitory computer-readable medium may store thereon computer software including a first set of instructions that, when executed by the at least one first processor, cause the computing system to: generate a user interface configured to collect training data using at least one of complete annotation or partial annotation of objects of interest within an image of a biological sample; display a first image including a field of view ("FOV") of a first biological sample within a display portion of the user interface; receive a first user input from a user via the user interface, the first user input indicating the presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface; and generate a training data set based at least in part on the location of each of the first plurality of objects identified by the first user input and at least generating a boundary around each of the first plurality of objects of interest based in part on analyzing pixels in or around the corresponding locations using an algorithm; generating at least one of a second image or an annotation dataset based on the first image, the second image including data regarding the location of each of the first plurality of objects of interest within the first image based on the received first user input and the generated boundary around each of the first plurality of objects of interest identified by the received first user input, and the annotation dataset including at least one of pixel position data or coordinate data for each of the first plurality of objects of interest within the first image based on the first user input and the generated boundary around each of the first plurality of objects of interest identified by the received first user input.

[0072] According to some embodiments, the computing system may include one of a computing system located in a work environment, a remote computing system located outside the work environment and accessible via a network, a network server, a network browser, a cloud computing system, and / or the like. In some cases, the work environment may include at least one of a laboratory, a clinic, a medical facility, a research facility, a healthcare facility, or a room. In some cases, the first biological sample may include one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like. In some cases, the object of interest may include at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, an organ structure, and / or the like. In some cases, the first user input may include one of a click input or a bounding area input, wherein the click input defines the location of the center of mass of each of at least one first object of the first plurality of objects of interest identified by the click input, wherein the bounding area input defines an area within the first image that marks the outer limit of the perimeter of at least one second object of the first plurality of objects of interest identified by the bounding area input, wherein the bounding area input includes one of a rectangular bounding area input, a circular bounding area input, a polygon placement input, or a line placement input.

[0073] Various modifications and additions may be made to the discussed embodiments without departing from the scope of the invention. For example, while the embodiments described above refer to particular features, the scope of the invention also includes embodiments having different combinations of features and embodiments that do not include all of the above features.

[0074] Specific exemplary embodiments

[0075] I. Deep Learning-Based Segmentation via Regression Layers:

[0076] In images with many instances of similar or related objects / features or objects / features of a particular type (which may be touching or partially overlapping with other objects / features), difficulties may arise in accurately identifying objects / features within the image. Instance segmentation reflects the challenge of identifying all instances of an object and its corresponding features (e.g., shape / contour). An example of such a task is cell nucleus segmentation in microscopy images, which is a primary task in many digital pathology procedures (e.g., cell nucleus counting, cell nucleus classification, and various cancer grading tasks, etc.). In such applications, manual segmentation may be difficult or impractical due to the large number of nuclei in a whole slide image ("WSI"), in which case automatic cell nucleus segmentation may be required. However, due to the diversity of cell nucleus shape, color, orientation, density, and other features, as well as other factors such as image quality, resolution, differences in tissue and staining type, and the large size of the WSI, robust automatic cell nucleus segmentation is an extremely challenging task.

[0077] In one embodiment, for robust encoding of cell nucleus morphology—instead of encoding each cell as a different class of object (as done in semantic segmentation)—the morphology of the cell nucleus (i.e., the object to be segmented) can be encoded using two surface encodings: the distance to the nucleus center and the nucleus outline. This encoding is robust because the morphology is encoded using many pixels and is not affected by single pixel errors. Using two surfaces to encode morphology (instead of just a distance) is novel and potentially more robust than existing methods.

[0078] In one embodiment, the advantages of a weighted scheme are provided where "important" pixels (e.g., pixels belonging to a shared boundary between two cell nuclei) are assigned higher weights, thus helping the network focus on "risky" pixels (where errors may lead to over-segmentation or under-segmentation).

[0079] In one embodiment, a method can be provided that minimizes the effort required to generate a training dataset for the instance segmentation task, built on two main components: (A) a novel scheme for instance segmentation ground truth data comprising a mixture of two types of nucleus annotations - complete annotations (where nuclei are fully specified with their centers and full outlines) and partial annotations (where nuclei are specified with only their centers); and (B) a novel approach to the instance segmentation problem comprising (i) encoding the ground truth data using two surfaces that can be robustly modeled by a fully convolutional biregression neural network (which can be trained with a mixture of complete and partial annotations) and (ii) decoding the network-predicted surfaces (for test images) into instance segmentations (based on a label-controlled watershed algorithm).

[0080] A. Example 1:

[0081] 1. Preprocessing:

[0082] 1.1 Dataset Preparation: The dataset used to train the model was provided as part of the MoNuSeg H&E stained multi-organ nucleus segmentation in the Digital Pathology Challenge held at the MICCAI 2018 conference. The training dataset consists of 30 1000x1000 image patches cropped from WSI (captured at 40x magnification) and downloaded from the TCGA archive. To ensure the diversity of the dataset, each image corresponds to a patient, with images taken from 18 hospitals and covering 7 organs. In each image patch, the nucleus segmentation (ground truth) is provided. For training purposes, since no validation set was provided, we selected 11 images for validation (these images were not used in the training phase).

[0083] 1.2 Dataset Augmentation: Due to the small number of images used for training and their diversity, we use extensive data augmentation on both the RGB and labeled image sets, including standard augmentation procedures such as rotation, mirroring, and minor resizing, as well as Figure 5 Elastic image transformation shown in . Elastic augmentation is applied by sampling the RGB image and the annotated label image using a random displacement field. The distortion level is controlled by convolving the displacement field with a Gaussian kernel with a predefined standard deviation and scale factor. Finally, the RGB image and the label image are sampled by the displacement field using bilinear interpolation and nearest neighbor, respectively. In addition, the RGB color change is done directly in the dye channel optical density space by color deconvolution. The dye optical density is randomly scaled and biased and then projected back to the RGB space as Figure 6 shown.

[0084] 2. Proposed Model

[0085] Our approach consists of three main steps (detailed below): first, the ground truth is encoded as a set of two surfaces (see Section 2.1 below); second, a fully convolutional neural network (“FCN”) based on the UNet or U-Net architecture proposed by Ronneberger et al., 2015 is trained to predict these surfaces; finally, in post-processing, the predicted surfaces are used to perform constrained watershed segmentation and predict cell nuclei segmentation (see Section 3 below).

[0086] 2.1 Ground truth encoding: For each training image, we segment the pixel-wise ground truth into non-overlapping objects (e.g., cell nuclei). We further compute the centroid of each nucleus (see Figure 4 and Figure 7 ). We now compute two distance metrics for each pixel: (a) the distance to the nucleus centroid (in pixels); and (b) the distance to the nearest nucleus edge pixel. Following the approach of Philipp Kainz et al. (Miccai 2015), we transform these distances from the nucleus center and edge. In addition, we assign a weight to each pixel. Intuitively, we want to assign higher weights to "critical" pixels (where a wrong prediction would result in over-segmentation). Specifically, we follow a weighting scheme similar to that of U-Net and assign higher weights to pixels that are close to two different nuclei.

[0087] 2.2 Network Architecture: We replace the last U-Net layer (classification layer, for semantic classification) with a regression layer that outputs two surface maps. As the loss function, we use the weighted sum of the squared differences between the encoded ground truth and the model output, as shown in Figure 2.

[0088] 3. Post-processing:

[0089] 3.1 Decoding the network predicted surface into instance segmentation: To convert the output network surface into a nucleus segmentation label map, we first apply several morphological operations, such as reconstruction opening and regional H-minimum transform, to find the foreground and background markers from the centroid surface. Finally, an edge surface regression layer is used to generate the predicted label map through the marker-controlled watershed algorithm. After applying Bayesian optimization and the aggregated Jaccard index (“AJI”) score as the objective function, the parameters of the morphological operations are set.

[0090] 3.2 Performance Evaluation: The instance segmentation performance is evaluated using two metrics that provide accuracy metrics for instance segmentation, namely AJI and Joint-Dice. Both metrics take into account various errors in instance segmentation, such as incorrectly detected pixels (false negatives or false positives), over-segmented instances (where one instance is predicted and split into two or more instances), and under-segmented instances (where two or more instances are merged and predicted as a single instance).

[0091] B. Example 2:

[0092] Dual-Regression Deep Neural Network for Nucleus Segmentation: Our approach consists of three main steps (detailed below): first, the ground truth is encoded as a set of two surfaces (see Section 2.1 below); second, a fully convolutional neural network (“FCN”) based on the UNet or U-Net architecture proposed by Ronneberger et al., 2015 is trained to predict these surfaces; finally, in post-processing, the predicted surfaces are used to perform constrained watershed segmentation and predict nucleus segmentation (see Section 3 below).

[0093] Ground truth encoding: For each training image, we segment the pixel-wise ground truth into non-overlapping objects (e.g., cell nuclei). We further compute the centroid of each nucleus (see Figure 4 and Figure 7 We now compute two distance metrics for each pixel: (a) the distance to the nucleus centroid (in pixels); and (b) the distance to the nearest nucleus edge pixel. Following the method of Philipp Kainz et al. (Miccai 2015), we transform these distances from the nucleus center and edge.

[0094] Network Architecture: We replace the last U-Net layer (classification layer, for semantic classification) with two regression layers that try to predict the surface map. As the loss function, we use the weighted sum of the squared differences between the encoded ground truth and the model output.

[0095] Post-processing: To convert the output network surface into a nucleus segmentation label map, we first apply several morphological operations, such as reconstruction opening and regional H-minimum transform, to find foreground and background markers from the centroid surface. Finally, an edge surface regression layer is used to generate a predicted label map using a marker-controlled watershed algorithm. After applying Bayesian optimization and the Aggregate Jaccard Index (“AJI”) score as the objective function, the parameters of the morphological operations are set.

[0096] Conditioning partial annotations for training deep neural networks: We exploit the inherent separation between detecting nuclei and tracking their contours, and during the training process we mask out partially annotated instances in the second channel. Specifically, we use boundary masking around semi-annotated nuclei, so that for those specific nuclei, the network is not scored for nucleus boundary predictions, but only for detection accuracy, as in

[15] . Figure 3D shown.

[0097] Results: To train our model, we used the MoNuSeg H&E-stained multi-organ nucleus segmentation dataset (from the MICCAI 2018 nucleus segmentation challenge). The competition dataset consists of 30 1000x1000 images, each cropped from WSI (captured at 40x magnification). To ensure diversity, the dataset covers seven organs collected from different patients across 18 hospitals. In each image, the nucleus annotations (ground truth) of the cells are provided. We used the method described above as part of the MoNuSeg competition and achieved an AJI score of 0.62 on the competition test set. For the evaluation described below, since the competition test set has not yet been released, we selected 14 of the 30 images for testing (these images were not used in the training phase). To simulate partially annotated data, we conducted a series of experiments with varying ratios from 10% to 100% (3-fold cross-validation), in which we randomly replaced fully annotated nuclei with only the nucleus center.

[0098] Preliminary results indicate that accessing only 50% of the fully segmented nuclei (and using only the approximate location of the nucleus center for the other 50%) only reduces algorithm performance by 2%-4%.

[0099] II. User Interface and Nucleus Segmentation Annotations:

[0100] This disclosure teaches a method that will greatly reduce the effort required to generate training datasets for instance segmentation tasks. This disclosure includes a novel deep learning training framework that is specifically developed to address the challenge of efficiently collecting segmentation training data and using it to train deep learning-based cell nucleus segmentation models that can also benefit from partially annotated training data.

[0101] The present disclosure teaches a method that greatly minimizes the effort required to generate a training dataset for cell nucleus segmentation by using a novel encoding method for training a convolutional deep neural network ("CNN") that allows for a combination of two types of cell nucleus annotations (i.e., fully segmented nuclei and only detected nuclei), and a novel user interface that facilitates the collection of these two types of annotations.

[0102] The present disclosure teaches a new scheme for instance segmentation ground truth data, comprising a hybrid of two types of cell nucleus annotations: full annotation (where the center and full outline of the cell nucleus are fully specified); and partial annotation (where only the center of the cell nucleus is specified).

[0103] This disclosure teaches a new approach to the instance segmentation problem: (i) encoding ground truth data using two surfaces that can be robustly modeled by a fully convolutional dual regression neural network (which can be trained using a mixture of complete and partial annotations) and (ii) decoding the network-predicted surfaces (for test images) into instance segmentations (based on a label-controlled watershed algorithm).

[0104] The present disclosure includes at least two parts: a user interface that facilitates the collection of complete and partial annotations and a method for combining the two types of annotations during the training of a new cell nucleus segmentation algorithm. The present disclosure further teaches the following:

[0105] (a) Ease or speed of generating training annotations: In one embodiment, a novel user interface is developed that allows domain experts to quickly (with a single mouse click) generate complete nucleus outlines or label only the approximate center of the nucleus.

[0106] (b) Leveraging Partial Annotations: Some nuclei are difficult to segment accurately, so allowing users to mark only nuclei for detection increases the number of nuclei used for training. In one embodiment, the training method marks the outlines and / or nuclei locations to provide a nucleus segmentation model.

[0107] (c) Simpler training process: The present disclosure teaches a method for rapidly labeling and / or segmenting all cells in a field of view, allowing for a simple training process. Thus, there is no need to pre-process or crop the data before training.

[0108] (d) Fully utilizing the field of view (“FOV”): All data in a particular FOV is annotated to some extent (either fully segmented or partially segmented), and the training method learns the partial annotations. This method does not require preprocessing and cropping of the data before it is useful.

[0109] (e) Improved performance: Incomplete annotations are useful for the training process and increase the amount of available training data. The present disclosure teaches that fully segmenting only 50% of the nuclei (while only recording the approximate locations of the centers of the remaining nuclei) only reduces algorithm performance by 2%-4%.

[0110] A. Collect data:

[0111] The first step in any machine learning task is to collect training data, which is often a tedious and time-consuming process, and in some cases can be very expensive (often, labeling requires experts). To this end, we teach a new semi-supervised algorithm and user interface, called "click-to-grow", which enables fast and efficient cell nucleus segmentation.

[0112] 1. Using our in-house whole slide image analysis software, the user is instructed to click on the object of interest (e.g., cell nucleus).

[0113] 2. The software applies a dedicated semi-supervised object segmentation algorithm that attempts to "expand" the click and find the precise boundaries of the object. More specifically, we apply a voting mechanism among multiple automatically generated segmentations to attempt to identify a stable outline of the cell nucleus. The resulting proposed object segmentation is immediately displayed on the screen (in real time).

[0114] 3. If the user agrees with the proposed segmentation, he or she can proceed to click on another nucleus. In this case, the proposed automatically generated segmentation is recorded and will be used as the nucleus outline during the training process. On the other hand, if the segmentation is inaccurate, the user can try to generate a new segmentation by moving (e.g., dragging, etc.) the click location, which will generate a new proposed segmentation. Alternatively, the user can mark the annotation as partial (e.g., by double-clicking it, etc.). In this case, only the click location is recorded, not the nucleus outline, which means that the user confirms the existence of the object rather than its outline.

[0115] B. Encoding partial segmentation data to train deep neural network:

[0116] Given a set of fully segmented nuclei, we teach a novel deep learning segmentation method that encodes the training data into a pair of regression channels. The first is the distance to the center of the nucleus (i.e., the detection channel), and the second is the distance to the nucleus contour (i.e., the boundary detection channel). The present disclosure teaches how this scheme can be extended to support fully annotated data and partially annotated data as well as unknown objects. The present disclosure teaches to exploit the separation between the channels that detect nuclei and the contours that track them, and to mask the partially annotated instances in the second channel during the training process. Specifically, the present disclosure teaches to use boundary masking around semi-annotated nuclei, so that for those specific nuclei, the network is not scored for nucleus boundary predictions, but only for detection accuracy, as Figure 3D In addition, we masked the pixels of unknown objects in both channels (i.e., the nucleus distance transform from the center channel and the boundary channel).

[0117] C. Double regression deep neural network for cell nucleus segmentation:

[0118] Our approach consists of three main steps (detailed below): first, the ground truth is encoded as a set of two surfaces; second, a fully convolutional neural network (“FCN”) based on the UNet or U-Net architecture proposed by Ronneberger et al., 2015 is trained to predict these surfaces; finally, in post-processing, the predicted surfaces are used to perform constrained watershed segmentation and predict cell nucleus segmentation.

[0119] Ground truth encoding: For each training image, we segment the pixel-wise ground truth into non-overlapping objects (e.g., cell nuclei). We further compute the centroid of each nucleus (see Figure 4 and Figure 7 We now compute two distance metrics for each pixel: (a) the distance to the nucleus centroid (in pixels); and (b) the distance to the nearest nucleus edge pixel. Following the method of Philipp Kainz et al. (Miccai 2015), we transform these distances from the nucleus center and edge.

[0120] Network Architecture: We replace the last U-Net layer (classification layer, for semantic classification) with two regression layers that try to predict the surface map. As the loss function, we use the weighted sum of the squared differences between the encoded ground truth and the model output.

[0121] Post-processing: To convert the output network surface into a nucleus segmentation label map, we first apply several morphological operations, such as reconstruction opening and regional H-minimum transform, to find foreground and background markers from the centroid surface. Finally, an edge surface regression layer is used to generate a predicted label map using a marker-controlled watershed algorithm. After applying Bayesian optimization and the Aggregate Jaccard Index (“AJI”) score as the objective function, the parameters of the morphological operations are set.

[0122] D. Adapting some annotations to train deep neural networks:

[0123] We exploit the inherent separation between the channels that detect nuclei and track their contours, and mask out partially annotated instances in the second channel during the training process. Specifically, we use boundary masks around semi-annotated nuclei, so that for those specific nuclei, the network is not scored on the nucleus boundary predictions, but only on the detection accuracy, e.g. Figure 3D shown.

[0124] Results: To train our model, we used the MoNuSeg H&E-stained multi-organ nucleus segmentation dataset (from the MICCAI 2018 nucleus segmentation challenge). The competition dataset consists of 30 1000x1000 images, each cropped from WSI (captured at 40x magnification). To ensure diversity, the dataset covers seven organs collected from different patients across 18 hospitals. In each image, the nucleus annotations (ground truth) of the cells are provided. We used the method described above as part of the MoNuSeg competition and achieved an AJI score of 0.62 on the competition test set. For the evaluation described below, since the competition test set has not yet been released, we selected 14 of the 30 images for testing (these images were not used in the training phase). To simulate partially annotated data, we conducted a series of experiments with varying ratios from 10% to 100% (3-fold cross-validation), in which we randomly replaced fully annotated nuclei with only the nucleus centers.

[0125] Preliminary results indicate that accessing only 50% of fully segmented nuclei (and using only the approximate location of the nucleus center for the other 50%) only reduces algorithm performance by 2%-4%.

[0126] III. Example as shown in the figure:

[0127] We now turn to the embodiment shown in the figures. Figures 1-11 Some features of methods, systems, and apparatus for implementing digital microscopy (e.g., digital pathology or live cell imaging, etc.) are shown, and more specifically, methods, systems, and apparatus for implementing digital microscopy using deep learning-based segmentation (in some cases, through multiple regression layers or other machine learning or deep learning architectures, etc.), implementing instance segmentation based on part annotations, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within a biological sample, as described above. Figures 1-11 The methods, systems, and apparatuses shown relate to examples of different embodiments including various components and steps, which may be considered alternatives or may be used in combination with each other in various embodiments. Figures 1-11 The descriptions of the illustrated methods, systems, and devices are provided for illustration purposes and should not be construed as limiting the scope of the various embodiments.

[0128] With reference to the accompanying drawings, Figure 1 is a schematic diagram illustrating a system 100 for implementing digital microscopy using deep learning-based segmentation, implementing instance segmentation based on part annotation, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within a biological sample, according to various embodiments.

[0129] exist Figure 1In a non-limiting embodiment, system 100 may include a computing system 105a, an artificial intelligence (“AI”) system 110a, and a data store or database 115a that is local to computing system 105a and / or AI system 110a. In some cases, database 115a may be external to computing system 105a but communicatively coupled to computing system 105a. In other cases, database 115a may be integrated within computing system 105a. In some embodiments, AI system 110a—which may include, but is not limited to, at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network (“CNN”), or a fully convolutional network (“FCN”) (which may include a U-Net framework, etc.), and / or the like—may be external to computing system 105a but communicatively coupled to computing system 105a, or may be integrated within computing system 105a.

[0130] According to some embodiments, the system 100 may also include a display device 120 that may allow a user 125 to view a field of view (“FOV”) of the biological sample or an image or video of the biological sample. The system 100 may also include one or more user devices 130, one or more audio sensors 135 (optional), a camera 140 (optional), and a microscope 145 (optional). In some cases, the one or more user devices 130 may include, but are not limited to, a smartphone, a mobile phone, a tablet computer, a laptop computer, a desktop computer, a keyboard, a keypad, a computer mouse, or a monitor, and / or the like. In some cases, the one or more audio sensors 135 may include, but are not limited to, one or more microphones, one or more voice recorders, or one or more audio recorders, and / or the like. In some cases, the camera 140 may include, but are not limited to, one or more eye tracking sensors, one or more motion sensors, or one or more tracking sensors, and / or the like.

[0131] According to some embodiments, one or more user devices 130 may be used to receive user input from user 125 indicating annotations or markings of objects of interest observed by user 125 while viewing the field of view of the biological sample, whether viewed on a display screen of display device 120 or viewed through an eyepiece of microscope 145. One or more audio sensors 135 may be used to record the voice or spoken annotations of user 125 while user 125 views the FOV of the biological sample on display device 120 or through an eyepiece of microscope 145. Camera 140 may capture images or video of user 125 (in some cases, capture images or video of at least one eye of user 125) while user 125 is within FOV 140a of camera 140.

[0132] The computing system 105a may be communicatively coupled (via wireless (as depicted by a lightning symbol, etc.) or a wired connection (as depicted by connecting lines) to one or more of the AI ​​system 110a, the database 115a, the display device 120, the one or more user devices 130, the one or more audio sensors 135, the camera 140, and / or the microscope 145). The computing system 105a, the AI ​​system 110a, the database 115a, the display device 120, the one or more user devices 130, the one or more audio sensors 135, the camera 140, and / or the microscope 145 may be disposed or located within a work environment 150, which may include, but is not limited to, one of a laboratory, a clinic, a medical facility, a research facility, a healthcare facility or room, and / or the like.

[0133] System 100 may also include a remote computing system 105b (optional), an AI system 110b (optional), and a database 115b (optional), which may be communicatively coupled to computing system 105a and / or AI system 110a via network 155. In some cases, remote computing system 105b may include, but is not limited to, a web server, a web browser, a cloud computing system, and / or the like. Remote computing system 105b, AI system 110b, and database 115b may also be similar, if not identical, to computing system 105a, AI system 110a, and database 115a, respectively.

[0134] By way of example only, the networks 155 may each include a local area network ("LAN"), including but not limited to a fiber optic network, an Ethernet network, a token ring network, and / or the like; a wide area network ("WAN"); a wireless wide area network ("WWAN"); a virtual network, such as a virtual private network ("VPN"); the Internet; an intranet; an extranet; a public switched telephone network ("PSTN"); an infrared network; a wireless network, including but not limited to any of the IEEE 802.11 protocol suite, Bluetooth, or any of the other protocols known in the art. TM protocol and / or any other wireless protocol; and / or any combination of these and / or other networks. In a particular embodiment, networks 155 may each include an access network of an Internet Service Provider ("ISP"). In another embodiment, networks 155 may each include an ISP's core network and / or the Internet.

[0135] In operation, computing system 105a, remote computing system 105b, and / or AI system 110a or 110b (collectively, "computing systems," etc.) may perform data augmentation on a first image and a second image (optional), the first image comprising a field of view ("FOV") of a first biological sample, and the second image comprising a marker of an instance of an object of interest in the first biological sample. In some cases, the first biological sample may include, but is not limited to, one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like, wherein the object of interest may include, but is not limited to, at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, or an organ structure, and / or the like. In some embodiments, the data augmentation of the first image and the second image may include, but is not limited to, at least one of elasticity enhancement or color enhancement and / or the like (in some cases, configured to facilitate instance segmentation).

[0136] While the focus is on biological samples as described herein, various embodiments are not limited thereto, and instance segmentation, training of the system to generate or update an AI model to predict instance segmentation, and / or user interfaces configured to facilitate user annotation of instance segmentation may be adapted for application to non-biological samples, including but not limited to chemical samples, humans, animals, plants, insects, tools, vehicles, structures, landmarks, planets, stars, specific animate objects, or specific inanimate objects, and / or the like. As used herein, "instance segmentation" may refer to the separation and / or identification of instances of an object of interest (e.g., cells, tissues, molecular structures, parts of humans, parts of animals, parts of plants, parts of insects, parts of tools, parts of vehicles, parts of physical structures, parts of landmarks, planets, stars, parts of specific animate objects, or parts of specific inanimate objects, etc.) from other instances of the object of interest or other objects of interest that are adjacent or proximate to each other. Elastic enhancement or color enhancement is used to manipulate images to highlight or move the relative position or orientation of adjacent objects of interest or adjacent instances of objects of interest, thereby facilitating instance segmentation of such objects of interest.

[0137] The computing system may receive an (enhanced) first image and an (enhanced) second image. The computing system may train the AI ​​system 110a or 110b to generate or update an AI model to predict instances of an object of interest based at least in part on a plurality of sets of at least two images generated from the second image, each of at least two images in the plurality of sets of at least two images being different from one another. In some embodiments, the at least two images may include, but are not limited to, a centroid layer image that at least highlights the centroid of each labeled instance of the object of interest in the second image and a boundary layer image that highlights the edge or boundary of each labeled instance of the object of interest in the second image. Alternatively, the at least two images may include, but are not limited to, at least a centroid layer image that highlights the centroid of each labeled instance of the object of interest in the second image, a boundary layer image that highlights the edge or boundary of each labeled instance of the object of interest in the second image, and a semantic segmentation layer image that includes semantic segmentation data for each labeled instance of the object of interest in the second image. In other alternative embodiments, the at least two images may include any number of images or surfaces that highlight different aspects of the instances of the object of interest in the first biological sample.

[0138] In some embodiments, as part of training an AI system to generate or update an AI model to predict instances of an object of interest based at least in part on multiple sets of at least two images generated from a second image, etc., the computing system may encode the (augmented) second image using an encoder (which may be part of the computing system's software and / or hardware, or may be a separate device in communication with the computing system, etc. (in some cases, a dedicated encoder, etc.)) to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image. In some embodiments, encoding the second image to generate the third encoded image may include computing, with the computing system, a center of mass of each labeled instance of the object of interest in the second image; and generating, with the computing system, the third encoded image, the third encoded image including a highlighting of the center of mass of each labeled instance of the object of interest. In some cases, encoding the second image to generate the fourth encoded image may include computing, with the computing system, an edge or boundary of each labeled instance of the object of interest in the second image; and generating, with the computing system, a fourth encoded image, the fourth encoded image including a highlighting of the edge or boundary of each labeled instance of the object of interest.

[0139] According to some embodiments, encoding the second image to generate a third encoded image may further include the computing system calculating: a first distance metric between each pixel in the third encoded image and each centroid of each labeled instance of the object of interest; and a first function generating a first proximity map, the first function being a function of the first distance metric, the third encoded image comprising the first proximity map. Similarly, encoding the second image to generate a fourth encoded image may further include the computing system calculating: a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of an edge or boundary of each labeled instance of the object of interest; and a second function generating a second proximity map, the second function being a function of the second distance metric, the fourth encoded image comprising the second proximity map. In some cases, the computing system may assign a first weighted pixel value to each pixel in the third encoded image based at least in part on at least one of the first distance metric, the first function, or the first proximity map calculated for each pixel; and may assign a second weighted pixel value to each pixel in the fourth encoded image based at least in part on at least one of the second distance metric, the second function, or the second proximity map calculated for each pixel.

[0140] In some embodiments, the computing system may determine a first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image; and may determine a second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image. The computing system may use a loss function to calculate the loss value based on a first weighted pixel value of each pixel in the third encoded image multiplied by the product of the first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image, and a second weighted pixel value of each pixel in the fourth encoded image multiplied by the product of the second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image. In some cases, the loss function may include, but is not limited to, one of a mean square error loss function, a mean square logarithmic error loss function, a mean absolute error loss function, a Huber loss function, a weighted sum of squared differences loss function, and / or the like. For example, calculating the loss value using the mean square error loss function may include adding the first weighted pixel value of each pixel in the third encoded image multiplied by the product of the first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image, and the second weighted pixel value of each pixel in the fourth encoded image multiplied by the product of the second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image.

[0141] In some embodiments, the AI ​​system may update the AI ​​model by updating one or more parameters of the AI ​​model based on the calculated loss value. In some cases, the one or more parameters may include, but are not limited to, a single parameter, a plurality of parameters between two and one hundred (inclusive), a plurality of parameters between one hundred and one thousand (inclusive), a plurality of parameters between one thousand and one million (inclusive), or more. The computing system may use the updated AI model to generate a fifth image and a sixth image based on the first image.

[0142] In some cases, the labeling of the instance of the object of interest in the second image may include, but is not limited to, at least one of a complete annotation of the first instance of the object of interest identifying the centroid and edges of the first instance of the object of interest, a partial annotation of the second instance of the object of interest identifying only the centroid of the second instance of the object of interest, or an unknown annotation (i.e., otherwise indicated as unknown) of the third instance of the object of interest identifying neither the centroid nor the edges, and / or the like. In some embodiments, the computing system may mask the second instance of the object of interest with the partial annotation in corresponding pixels in the fourth and sixth images before calculating the loss value, but not with the partial annotation in the third or fifth images; and may mask the third instance of the object of interest with the unknown annotation in corresponding pixels in the third and fifth images and in corresponding pixels in the fourth and sixth images before calculating the loss value. In some cases, for the partial or unknown annotation, masking at least a portion of the second or third instance of the object of interest may include masking out a circle in the third and / or fourth encoded images that represents a distance from the centroid or a point within the partially annotated object indicated by a user input (e.g., a mouse click, etc.). In some cases, the circle radius may be predefined, or may be calculated "on the fly" based on information from a complete annotation of objects in the same area. Although a circular mask is described, other polygonal or geometric shapes may be used as appropriate or desired. Alternatively, the mask may include changing the weight of a particular pixel in the third encoded image and the corresponding pixel in the fifth image (or a particular pixel in the fourth encoded image and the corresponding pixel in the sixth image) to the same value so that they cancel each other out when compared pixel by pixel.

[0143] The computing system may decode the fifth and sixth images using a decoder (which may be part of the computing system's software and / or hardware, or may be a separate device (in some cases, a dedicated decoder, etc.) in communication with the computing system, etc.) to generate a seventh image, the seventh image including predicted markers of instances of the object of interest in the first biological sample, in some cases by applying at least one of one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or one or more machine learning operations that directly decode the fifth and sixth images to generate the seventh image. In some cases, applying at least one of the one or more morphological operations or the one or more machine learning operations may include applying one or more morphological operations, wherein after decoding the fifth and sixth images by applying the one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images, the computing system may apply a watershed algorithm to generate the seventh image. In some cases, the one or more morphological operations may include, but are not limited to, at least one of a reconstruction open transform or a regional H-minimum transform, and / or the like.

[0144] In some embodiments, the first image and the second image (or the enhanced first and second images) can be fed through the system multiple times (i.e., over multiple iterations, including but not limited to less than ten times, between ten and one hundred times (inclusive), between one hundred and one thousand times (inclusive), between one thousand and one million times (inclusive), or more). Each time, the third encoded image is compared to the fifth image, and the fourth encoded image is compared to the sixth image, and a loss value is calculated based on a first weighted pixel value for each pixel in the third encoded image multiplied by a first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image, and based on a second weighted pixel value for each pixel in the fourth encoded image multiplied by a second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image. The calculated loss value is used to update one or more parameters of the AI ​​model to generate successive regression layers, each regression layer generating fifth and sixth images that are gradually or continuously closer to being identical to the corresponding third and fourth encoded images. As a result, with each iteration (and using each resulting or consecutive regression layer), the decoded image (i.e., the seventh image) will gradually or continuously become closer to being identical to the second image (which may be referred to herein as the ground truth image).

[0145] According to some embodiments, the computing system may compare the seventh image with the second image to generate an instance segmentation evaluation result. In some cases, generating the instance segmentation evaluation result may include evaluating the instance segmentation performance using one or more indicators, which may include but are not limited to at least one of the aggregated Jaccard index ("AJI") indicator, the F1 indicator, the dice indicator, the average dice indicator, or the joint dice indicator and / or the like. In some cases, the instance segmentation evaluation result may include but are not limited to at least one of the instance segmentation evaluation indicator, one or more numerical instance segmentation evaluation scores, or instance segmentation classification (including but not limited to true positive ("TP"), true negative ("TN"), false positive ("FP"), false negative ("FN"), over-segmentation or under-segmentation, etc.) and / or the like. The computing system may display the generated instance segmentation evaluation result on a display screen. In some cases, the seventh image may be generated by a marker-controlled watershed algorithm using a regression layer (which may include an edge surface regression layer, etc.). In some cases, the parameters for the morphological operation may be set using the instance segmentation evaluation result (e.g., AJI score, etc.) as the objective function after applying Bayesian optimization.

[0146] In some cases, training the AI ​​system to generate or update the AI ​​model to predict an instance of the object of interest based at least in part on multiple sets of at least two images generated from the second image may include encoding at least the second image to generate a third encoded image and a fourth encoded image, training the AI ​​system to generate or update the AI ​​model to predict an instance of the object of interest based at least in part on the third encoded image and the fourth encoded image, generating a fifth image and a sixth image, decoding the fifth image and the sixth image to generate a seventh image, and comparing the seventh image to the second image, etc. Although the AI ​​system is trained using two images (in this case, the third encoded image and the fourth encoded image), various embodiments are not limited thereto, and more than two images (or surfaces) may be used.

[0147] According to some embodiments, a computing system may receive an eighth image, the eighth image including a FOV of a second biological sample that is different from the first biological sample; generate two or more images based on the eighth image using an AI model generated or updated by a trained AI system, the two or more images being different from each other; and decode the two or more images using a decoder to generate a ninth image, the ninth image including predicted markers of instances of an object of interest in the second biological sample. Similar to decoding the fifth and sixth images, decoding the two or more images to generate the ninth image may include decoding the two or more images using the computing system and the decoder by applying at least one of one or more morphological operations that identify foreground and background markers in each of the two or more images before generating the ninth image, or one or more machine learning operations that directly decode the two or more images to generate the ninth image. In the case of applying the one or more morphological operations, after decoding the two or more images by applying the one or more morphological operations that identify foreground and background markers in each of the two or more images, the computing system may apply a watershed algorithm to generate the ninth image. In this way, a trained AI system and / or AI model can be used to predict labels for instances of objects of interest in new biological samples—in some cases where there are no ground truth images (or previously user-annotated images) corresponding to the new biological sample.

[0148] Alternatively or additionally, the computing system may generate a user interface configured to collect training data using at least one of complete or partial annotations of objects of interest within an image of a biological sample, and may display a first image of a FOV including a first biological sample within a display portion of the user interface. The computing system may receive a first user input from a user (e.g., a pathologist, clinician, doctor, nurse, or laboratory technician, etc.) via the user interface, the first user input indicating the presence or location of each of a first plurality of objects of interest contained in the first image displayed in the display portion of the user interface. The computing system may generate a boundary around each of the first plurality of objects of interest based at least in part on the location of each of the first plurality of objects within the first image identified by the first user input and at least in part on analyzing pixels in or around the corresponding location using an algorithm (which may include, but is not limited to, an object detection algorithm, a pixel recognition algorithm, an edge detection algorithm, and / or the like).

[0149] In some cases, the computing system may receive a second user input from a user via a user interface indicating that a point within one of the first plurality of objects of interest is moved from a previous location within the first image to a new location, and may generate a new boundary around the one of the first plurality of objects of interest contained within the first image displayed in the display portion of the user interface based at least in part on the new location of the point within the one of the first plurality of objects of interest within the first image indicated by the second user input and based at least in part on analyzing pixels in or around the new location of the point within the one of the first plurality of objects of interest using an algorithm, the new boundary replacing a previously generated boundary around the one of the first plurality of objects of interest. In some cases, the computing system may receive a third user input from a user via the user interface indicating a partial annotation of one of the second plurality of objects of interest contained within the first image displayed in the display portion of the user interface, and may generate a partial annotation symbol in the first image that identifies a location of a centroid of the one of the second plurality of objects of interest without a boundary based at least in part on the location of the third user input within the first image. In some cases, the computing system may receive a fourth user input from the user via the user interface, indicating that one of the third plurality of objects of interest is unknown or that the instance class of one of the third plurality of objects of interest should be switched to another instance class (e.g., cancer, benign, etc.), and may generate an unknown annotation symbol (i.e., a symbol or annotation, etc. representing an unknown instance or object) in the first image that identifies the location of the unknown object represented by the fourth user input based at least in part on the location of the fourth user input within the first image, or may switch the instance class of the selected one of the third plurality of objects of interest to another instance class selected by the fourth user input (e.g., switching between cancer and benign, switching between complete annotation and partial annotation, switching between partial annotation and unknown annotation, switching between complete annotation and unknown annotation, etc.).

[0150] According to some embodiments, the first user input may include, but is not limited to, one of a click input or a boundary area input. In some cases, the click input may define the location of the center of mass of a first object in the first plurality of objects of interest identified by the click input, while the boundary area input may define an area within the first image that marks the outer limits of the boundary of a second object in the first plurality of objects of interest identified by the boundary area input. In some cases, the boundary area input may include, but is not limited to, one of a rectangular boundary area input, a circular boundary area input, a polygon placement input, a line placement input, and / or the like. In some embodiments, the second user input may include, but is not limited to, a click and drag input. In some cases, the third user input may include, but is not limited to, a double-click input, wherein the third user input selects or deselects a boundary around one of the second plurality of objects of interest. In some cases, the fourth user input may include, but is not limited to, one of a shift-mouse click input or a key-press-mouse click input, wherein the fourth user input may include, but is not limited to, one of switching between a complete annotation and an unknown annotation or switching between instance classes from a list of instance classes. However, various embodiments are not limited to these specific inputs, and these inputs may be any suitable input for indicating a complete annotation, a partial annotation, and / or an unknown annotation, etc.

[0151] The computing system can generate at least one of a second image or an annotation dataset based on the first image, the second image including data regarding the location of each of the first plurality of objects of interest within the first image based on a received first user input and a generated boundary around each of the first plurality of objects of interest identified by the received first user input, and the annotation dataset including at least one of pixel location data or coordinate data for each of the first plurality of objects within the first image based on the first user input and the generated boundary around each of the first plurality of objects of interest identified by the received first user input. In this manner, the system provides a fast and efficient UI that allows users (or annotators) to generate annotations in an efficient manner. In particular, the user does not need to open any menus or follow a complex set of operations to interact with the UI of the annotation system. A complete annotation (i.e., a boundary generated around the location marked by the click input, the boundary region input, etc.) can be generated through a single operation (i.e., using a point-and-click input, a boundary region input, etc.). To change the automatically generated boundary, the user simply moves a point within the instance or object using a single operation (i.e., using a click-and-drag input, etc.), causing the system to redraw or regenerate a new boundary around the instance or object. Thus, users don't need to waste time manually drawing around the edges or boundaries of an instance or object to achieve a complete annotation. Similarly, with a single action (i.e., shift-mouse click input, key-mouse click input, or mouse / keyboard combination), a complete annotation can be changed to a partial annotation, or the class of an instance or object can be changed. Actions are not tied to specific mouse / keyboard actions; rather, any combination can be used or customized as appropriate or needed.

[0152] In some embodiments, the computing system can train the AI ​​system 110a or 110b to generate or update an AI model to predict instances of an object of interest in a first biological sample based at least in part on multiple sets of at least two images generated from at least one of the second image or annotation dataset, each of at least two images in the multiple sets of at least two images being different from each other. In some cases, training an AI system to generate or update an AI model to predict an instance of an object of interest based at least in part on at least two images can include: encoding a second image or at least one of the annotation datasets with a computing system and using an encoder (which can be part of the computing system's software and / or hardware, or can be a separate device that communicates with the computing system, etc. (in some cases, a dedicated encoder, etc.)) to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image; training the AI ​​system to generate or update the AI ​​model to predict an instance of the object of interest based at least in part on the third encoded image and the fourth encoded image; generating a fifth image and a sixth image based on the first image and based on the training using the AI ​​model generated or updated by the AI ​​system, the sixth image being different from the fifth image; decoding the fifth image and the sixth image with a computing system and using a decoder (which can be part of the computing system's software and / or hardware, or can be a separate device that communicates with the computing system, etc. (in some cases, a dedicated decoder, etc.)) to generate a seventh image, the seventh image including predicted markers of the instance of the object of interest in the first biological sample; and (optionally) comparing the seventh image with the second image with the computing system to generate an instance segmentation evaluation result. The encoding of the second image and the training of the AI ​​system 110a or 110b can also be as follows: Figure 9B etc. as described.

[0153] These and other functions of system 100 (and its components) are described in greater detail below with respect to Figures 2-9.

[0154] Figure 2A-2C (collectively, “ FIG. 2 ”) is a system flow diagram illustrating various systems 200, 200′, and 200″ for implementing digital microscopy imaging using deep learning-based segmentation, implementing instance segmentation based on part annotation, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within a biological sample, according to various embodiments. In FIG. 2 , system 200 may include a computing system 205 including an encoder 210, a U-Net framework 215 or a regression layer of the U-Net framework 215 (the U-Net framework 215 is an implementation of a fully convolutional network (“FCN”), etc.), a loss function system 220, a decoder 225, and an accuracy assessment system 230. System 200′ may differ from system 200 in that the computing system 205′ of system 200′ may further include a data augmentation system 235. The computing system 205 or 205′ may correspond to Figure 1 The computing system 105a or the computing system 105b of the system 100, etc.

[0155] refer to Figure 2A In a non-limiting example, the U-Net framework 215 may receive a first image 240 comprising a field of view ("FOV") of a first biological sample. According to some embodiments, the first biological sample may include, but is not limited to, one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like, and the object of interest may include, but is not limited to, at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, an organ structure, and / or the like. The encoder 210 may receive a second image 245 comprising labeling of instances of the object of interest in the first biological sample. In some cases, the labeling of the instances of the object of interest in the second image 245 may include, but is not limited to, at least one of a complete annotation (by a user) of the first instance of the object of interest identifying the centroid and edges of the first instance of the object of interest, a partial annotation (by a user) of only the centroid of the second instance of the object of interest identifying the centroid, or an unknown annotation (by a user) of the third instance of the object of interest not identifying the centroid and edges (i.e., otherwise indicated as unknown), and / or the like. According to some embodiments, the user may include, but is not limited to, a pathologist, a clinician, a doctor, a nurse, or a laboratory technician, among others. In some cases, first image 240 may be an image of size H x W (as in the case of a monochrome camera for brightfield microscopy, phase microscopy, etc.), H x W x RGB, or H x W x N (i.e., height x width x red / green / blue, or height x width x N, etc., where N is an integer value; as in the case of spectral imaging, fluorescence, etc.), and second image 245 may be an image of size H x W. In some cases, second image 245 may be referred to as a ground truth image, an instance segmentation ground truth image, etc.

[0156] In some embodiments, the computing system 205 or the encoder 210 may mask the second instance of the object of interest with the partial annotations in the corresponding pixels in the fourth and sixth encoded images, but not with the partial annotations in the third or fifth encoded images, before calculating the loss value. Furthermore, the computing system 205 or the encoder 210 may mask the third instance of the object of interest with the corresponding pixels in the third and fifth encoded images and the unknown annotations in the object pixels in the fourth and sixth encoded images before calculating the loss value. In some cases, for partial or unknown annotations, masking at least a portion of the second or third instance of the object of interest may include masking out a circle in the third and / or fourth encoded images that represents the distance from the centroid or a point within the partially annotated object indicated by user input (e.g., a mouse click, etc.). In some cases, the radius of the circle may be predefined or calculated "on the fly" based on information from the complete annotations of objects in the same area. While circular masking is described, other polygonal or geometric shapes may be used as appropriate or desired. Alternatively, masking may include changing the weights of a particular pixel in the third encoded image and the corresponding pixel in the fifth image (or a particular pixel in the fourth encoded image and the corresponding pixel in the sixth image) to the same value so that they cancel each other out in a pixel-by-pixel comparison.

[0157] The encoder 210 may calculate a centroid for each labeled instance of the object of interest in the second image 245; may calculate an edge or boundary for each labeled instance of the object of interest in the second image 245; may generate a third encoded image 250a, the third encoded image 250a including a highlighting of the centroid for each labeled instance of the object of interest; and may generate a fourth encoded image 250b, the fourth encoded image 250b including a highlighting of the edge or boundary for each labeled instance of the object of interest. In some embodiments, encoding the second image 245 to generate the third encoded image 250a may further include the system 200 or encoder 210 calculating: a first distance metric between each pixel in the third encoded image and each centroid for each labeled instance of the object of interest; and a first function generating a first proximity map, the first function being a function of the first distance metric, the third encoded image including the first proximity map. Similarly, encoding the second image to generate a fourth encoded image may also include system 200 or encoder 210 calculating: a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of each labeled instance of the object of interest; and a second function that generates a second proximity map, the second function being a function of the second distance metric, the fourth encoded image including the second proximity map. In some cases, the computing system may assign a first weighted pixel value (collectively, “first weight value 250c,” “nucleus channel weight 250c,” “centroid channel weight 250c,” “weight 250c,” etc.) to each pixel in the third encoded image based at least in part on at least one of the first distance metric, the first function, or the first proximity map calculated for each pixel; and may assign a second weighted pixel value (collectively, “second weight value 250d,” “edge channel weight 250d,” “edge channel weight 250d,” “weight 250d,” etc.) to each pixel in the fourth encoded image based at least in part on at least one of the second distance metric, the second function, or the second proximity map calculated for each pixel. The encoder 210 may output the third encoded image 250a and the fourth encoded image 250b (collectively referred to as “encoded image 250” or “transformed image 250”, etc.) to the loss function system 220. The encoder 210 may also output the first weighted pixel value assigned to each pixel in the third encoded image and the second weighted pixel value assigned to each pixel in the fourth encoded image to the loss function system 220.

[0158] Meanwhile, the U-Net framework 215 can receive the first image 240 and can generate a fifth image 260a and a sixth image 260b based on the first image 240, based on one or more parameters of the regression layer or AI model, or a determined update of the one or more parameters, using a regression layer or AI model of the U-Net framework 215. The generated fifth image 260a can simulate an image that includes a highlighted center of mass for each predicted instance of the object of interest (e.g., the third encoded image 250a, etc.), while the generated sixth image 260b can simulate an image that includes a highlighted edge or boundary for each predicted instance of the object of interest (e.g., the fourth encoded image 250b, etc.). The U-Net framework 215 can send the generated fifth image 260a and the generated sixth image 260b (collectively referred to as "generated image 260" or "predicted image 260," etc.) to the decoder 225 and also to the loss function system 220.

[0159] The loss function system 220 may determine a first pixel loss value between each pixel in the third encoded image 250a and a corresponding pixel in the fifth image 260a; and may determine a second pixel loss value between each pixel in the fourth encoded image 250b and a corresponding pixel in the sixth image 260b. The loss function system 220 may use a loss function to calculate the loss value based on a first weighted pixel value 250c for each pixel in the third encoded image 250a multiplied by the first pixel loss value between each pixel in the third encoded image 250a and the corresponding pixel in the fifth image 260a, and a second weighted pixel value 250d for each pixel in the fourth encoded image 250b multiplied by the second pixel loss value between each pixel in the fourth encoded image 250b and the corresponding pixel in the sixth image 260b. In some embodiments, the loss function may include, but is not limited to, one of a mean squared error loss function, a mean squared logarithmic error loss function, a mean absolute error loss function, a Huber loss function, a weighted sum of squared differences loss function, and / or the like. The loss function system 220 can update one or more parameters of the regression layer or AI model based on the calculated loss value, and can send the updated one or more parameters or the calculated loss value 255 to the U-Net framework 215. The system 200 can train the U-Net framework 215 to generate or update the AI ​​model to predict an instance of the object of interest based at least in part on the third encoded image 250a and the fourth encoded image 250b by generating or regenerating the fifth image 260a and the sixth image 260b using the updated one or more parameters. Although the system 200 uses the U-Net framework 215, various embodiments are not limited thereto and any suitable AI system can be used, including but not limited to at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network ("CNN"), or a fully convolutional network ("FCN"), and / or the like.

[0160] The decoder 225 may decode the fifth image 260a and the sixth image 260b to generate a seventh image 265, the seventh image 265 including predicted markers of instances of the object of interest in the first biological sample. In some embodiments, decoding the fifth image 260a and the sixth image 260b to generate the seventh image 265 may include decoding the fifth image 260a and the sixth image 260b to generate the seventh image 265 by applying at least one of one or more morphological operations to identify foreground and background markers in each of the fifth image 260a and the sixth image 260b before generating the seventh image 265, or one or more machine learning operations to directly decode the fifth image 260a and the sixth image 260b to generate the seventh image 265. In the case of applying the one or more morphological operations to identify foreground and background markers in each of the fifth image 260a and the sixth image 260b, after decoding the fifth image 260a and the sixth image 260b by applying the one or more morphological operations, the decoder 225 may apply a watershed algorithm to generate the seventh image 265. In some cases, the one or more morphological operations may include, but are not limited to, at least one of a reconstruction open transform or a regional H-minimum transform, and / or the like. During training, decoder 225 may output seventh image 265 to accuracy assessment system 230 .

[0161] In some cases, the accuracy assessment system 230 may compare the seventh image 265 with the enhanced second image 245' to generate an instance evaluation result by evaluating the instance segmentation performance using one or more metrics. In some cases, the one or more metrics may include, but are not limited to, at least one of an aggregate Jaccard index ("AJI") metric, an F1 metric, a dice metric, an average dice metric, or a joint dice metric and / or the like. As described above, in some cases, the seventh image 265 may be generated by a marker-controlled watershed algorithm using a regression layer (which may include an edge surface regression layer, etc.). In some cases, the parameters for the morphological operation may be set with the instance segmentation evaluation result (e.g., AJI score, etc.) as the objective function after applying Bayesian optimization. The accuracy assessment system 230 may output the instance evaluation result or comparison value as a feedback value 270. In some cases, the generated instance segmentation evaluation result 270 may be displayed on a display device (e.g., Figure 1In some cases, the instance segmentation evaluation result 270 may include, but is not limited to, at least one of an instance segmentation evaluation metric, one or more numerical instance segmentation evaluation scores, or an instance segmentation classification (including, but not limited to, true positive (“TP”), true negative (“TN”), false positive (“FP”), false negative (“FN”), over-segmentation, or under-segmentation, etc.), and / or the like.

[0162] refer to Figure 2B In a non-limiting example, the first image 240 and the second image 245 may be data enhanced by the data enhancement system 235 to generate an enhanced first image 240′ and an enhanced second image 245′, respectively. In some cases, the data enhancement of the first image 240 and the second image 245 may include, but is not limited to, elasticity enhancement or color enhancement (in some cases configured to facilitate instance segmentation) and / or the like (e.g., as shown in FIG. Figure 5 and Figure 6 The U-Net framework 215 may receive the enhanced first image 240', and the encoder 210 may receive the enhanced second image 245'. Figure 2B The encoder 210, U-Net framework (or regression layer or AI model of the U-Net framework) 215, loss function system 220, decoder 225 and accuracy evaluation system 230 can be used with Figure 2A The encoder 210, U-Net framework (or regression layer of the U-Net framework) 215, loss function system 220, decoder 225 and accuracy evaluation system 230 function in a similar manner, except that an enhanced first image 240' and an enhanced second image 245' are used instead of the first image 240 and the second image 245.

[0163] Steering Figure 2C A non-limiting example of a method of training a computer program is provided, providing a visual depiction to illustrate the training process, as described above with respect to Figure 2AAs described. In particular, the U-Net regression layer or framework 215 can receive a first image or input image 240, the first image or input image 240 comprising a field of view ("FOV") of a first biological sample. The encoder 210 can receive a second image or ground truth image 245, the second image or ground truth image 245 comprising labels of instances of an object of interest in the first biological sample. In some cases, the labels of the instances of the object of interest in the second image or ground truth image 245 can include, but are not limited to, at least one of a complete annotation (by a user) of the first instance of the object of interest identifying a centroid and edges of the first instance of the object of interest, a partial annotation (by a user) of the second instance of the object of interest identifying only a centroid of the second instance of the object of interest, or an unknown annotation (by a user) of a third instance of the object of interest identifying neither a centroid nor an edge (i.e., otherwise indicated as unknown), and / or the like. In some cases, the first image or input image 240 may be an image of size H x W (as in the case of a monochrome camera for brightfield microscopy, phase microscopy, etc.), H x W x RGB, or H x W x N (i.e., height x width x red / green / blue, or height x width x N, etc., where N is an integer value; as in the case of spectral imaging, fluorescence, etc.), and the second image or ground truth image 245 may be an image of size H x W. In some cases, the second image or ground truth image 245 may be referred to as an instance segmentation ground truth image, etc.

[0164] In some embodiments, the computing system 205 or 205' or the encoder 210 may mask the second instance of the object of interest with the partial annotation in the corresponding pixels in the fourth encoded image and the sixth image before calculating the loss value, without masking the second instance of the object of interest with the partial annotation in the third encoded image or the fifth image, and may mask the third instance of the object of interest with the unknown annotation in the corresponding pixels in the third encoded image and the fifth image and the corresponding pixels in the fourth encoded image and the sixth image before calculating the loss value. In some cases, for partial annotations or unknown annotations, masking at least a portion of the second instance or the third instance of the object of interest may include masking out a circle in the third encoded image and / or the fourth encoded image that represents the distance from the centroid or from a point within the partially annotated object represented by user input (e.g., a mouse click, etc.). In some cases, the radius of the circle may be predefined or may be calculated "on the fly" based on information from the complete annotation of objects in the same area. Although circular masks are described, other polygonal or geometric shapes may be used as appropriate or desired. Alternatively, masking may include changing the weights of a particular pixel in the third encoded image and a corresponding pixel in the fifth image (or a particular pixel in the fourth encoded image and a corresponding pixel in the sixth image) to the same value so that they cancel each other out when compared pixel by pixel.

[0165] The encoder 210 may calculate the centroid or nucleus of each labeled instance of the object of interest in the second image or ground truth image 245; may calculate the edge or boundary of each labeled instance of the object of interest in the second image or ground truth image 245; may generate a third encoded image or nucleus distance image 250a, the third encoded image 250a including a highlighting of the centroid of each labeled instance of the object of interest; and may generate a fourth encoded image or edge distance image 250b, the fourth encoded image 250b including a highlighting of the edge or boundary of each labeled instance of the object of interest. In some embodiments, encoding the second image or ground truth image 245 to generate the third encoded image 250a may further include the computing system 205 or the encoder 210 calculating: a first distance metric between each pixel in the third encoded image and each centroid of each labeled instance of the object of interest; and a first function generating a first proximity map, the first function being a function of the first distance metric, the third encoded image 250a comprising a first proximity map (or a first proximity score image). Similarly, encoding the second image or ground truth image to generate a fourth encoded image may also include the computing system 205 or encoder 210 calculating: a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of the edge or boundary of each labeled instance of the object of interest; and a second function that generates a second proximity map, the second function being a function of the second distance metric, the fourth encoded image 250b including a second proximity map (or a second proximity score image). In some cases, the computing system 205 or encoder 210 may assign a first weighted pixel value 250c to each pixel in the third encoded image based at least in part on at least one of the first distance metric, the first function, or the first proximity map calculated for each pixel; and may assign a second weighted pixel value 250d to each pixel in the fourth encoded image based at least in part on at least one of the second distance metric, the second function, or the second proximity map calculated for each pixel. As described above with respect to Figure 2A described, but as Figure 2CAs depicted in the example image representation in , the computing system 205 or encoder 210 can generate a first weight image or cell nucleus channel weight image 250c that includes a first weighted pixel value for each pixel in the third encoded image 250a, and can generate a second weight image or edge channel weight image 250d that includes a second weighted pixel value for each pixel in the fourth encoded image 250b. The encoder 210 can output the third encoded image 250a and the fourth encoded image 250b (collectively referred to as "encoded image 250" or "transformed image 250", etc.). The encoder 210 can also output the first weighted pixel value assigned to each pixel in the third encoded image and the second weighted pixel value assigned to each pixel in the fourth encoded image to the loss function system 220, in some cases as the first weight image or cell nucleus channel weight image 250c and the second weight image or edge channel weight image 250d (collectively referred to as "weight images", "weights", "weight matrices", etc.).

[0166] At the same time, the U-Net regression layer or framework 215 may receive the first image or input image 240 and may generate a fifth image 260a and a sixth image 260b based on the first image or input image 240, based on one or more parameters of the regression layer or AI model, or a determined update of the one or more parameters, etc., using the regression layer or AI model of the U-Net regression layer or framework 215 (in some cases, they may be combined into a single output image, such as the U-Net output image 260 (which is an H x W x 2 image), etc.). The generated fifth image 260a may simulate an image including a highlighting of the center of mass or cell nucleus of each predicted instance of the object of interest (such as the third encoded image 250a, etc.), and the generated sixth image 260b may simulate an image including a highlighting of the edge or boundary of each predicted instance of the object of interest (such as the fourth encoded image 250b, etc.). The U-Net regression layer or framework 215 can send the generated fifth image 260a and the generated sixth image 260b (collectively referred to as “generated image 260” or “predicted image 260”, etc.) to the decoder 225 and also to the loss function system 220.

[0167] The loss function system 220 may determine a first pixel loss value between each pixel in the third encoded image 250a and the corresponding pixel in the fifth image 260a; and may determine a second pixel loss value between each pixel in the fourth encoded image 250b and the corresponding pixel in the sixth image 260b. The loss function system 220 may use a loss function to calculate the loss value based on the product of the first weighted pixel value of each pixel in the third encoded image 250a (or the first weight image or the cell nucleus channel weight image 250c) multiplied by the first pixel loss value between each pixel in the third encoded image 250a and the corresponding pixel in the fifth image 260a, and the product of the second weighted pixel value of each pixel in the fourth encoded image 250b (or the second weight image or the edge channel weight image 250d) multiplied by the second pixel loss value between each pixel in the fourth encoded image 250b and the corresponding pixel in the sixth image 260b. In some embodiments, the loss function may include, but is not limited to, one of a mean squared error loss function, a mean squared logarithmic error loss function, a mean absolute error loss function, a Huber loss function, a weighted sum of squared differences loss function, and / or the like. The loss function system 220 may update one or more parameters of the regression layer or AI model based on the calculated loss value and may send the updated one or more parameters or the calculated loss value 255 (collectively referred to as "net weight update 255" or "parameter update 255," etc.) to the U-Net regression layer or framework 215. The computing system 205 may train the U-Net regression layer or framework 215 to generate or update the AI ​​model to predict an instance of the object of interest based at least in part on the third encoded image 250a and the fourth encoded image 250b by generating or regenerating the fifth image 260a and the sixth image 260b using the updated one or more parameters. Although the computing system 205 uses the U-Net regression layer or framework 215, various embodiments are not limited thereto and may use any suitable AI system, including but not limited to one of a machine learning system, a deep learning system, a neural network, a convolutional neural network ("CNN"), a fully convolutional network ("FCN"), and / or the like.

[0168] The decoder 225 can decode the fifth image 260a and the sixth image 260b to generate a seventh image or decoded image 265, the seventh image or decoded image 265 including predicted markers of instances of the object of interest in the first biological sample. In some embodiments, decoding the fifth image 260a and the sixth image 260b to generate the seventh image or decoded image 265 can include decoding the fifth image 260a and the sixth image 260b to generate the seventh image or decoded image 265 by applying at least one of one or more morphological operations that identify foreground and background markers in each of the fifth image 260a and the sixth image 260b before generating the seventh image or decoded image 265, or one or more machine learning operations that directly decode the fifth image 260a and the sixth image 260b to generate the seventh image or decoded image 265. After decoding the fifth image 260a and the sixth image 260b by applying the one or more morphological operations to identify foreground and background markers in each of the fifth image 260a and the sixth image 260b, the decoder 225 may apply a watershed algorithm to generate a seventh image or decoded image 265. In some cases, the one or more morphological operations may include, but are not limited to, at least one of a reconstruction open transform or a regional H-minimum transform, among others.

[0169] In some embodiments, the first image 240 and the second image 245 (or the enhanced first and second images 240' and 245') may be fed through the system multiple times (i.e., over multiple iterations, including but not limited to less than ten times, between ten and one hundred times (inclusive), between one hundred and one thousand times (inclusive), between one thousand and one million times (inclusive), or more). Each time, the third encoded image 250a is compared with the fifth image 260a, and the fourth encoded image 250b is compared with the sixth image 260b, and a loss value is calculated based on a first weighted pixel value for each pixel in the third encoded image (or the first weight image or the nucleus channel weight image 250c) multiplied by a first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image, and based on a second weighted pixel value for each pixel in the fourth encoded image (or the second weight image or the edge channel weight image 250d) multiplied by a second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image. The calculated loss value is used to update one or more parameters 255 of the AI ​​model to generate successive regression layers, each of which generates fifth and sixth images that are gradually or continuously closer to being identical to the corresponding third and fourth encoded images. As a result, with each iteration (and using each resulting or successive regression layer), the decoded image 260 (i.e., the seventh image) will gradually or continuously become closer to being identical to the ground truth image 245 (i.e., the second image). Although two images (in this case, the third encoded image 250a and the fourth encoded image 250b) are used to train the AI ​​system, various embodiments are not limited thereto, and more than two images (or surfaces) may be used.

[0170] Figure 3A-Figure 3E (collectively, “ FIG3 ”) is a diagram illustrating various embodiments 300 , 300 ′, 300 ″, 300 ′″, and 300 ″″ of a user interface for facilitating user annotation of instance segmentation within a biological sample, according to various embodiments. The user interface 305 can be configured to collect training data for predicting instance segmentation within a biological sample.

[0171] refer to Figure 3A In the non-limiting embodiment 300 of FIG. 1 , a user interface 305 can display a first image 310 (e.g., an image or video, etc.) of a first biological sample and, in some cases, can also display a field of view (“FOV”) 315 of the first image 310 of the first biological sample. Figure 1The computing system 105a or 105b of FIG. 2 or the computing system 205 or 205′ of FIG. 2 , etc., may receive a first user input from a user (e.g., a pathologist, clinician, doctor, nurse, or laboratory technician, etc.) via the user interface 305, the first user input indicating the presence or location of each of the first plurality of objects of interest contained within the first image displayed in the display portion of the user interface. The computing system may generate a boundary around each of the first plurality of objects of interest based at least in part on the location of each of the first plurality of objects within the first image identified by the first user input and based at least in part on analyzing pixels in or around the corresponding location using an algorithm (which may include, but is not limited to, an object detection algorithm, a pixel recognition algorithm, an edge detection algorithm, and / or the like).

[0172] In some cases, the computing system may receive a second user input from a user via a user interface, the second user input indicating that a point within one of the first plurality of objects of interest is moved from a previous location within the first image to a new location, and may generate a new boundary around the one of the first plurality of objects of interest contained in the first image displayed in the display portion of the user interface based at least in part on the new location of the point within the one of the first plurality of objects of interest within the first image indicated by the second user input and based at least in part on analyzing pixels in or around the new location of the point within the one of the first plurality of objects of interest using an algorithm, the new boundary replacing a boundary previously generated around the one of the first plurality of objects of interest. In some cases, the computing system may receive a third user input from a user via the user interface indicating a partial annotation of one of the second plurality of objects of interest contained in the first image displayed in the display portion of the user interface, and may generate a partial annotation symbol in the first image that identifies a location of a centroid of the one of the second plurality of objects of interest without a boundary based at least in part on the location of the third user input within the first image. In some cases, the computing system may receive a fourth user input from the user via the user interface, the fourth user input indicating that one of the third plurality of objects of interest is unknown or that the instance class of one of the third plurality of objects of interest should be switched to another instance class (e.g., cancer, benign, etc.), and may generate an unknown annotation symbol (i.e., a symbol or annotation representing an unknown instance or object, etc.) in the first image that identifies the location of the unknown object represented by the fourth user input based at least in part on the location of the fourth user input within the first image, or may switch the instance class of a selected object of the third plurality of objects of interest to another instance class selected by the fourth user input (e.g., switching between cancer and benign, switching between complete annotation and partial annotation, switching between partial annotation and unknown annotation, switching between complete annotation and unknown annotation, etc.).

[0173] In some embodiments, the first user input may include, but is not limited to, one of a click input or a boundary area input. In some cases, the click input may define the location of the center of mass of a first object in the first plurality of objects of interest identified by the click input, while the boundary area input may define an area within the first image that marks the outer limits of the boundary of a second object in the first plurality of objects of interest identified by the boundary area input. In some cases, the boundary area input may include, but is not limited to, one of a rectangular boundary area input, a circular boundary area input, a polygon placement input, a line placement input, and / or the like. In some embodiments, the second user input may include, but is not limited to, a click and drag input. In some cases, the third user input may include, but is not limited to, a double-click input, wherein the third user input selects or deselects a boundary around one of the second plurality of objects of interest. In some cases, the fourth user input may include, but is not limited to, one of a shift-mouse click input or a key-press-mouse click input, wherein the fourth user input may include, but is not limited to, one of switching between a complete annotation and an unknown annotation or switching between instance classes from a list of instance classes. However, various embodiments are not limited to these specific inputs, and these inputs may be any suitable input for indicating a complete annotation, a partial annotation, and / or an unknown annotation, etc.

[0174] like Figure 3A As shown, object 320 corresponds to a fully annotated object of interest, while object 325 corresponds to a partially annotated object of interest, and object 330 corresponds to an unknown object of interest.

[0175] Steering Figure 3B 2 , etc.), and displaying a fifth image 310 d (which may correspond to the same type of image as image 250 b in FIG. 2 , etc.) depicting a boundary or boundary region 345 surrounding each of the first plurality of objects of interest contained within the third image 310 b displayed in the display portion of the user interface.

[0176] Figure 3CA user interface 305 is depicted in which the second to fifth images 310a-310d are zoomed out to result in second to fifth images 310a'-310d' (here, images 310c' and 310d' may correspond to images of the same type as images 250c and 250d in FIG. 2, respectively, etc.), and weight values ​​(e.g., a weight matrix, etc.) may be introduced to focus the model's attention on challenging areas of the biological sample (e.g., crowded areas, etc.). In some cases, the weight values ​​or matrix may be defined by an equation, such as, but not limited to:

[0177]

[0178] Where w(x) is a weighting function applied to each pixel x, w0 is a weighting constant, d1(x) is a first distance metric between each pixel x in image 310c and each centroid or nearest center pixel of each of the first plurality of objects of interest in the center image (e.g., image 250a, image 310c, etc.), d2(x) is a second distance metric between each pixel x in image 310d and the nearest (or second nearest) edge pixel of the edge or boundary of each of the first plurality of objects of interest, and σ is a sigma value representing the cluster edge for each object of interest, and d1(x) and d2(x) always refer to pixels within the same source image.

[0179] exist Figure 3C In the non-limiting example 300", the centroid weight transformation may utilize a w0 value of, for example, 5 and a σ or sigma value of 10, resulting in a weighted image as shown in the fourth image 310c' (similar to the weighted image 250c in Figure 2, etc.), and the edge weight transformation may utilize a w0 value of, for example, 10 and a σ or sigma value of 3, resulting in a weighted image as shown in the fifth image 310d' (similar to the weighted image 250d in Figure 2, etc.). Alternatively, the weight transformation may include changing the weight of a particular pixel in the third encoded image and the corresponding pixel in the fifth image (or a particular pixel in the fourth encoded image and the corresponding pixel in the sixth image) to the same value so that they cancel each other out when compared pixel by pixel. Although two images (in this case, the fourth image 310c and the fifth image 310d) are used to train the AI ​​system, different embodiments are not so limited and more than two images (or surfaces) may be used.

[0180] refer to Figure 3D In a non-limiting embodiment 300'", the user interface 305 may display a second biological sample (different from Figure 3B and Figure 3C320, the partially annotated object 325, the unknown object 330, etc.) annotated or labeled by the user, and a seventh image 310 f depicting instance segmentations 335′ of objects of interest annotated or labeled by the user (e.g., the fully annotated object 320, the partially annotated object 325, the unknown object 330, etc.), a third image 310 f depicting a centroid 340 for each of the first plurality of objects of interest (e.g., the fully annotated object 320, etc.) and a first mask 350 depicting each of the second plurality of objects of interest (e.g., the unknown object 330, but not the partially annotated object 325, etc.) contained within the seventh image 310 f displayed in the display portion of the user interface. An eighth image 310 g (which is a centroid proximity map, which may correspond to the same type of image as image 250 a in FIG. 2 , etc.) is displayed, and a ninth image 310 h (which is an edge proximity map, which may correspond to the same type of image as image 250 b in FIG. 2 , etc.) is displayed that depicts a boundary region 345 around each object of interest in the first plurality of objects of interest (e.g., fully annotated objects 320 , etc.) and a second mask 355 depicting each of the second plurality of objects of interest (e.g., unknown object 330 and partially annotated objects 325 , etc.) contained within the seventh image 310 f displayed in the display portion of the user interface.

[0181] Steering Figure 3E In a non-limiting embodiment 300″, the user interface 305 can display a sixth image 310e of a second biological sample, and display a seventh image 310f depicting instance segmentations 335 of objects of interest annotated or labeled by a user (e.g., fully annotated objects 320, partially annotated objects 325, unknown objects 330, etc.), display a tenth image 310i depicting predicted edge proximity scores with foreground and background labels overlaid on top of the image, and display an eleventh image 310j depicting predicted instance segmentations of a first plurality of objects of interest contained within the seventh image 310f displayed in the display portion of the user interface. The system can be used to train an AI system (e.g., Figure 1 2 ) to generate or update the AI ​​model to predict an instance of the object of interest, wherein the regression layer of the AI ​​system generates a tenth image 310i, such as Figure 3EAs shown. The tenth image 310i can be used as an input image for a marker-based watershed algorithm, which can be used to generate an eleventh image 310j, in some cases by applying at least one of one or more morphological operations that identify foreground and background markers in the tenth image 310i before generating the eleventh image 310j, or one or more machine learning operations that directly decode the tenth image 310i to generate the eleventh image 310j. In the case of applying one or more morphological operations, after decoding the tenth image 310i by applying one or more morphological operations that identify foreground and background markers in the tenth image 310i, the computing system can apply the watershed algorithm to generate the eleventh image 310j. In some cases, the one or more morphological operations may include, but are not limited to, at least one of a reconstruction open transform or a regional H-minimum transform, and / or the like. The eleventh image 310j may include predicted markers for instances of the object of interest in the second biological sample. As described above, in some cases, the eleventh image 310j can be generated by a marker-controlled watershed algorithm using a regression layer (which may include an edge surface regression layer, etc.). In some cases, the parameters for the morphological operation may be set after applying Bayesian optimization using the instance segmentation evaluation result (e.g., AJI score, etc.) as the objective function. Although two images (in this case, images (not shown) used to create proximity map images 310 g and 310 h) are used to train the AI ​​system, various embodiments are not limited thereto, and more than two images (or surfaces) may be used.

[0182] Figure 4 Depicted is an example 400 of various images showing annotations of an object of interest in a raw image of a first biological sample and showing predictions of the object of interest by an artificial intelligence ("AI") system, in accordance with various embodiments.

[0183] refer to Figure 4400, depicting a ground truth image 405 and a predicted image 410. A first image 405a - also referred to herein as an "RGB image," etc. - may include a field of view ("FOV") of a first biological sample, and a second image 405b may include markings (by a user) of instances of objects of interest in the first biological sample. According to some embodiments, the first biological sample may include, but is not limited to, one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like, and the objects of interest may include, but are not limited to, at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, or an organ structure, and / or the like. The second image 405b - also referred to herein as a "ground truth segmented image," etc. - may include fully annotated (depicted in the second image 405b by a colored shape without a white dot in the middle portion) and partially annotated objects of interest (depicted in the second image 405b by a colored shape with a white dot in the middle portion). The third image 405c - also referred to herein as a "center of mass distance transform image" or the like - may include a highlighting of the center of mass of each labeled instance of the object of interest (both fully annotated and partially annotated), while the fourth image 405d - also referred to herein as an "edge distance transform image" or the like - may include a highlighting of the edges or boundaries of each labeled instance of the object of interest (for fully annotated objects), with each partially annotated object being masked out.

[0184] The fifth image 410a - also referred to herein as a "predicted centroid distance transform image" or the like - may include a highlighting of the centroid of each predicted instance of the object of interest, while the sixth image 410b - also referred to herein as a "predicted edge distance transform image" or the like - may include a highlighting of the edge or boundary of each predicted instance of the object of interest. The seventh image 410c may include foreground markers (depicted by red dots, etc. in the seventh image 410c) and background markers (depicted by a green background, etc. in the seventh image 410c). The eighth image 410d - also referred to herein as an "instance segmentation prediction image" or "decoded image" or the like - may include predicted markers of instances of the object of interest in the first biological sample. In some embodiments, in addition to highlighting nuclei that were missed in the original annotation, the system may also highlight weakly or partially annotated nuclei or correctly segmented objects.

[0185] Figure 5 Depicted are examples 500 of various images showing elastic enhancement of an original image of a first biological sample and elastic enhancement of an annotated image of the original image, in accordance with various embodiments.

[0186] refer to Figure 5In the non-limiting example 500 of , a first image 505a - also referred to herein as an "RGB image," etc. - may include a field of view ("FOV") of a first biological sample, and a second image 505b - also referred to herein as an "instance segmentation image," etc. - may include markings (by a user) of instances of objects of interest in the first biological sample. According to some embodiments, the first biological sample may include, but is not limited to, one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like, and the objects of interest may include, but are not limited to, at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, or an organ structure, and / or the like. The first image 505a and the second image 505b may each be provided with grid lines overlaid on the FOV of the first biological sample or the annotated image of the first biological sample to illustrate deformation, but need not be used for the instance segmentation process.

[0187] The third image 505c—also referred to herein as a “deformed RGB image” or the like—may include an elastic enhancement of the first image 505a, and the fourth image 505d—also referred to herein as a “deformed instance segmentation image” or the like—may include an elastic enhancement of the second image 505b. Figure 5 As shown, the grid lines in the third image 505c and the fourth image 505d are used to highlight the increased elasticity of the first image 505a and the second image 505b.

[0188] Figure 6 Depicted are examples 600 of various images showing color enhancement of an original image of a first biological sample, according to various embodiments.

[0189] refer to Figure 6 In the non-limiting example 600, a first image 605a, also referred to herein as an "RGB image," etc., may include a field of view ("FOV") of a first biological sample, and each of a second image 605b, a third image 605c, and a fourth image 605d, also referred to herein as a "color-changed RGB image," etc., may include a color change of the FOV of the first biological sample to highlight an object of interest of a different color contained within the FOV of the first biological sample. According to some embodiments, the first biological sample may include, but is not limited to, one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like, and the object of interest may include, but is not limited to, at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, an organ structure, and / or the like.

[0190] Figure 7 Depicted is an example 700 of various images showing predicted efficacy based on fully and partially segmented objects of interest, in accordance with various embodiments.

[0191] refer to Figure 7 In the non-limiting example 700 of , a first image 705a - also referred to herein as an "RGB image," etc. - may include a field of view ("FOV") of a first biological sample. According to some embodiments, the first biological sample may include, but is not limited to, one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like, and the object of interest may include, but is not limited to, at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, or an organ structure, and / or the like. A second image 705b, a third image 705c, and a fourth image 705d (collectively referred to herein as "ground truth images," etc.) may include markings (by a user) of instances of the object of interest in the first biological sample. The second image 705b - also referred to herein as a "ground truth centroid distance transform image," etc. - may include highlighting of the centroid of each marked instance of the object of interest, while the third image 705c - also referred to herein as a "ground truth edge distance transform image," etc. - may include highlighting of the edge or boundary of each marked instance of the object of interest. The fourth image 705d—also referred to herein as a “ground truth instance segmentation image” or the like—may include markings (by a user) of instances of objects of interest in the first biological sample based at least in part on a combination of the second image 705b and the third image 705c.

[0192] The fifth image 710b, the sixth image 710c, and the seventh image 710d (collectively referred to as "fully segmented images 710," etc.) depict AI model predictions when the training model is trained with fully segmented annotations (i.e., 100% labeling (by the user) of instances of the object of interest in the first biological sample). The fifth image 710b—also referred to herein as a "fully segmented centroid distance transform image," etc.—may include a highlighting of the predicted centroid for each labeled instance of the object of interest, while the sixth image 710c—also referred to herein as a "fully segmented edge distance transform image," etc.—may include a highlighting of the predicted edges or boundaries for each labeled instance of the object of interest. The seventh image 710d—also referred to herein as a "fully segmented instance segmentation image," etc.—may include predicted labels for instances of the object of interest in the first biological sample based at least in part on a combination of the fifth image 710b and the sixth image 710c.

[0193] The eighth image 715b, the ninth image 715c, and the tenth image 715d (collectively referred to as "50% partial segmentation images 715," etc.) depict AI model predictions when the training model is trained with 50% segmentation annotations (i.e., the labels (by the user) of instances of an object of interest in a first biological sample include ground truth with both centroids and edges for 50% of the instances, while 50% of the instances have ground truth with only centroids). The eighth image 715b, also referred to herein as a "50% partial segmentation centroid distance transform image," etc., may include a highlighting of the predicted centroid for each labeled instance of the object of interest, while the ninth image 715c, also referred to herein as a "50% partial segmentation edge distance transform image," etc., may include a highlighting of the predicted edges or boundaries for each labeled instance of the object of interest. The tenth image 715d, also referred to herein as a "50% partial segmentation instance segmentation image," etc., may include predicted labels for instances of the object of interest in the first biological sample based at least in part on a combination of the eighth image 715b and the ninth image 715c.

[0194] Figures 8A-8D (collectively referred to as “ FIG. 8 ”) is a flow diagram illustrating a method 800 for implementing digital microscopy using deep learning-based segmentation and / or instance segmentation based on part annotation, in accordance with various embodiments. Figure 8A The method 800 continues after the circle marked "A" to Figure 8C , and after the circle mark denoted as "B" Figure 8C Return to Figure 8A . Figure 8A The method 800 continues after the circle marked "C" to Figure 8D .

[0195] Although techniques and processes are depicted and / or described in a particular order for purposes of illustration, it should be understood that certain processes may be reordered and / or omitted within the scope of various embodiments. Furthermore, although the method 800 shown in FIG. 8 may be used by (and in some cases described below with respect to) Figure 1 、 Figure 2A 、 Figure 2B 、 Figure 2C 、 Figure 3A 、 Figure 3B 、 Figure 3C 、 Figure 3D 、 Figure 3E 、 Figure 4 、 Figure 5 、 Figure 6 and Figure 7Although the corresponding systems, examples or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300", 400, 500, 600 and 700 (or components thereof) are implemented, such methods may also be implemented using any suitable hardware (or software) implementation. Similarly, although Figure 1 、 Figure 2A 、 Figure 2B 、 Figure 2C 、 Figure 3A 、 Figure 3B 、 Figure 3C 、 Figure 3D 、 Figure 3E 、 Figure 4 、 Figure 5 、 Figure 6 and Figure 7 Each of the respective systems, examples, or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300"", 400, 500, 600, and 700 (or components thereof) may operate according to the method 800 shown in FIG8 (e.g., by executing instructions embodied on a computer-readable medium), but Figure 1 、 Figure 2A 、 Figure 2B 、 Figure 2C 、 Figure 3A 、 Figure 3B 、 Figure 3C 、 Figure 3D 、 Figure 3E 、 Figure 4 、 Figure 5 、 Figure 6 and Figure 7 The systems, examples or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300"", 400, 500, 600 and 700 may also each operate according to other operating modes and / or execute other suitable procedures.

[0196] exist Figure 8A In a non-limiting embodiment, method 800 may include, at optional block 802, performing, with a computing system, data augmentation on a first image, the first image comprising a field of view (“FOV”) of a first biological sample. At optional block 804, method 800 may include performing, with the computing system, the (same) data augmentation on a second image, the second image comprising a marker of an instance of an object of interest in the first biological sample.

[0197] In some embodiments, the computing system may include, but is not limited to, one of a computing system located in a work environment, a remote computing system located outside the work environment and accessible via a network, a network server, a network browser, a cloud computing system, and / or the like. In some cases, the work environment may include, but is not limited to, at least one of a laboratory, a clinic, a medical facility, a research facility, a healthcare facility, or a room, and / or the like. In some cases, the first biological sample may include, but is not limited to, one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like. In some cases, the object of interest may include, but is not limited to, at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, or an organ structure, and / or the like. In some cases, the labeling of the instance of the object of interest in the second image may include, but is not limited to, at least one of a complete annotation of the first instance of the object of interest identifying the center of mass and edges of the first instance of the object of interest, or a partial annotation of the second instance of the object of interest identifying only the center of mass of the second instance of the object of interest, and / or the like. In some cases, data augmentation of the first and second images may include, but is not limited to, at least one of elasticity augmentation or color augmentation (in some cases configured to facilitate instance segmentation), and / or the like.

[0198] The method 800 may include receiving a first image or an enhanced first image (block 806) and receiving a second image or an enhanced second image (block 808). At block 810, the method 800 may include encoding the second image using an encoder to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image. The method 800 may continue to the process at block 812, or may continue at block 813. Figure 8C The process continues to box 838 after the circular mark represented by "A".

[0199] At block 812, method 800 may include training an artificial intelligence ("AI") system to generate or update an AI model to predict instances of an object of interest based at least in part on the third encoded image and the fourth encoded image. In some embodiments, the AI ​​system may include, but is not limited to, at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network ("CNN"), a fully convolutional network ("FCN") (which may include, for example, a U-Net framework), and / or the like. Method 800 may also include, at block 814, generating a fifth image and a sixth image based on the first image using a regression layer of the AI ​​system or the (updated) AI model, the sixth image being different from the fifth image. Method 800 may also include decoding, with a computing system, the fifth image and the sixth image using a decoder to generate a seventh image, the seventh image including predicted markers of instances of the object of interest in the first biological sample (block 816); comparing, with the computing system, the seventh image with the second image to generate an instance segmentation assessment result (optional block 818); and displaying, with the computing system, the generated instance segmentation assessment result on a display screen (optional block 820). In some embodiments, decoding the fifth and sixth images to generate the seventh image (at block 816) may include decoding the fifth and sixth images using a decoder with a computing system to generate the seventh image by applying one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or by directly decoding the fifth and sixth images to generate the seventh image. In the case of applying the one or more morphological operations, after decoding the fifth and sixth images by applying the one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images, method 800 may include applying a watershed algorithm with the computing system to generate the seventh image. In some cases, the one or more morphological operations may include, but are not limited to, at least one of a reconstruction open transform or a regional H-minimum transform, and / or the like. According to some embodiments, generating an instance segmentation evaluation result (at block 818) may include evaluating instance segmentation performance using one or more metrics, which may include, but are not limited to, at least one of an aggregate Jaccard index ("AJI") metric, an F1 metric, a dice metric, an average dice metric, a joint dice metric, and / or the like. In some cases, the instance segmentation evaluation results may include, but are not limited to, at least one of an instance segmentation evaluation metric, one or more numerical instance segmentation evaluation scores, or an instance segmentation classification (including but not limited to true positive ("TP"), true negative ("TN"), false positive ("FP"), false negative ("FN"), over-segmentation or under-segmentation, etc.) and / or the like.

[0200] Method 800 may continue after the circle marked "C" to Figure 8DThe process at box 852 in .

[0201] refer to Figure 8B In some embodiments, encoding the second image to generate the third and fourth encoded images (at block 810) may include computing, with the computing system, a centroid of each labeled instance of an object of interest in the second image (block 822); and generating, with the computing system, a third encoded image, the third encoded image including a highlighting of the centroid of each labeled instance of the object of interest (block 824). In some embodiments, encoding the second image to generate the third and fourth encoded images (at block 810) may also include computing, with the computing system, a first distance metric between each pixel in the third encoded image and each centroid of each labeled instance of the object of interest (block 826); and computing, with the computing system, a first function to generate a first proximity map, the first function being a function of the first distance metric, the third encoded image including the first proximity map (block 828). Alternatively or additionally, encoding the second image to generate the third and fourth encoded images (at block 810) may include computing, with the computing system, an edge or boundary of each labeled instance of the object of interest in the second image (block 830); and generating, with the computing system, a fourth encoded image, the fourth encoded image including a highlighting of the edge or boundary of each labeled instance of the object of interest (block 832). In some embodiments, encoding the second image to generate the third and fourth encoded images (at block 810) may also include computing, with the computing system, a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of the edge or boundary of each labeled instance of the object of interest (block 834); and computing, with the computing system, a second function to generate a second proximity map, the second function being a function of the second distance metric, the fourth encoded image including the second proximity map (block 836).

[0202] According to some embodiments, the labeling of the instance of the object of interest in the second image may include, but is not limited to, at least one of a complete annotation of the first instance of the object of interest identifying the centroid and edges of the first instance of the object of interest, a partial annotation of the second instance of the object of interest identifying only the centroid of the second instance of the object of interest, or an unknown annotation of the third instance of the object of interest identifying neither the centroid nor the edges, and / or the like. Figure 8CAt block 838 (following the circled marker designated "A") in the method 800, the method 800 may include masking, with the computing system, the second instance of the object of interest using the partial annotation in corresponding pixels in the fourth encoded image and the sixth image without masking the second instance of the object of interest using the partial annotation in the third encoded image or the fifth image, and masking, with the computing system, the third instance of the object of interest using the unknown annotation in corresponding pixels in the third encoded image and the fifth image and corresponding pixels in the fourth encoded image and the sixth image. The method 800 may also include assigning, with the computing system, a first weighted pixel value to each pixel in the third encoded image based at least in part on at least one of a first distance metric, a first function, or a first proximity map calculated for each pixel (block 840), and assigning a second weighted pixel value to each pixel in the fourth encoded image based at least in part on at least one of a second distance metric, a second function, or a second proximity map calculated for each pixel (block 842). At block 844, the method 800 may include determining, with the computing system, a first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image. At box 846, method 800 may include determining, with the computing system, a second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image. Method 800 may also include, at box 848, calculating, with the computing system, a loss value based on a first weighted pixel value for each pixel in the third encoded image multiplied by a first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image, and a second weighted pixel value for each pixel in the fourth encoded image multiplied by a second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image using a loss function. The loss function may include, but is not limited to, one of a mean square error loss function, a mean square logarithmic error loss function, a mean absolute error loss function, a Huber loss function, a weighted sum of squared differences loss function, and / or the like. At box 850, method 800 may include updating, with the AI ​​system, the AI ​​model by updating one or more parameters of the AI ​​model based on the calculated loss value. Method 800 may return to after the circular marker indicated as "B" Figure 8A In some cases, generating the fifth and sixth images (at block 814) may include generating the fifth and sixth images based on the first image using a regression layer of the AI ​​system or using an updated AI model.

[0203] exist Figure 8DAt block 852 in FIG. 8 (following the circular marker denoted as “C”), method 800 may include receiving, with the computing system, an eighth image, the eighth image including a FOV of a second biological sample that is different from the first biological sample. Method 800 may also include, at block 854, generating, using the AI ​​model generated or updated by the trained AI system, two or more images based on the eighth image, the two or more images being different from each other. At block 856, method 800 may include decoding, with the computing system and using a decoder, the two or more images to generate a ninth image, the ninth image including predicted labels for instances of the object of interest in the second biological sample.

[0204] In some embodiments, decoding the fifth and sixth images to generate the seventh image may include decoding the fifth and sixth images using a computing system and a decoder by applying at least one of one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or one or more machine learning operations that directly decode the fifth and sixth images to generate the seventh image. In the case of applying one or more morphological operations, the method may include, after decoding the fifth and sixth images by applying one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images, applying a watershed algorithm to generate the seventh image (not shown in FIG8 ). In some cases, the one or more morphological operations may include, but are not limited to, at least one of a reconstruction open transform or a regional H-minimum transform and / or the like.

[0205] Similarly, decoding the two or more images to generate the ninth image may include decoding the two or more images to generate the ninth image using a computing system and a decoder by applying at least one of one or more morphological operations that identify foreground and background markers in each of the two or more images before generating the ninth image, or one or more machine learning operations that directly decode the two or more images to generate the ninth image. In the case of applying one or more morphological operations, the method may include, after decoding the two or more images by applying the one or more morphological operations that identify foreground and background markers in each of the two or more images, applying a watershed algorithm to generate the ninth image (also not shown in FIG8 ).

[0206] Figures 9A-9D (collectively, “FIG. 9”) is a flowchart illustrating a method 900 for implementing digital microscopy imaging using deep learning-based segmentation, implementing instance segmentation based on partial annotation, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within a biological sample, according to various embodiments. Figure 9A The method 900 continues after the circle marked "A" to Figure 9B, after the circle marked "B" Figure 9B Continue to Figure 9C , and after the circle mark denoted as "C" Figure 9C Continue to Figure 9D .

[0207] Although techniques and processes are depicted and / or described in a particular order for purposes of illustration, it should be understood that certain processes may be reordered and / or omitted within the scope of various embodiments. Furthermore, although the method 900 shown in FIG. 9 may be used by (and in some cases described below with respect to) Figure 1 、 Figure 2A 、 Figure 2B 、 Figure 2C 、 Figure 3A 、 Figure 3B 、 Figure 3C 、 Figure 3D 、 Figure 3E 、 Figure 4 、 Figure 5 、 Figure 6 and Figure 7 Although the corresponding systems, examples or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300", 400, 500, 600 and 700 (or components thereof) are implemented, such methods may also be implemented using any suitable hardware (or software) implementation. Similarly, although Figure 1 、 Figure 2A 、 Figure 2B 、 Figure 2C 、 Figure 3A 、 Figure 3B 、 Figure 3C 、 Figure 3D 、 Figure 3E 、 Figure 4 、 Figure 5 、 Figure 6 and Figure 7 Each of the respective systems, examples, or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300"", 400, 500, 600, and 700 (or components thereof) may operate according to the method 900 shown in FIG. 9 (e.g., by executing instructions embodied on a computer-readable medium), but Figure 1 、 Figure 2A 、 Figure 2B 、 Figure 2C 、 Figure 3A 、 Figure 3B 、 Figure 3C 、 Figure 3D 、 Figure 3E 、 Figure 4 、 Figure 5 、 Figure 6 and Figure 7 The systems, examples or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300"", 400, 500, 600 and 700 may also each operate according to other operating modes and / or execute other suitable procedures.

[0208] exist Figure 9A In a non-limiting embodiment, method 900 may include, at block 902, generating, with a computing system, a user interface configured to collect training data using at least one of a complete annotation or a partial annotation of an object of interest within an image of a biological sample. At block 904, method 900 may include displaying, with the computing system and within a display portion of the user interface, a first image including a field of view (“FOV”) of a first biological sample.

[0209] In some embodiments, the computing system may include, but is not limited to, a computing system located within the work environment, a remote computing system located outside the work environment and accessible via a network, a network server, a network browser, a cloud computing system, and / or the like. In some cases, the work environment may include, but is not limited to, at least one of a laboratory, a clinic, a medical facility, a research facility, a healthcare facility, or a room, and / or the like. In some cases, the first biological sample may include, but is not limited to, a human tissue sample, an animal tissue sample, a plant tissue sample, and / or the like.

[0210] The method 900 may also include receiving, with the computing system, a first user input from a user (e.g., a pathologist, clinician, doctor, nurse, or laboratory technician, etc.) via a user interface, the first user input indicating the presence or location of each of a first plurality of objects of interest contained within a first image displayed in a display portion of the user interface (block 906); generating, with the computing system, a boundary around each of the first plurality of objects of interest based at least in part on the location of each of the first plurality of objects within the first image identified by the first user input and based at least in part on analyzing pixels in or around the corresponding location using an algorithm (which may include, but is not limited to, an object detection algorithm, a pixel recognition algorithm, an edge detection algorithm, and / or the like) (block 908); receiving, with the computing system, a second user input from the user via the user interface, the second user input indicating a movement of a point within one of the first plurality of objects of interest from a previous location to a new location within the first image (optional block 910); generating, with the computing system, a boundary around each of the first plurality of objects of interest based at least in part on the location of each of the first plurality of objects within the first image identified by the first user input and based at least in part on analyzing pixels in or around the corresponding location using an algorithm (which may include, but is not limited to, an object detection algorithm, a pixel recognition algorithm, an edge detection algorithm, and / or the like) (block 911); The method 900 may be performed by: generating a new boundary around the one of the first plurality of objects of interest contained within the first image displayed in the display portion of the user interface based at least in part on analyzing pixels in or around the new location of the point within the one of the first plurality of objects of interest using an algorithm; receiving, with the computing system, a third user input from the user via the user interface, the third user input indicating a partial annotation of one of the second plurality of objects of interest contained within the first image displayed in the display portion of the user interface; and generating, with the computing system, a partial annotation symbol in the first image based at least in part on the location of the third user input within the first image, the partial annotation symbol identifying a location of a centroid of the one of the second plurality of objects of interest without a boundary; and generating, with the computing system, a partial annotation symbol in the first image that identifies a location of a centroid of the one of the second plurality of objects of interest without a boundary based at least in part on the location of the third user input within the first image. The method 900 may continue to the circular marker indicated as "A" after the circular marker indicated as "A" Figure 9B The process at optional box 918 in.

[0211] exist Figure 9BAt optional block 918 (following the circular marker indicated as “A”) in the first image, method 900 may include receiving, with the computing system, a fourth user input from a user via the user interface, the fourth user input indicating that one of the third plurality of objects of interest is unknown or that the instance class of one of the third plurality of objects of interest should be switched to another instance class (e.g., cancer, benign, etc.). Method 900 may also include, at optional block 920, generating, with the computing system, in the first image, an unknown annotation symbol (i.e., a symbol or annotation representing an unknown instance or object, etc.) based at least in part on a location of the fourth user input within the first image, the unknown annotation symbol identifying the location of the unknown object indicated by the fourth user input, or the instance class of the selected one of the third plurality of objects of interest may be switched to another instance class selected by the fourth user input (e.g., switching between cancer and benign, switching between a full annotation and a partial annotation, switching between a partial annotation and an unknown annotation, switching between a full annotation and an unknown annotation, etc.).

[0212] According to some embodiments, the first user input may include, but is not limited to, one of a click input or a boundary area input. In some cases, the click input may define the location of the center of mass of a first object in the first plurality of objects of interest identified by the click input, while the boundary area input may define an area within the first image that marks the outer limits of the boundary of a second object in the first plurality of objects of interest identified by the boundary area input. In some cases, the boundary area input may include, but is not limited to, one of a rectangular boundary area input, a circular boundary area input, a polygon placement input, a line placement input, and / or the like. In some embodiments, the second user input may include, but is not limited to, a click and drag input. In some cases, the third user input may include, but is not limited to, a double-click input, wherein the third user input selects or deselects a boundary around one of the second plurality of objects of interest. In some cases, the fourth user input may include, but is not limited to, one of a shift-mouse click input or a key-press-mouse click input, wherein the fourth user input may include, but is not limited to, one of switching between a complete annotation and an unknown annotation or switching between instance classes from a list of instance classes. However, various embodiments are not limited to these specific inputs, and these inputs may be any suitable input for indicating a complete annotation, a partial annotation, and / or an unknown annotation, etc.

[0213] At box 922, method 900 may include generating, with a computing system, at least one of a second image or an annotation dataset based on the first image, the second image including data regarding the location of each of the first plurality of objects of interest within the first image based on the received first user input and a generated boundary around each of the first plurality of objects of interest identified by the received first user input, and the annotation dataset including at least one of pixel position data or coordinate data for each of the first plurality of objects of interest within the first image based on the first user input and a generated boundary around each of the first plurality of objects of interest identified by the received first user input.

[0214] At optional block 924, method 900 may include performing data augmentation on the first image and the second image with the computing system. In some cases, the data augmentation of the first image and the second image may include, but is not limited to, at least one of elasticity augmentation or color augmentation (in some cases configured to facilitate instance segmentation) and / or the like. Method 900 may also include, at optional block 926, encoding the second image with the computing system and using an encoder to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image. In some cases, the third encoded image may include a first user input for each of the first plurality of objects of interest, and the fourth encoded image may include a second user input for each of the second plurality of objects of interest.

[0215] The method 900 may also include computing, with the computing system, a first distance metric between each pixel in the third encoded image and each centroid of each labeled instance of the object of interest (optional block 928); computing, with the computing system, a first function to generate a first proximity map, the first function being a function of the first distance metric, the third encoded image comprising the first proximity map (optional block 930); computing, with the computing system, a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of an edge or boundary of each labeled instance of the object of interest (optional block 932); and computing, with the computing system, a second function to generate a second proximity map, the second function being a function of the second distance metric, the fourth encoded image comprising the second proximity map (optional block 934). The method 900 may continue to the following circular marker denoted as "B" Figure 9C The process at optional box 939 in.

[0216] According to some embodiments, the labeling of the instance of the object of interest in the second image may include, but is not limited to, at least one of a complete annotation of the first instance of the object of interest identifying a centroid and edges of the first instance of the object of interest, a partial annotation of the second instance of the object of interest identifying only a centroid of the second instance of the object of interest, or an unknown annotation of the third instance of the object of interest identifying neither a centroid nor an edge, and / or the like. Figure 9C At optional box 936 (following the circular marker indicated as "B") in the fourth encoded image, method 900 may include masking, with the computing system, a second instance of the object of interest with the partial annotation in the fourth encoded image and the object pixels in the sixth image without masking the second instance of the object of interest with the partial annotation in the third encoded image or the fifth image, and masking, with the computing system, a third instance of the object of interest with the unknown annotation in the corresponding pixels in the third encoded image and the fifth image and the corresponding pixels in the fourth encoded image and the sixth image. Method 900 may also include assigning, with the computing system, a first weighted pixel value to each pixel in the third encoded image based at least in part on a first distance metric, a first function, or a first proximity map calculated for each pixel (optional box 938); and assigning, with the computing system, a second weighted pixel value to each pixel in the fourth encoded image based at least in part on at least one of a second distance metric, a second function, or a second proximity map calculated for each pixel (optional box 940). Method 900 may also include determining, with the computing system, a first pixel loss value between each pixel in the third encoded image and a corresponding pixel in the fifth image (optional block 942); determining, with the computing system, a second pixel loss value between each pixel in the fourth encoded image and a corresponding pixel in the sixth image (optional block 944); and calculating, with the computing system, a loss value based on a first weighted pixel value for each pixel in the third encoded image multiplied by the product of the first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image, and a second weighted pixel value for each pixel in the fourth encoded image multiplied by the product of the second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image (optional block 946). In some cases, the loss function may include, but is not limited to, one of a mean squared error loss function, a mean squared logarithmic error loss function, a mean absolute error loss function, a Huber loss function, a weighted sum of squared differences loss function, and / or the like. At optional block 948, method 900 may include updating, with the AI ​​system, the AI ​​model by updating one or more parameters of the AI ​​model based on the calculated loss value. Method 900 may return to the Figure 9D The process at box 950 in .

[0217] At optional block 950 (following the circular marker denoted as "C"), method 900 may include training an artificial intelligence ("AI") system to generate or update an AI model to predict instances of an object of interest based at least in part on the third encoded image and the fourth encoded image. Method 900 may also include generating a fifth image and a sixth image based on the first image using a regression layer of the AI ​​system or the (updated) AI model, the sixth image being different from the fifth image (optional block 952); decoding the fifth image and the sixth image using a decoder with a computing system to generate a seventh image, the seventh image including predicted labels of instances of the object of interest in the first biological sample (optional block 954); comparing the seventh image with the second image with a computing system to generate an instance segmentation assessment result (optional block 956); and displaying the generated instance segmentation assessment result on a display screen with the computing system (optional block 958). According to some embodiments, generating an instance segmentation evaluation result (at block 956) may include evaluating instance segmentation performance using one or more metrics, which may include, but are not limited to, at least one of an aggregated Jaccard index ("AJI") metric, an F1 metric, a dice metric, an average dice metric, or a joint dice metric, and / or the like. In some cases, the instance segmentation evaluation result may include, but are not limited to, at least one of an instance segmentation evaluation metric, one or more numerical instance segmentation evaluation scores, or an instance segmentation classification (including, but not limited to, true positive ("TP"), true negative ("TN"), false positive ("FP"), false negative ("FN"), over-segmentation, under-segmentation, etc.), and / or the like.

[0218] Exemplary Systems and Hardware Implementations

[0219] Figure 10 is a block diagram illustrating an exemplary computer or system hardware architecture according to various embodiments. Figure 10 A schematic diagram of one embodiment of a computer system 1000 of service provider system hardware is provided, which can perform the methods provided by various other embodiments, as described herein, and / or can perform the functions of a computer or hardware system (i.e., computing systems 105a, 105b, 205, and 205', artificial intelligence ("AI") systems 110a and 110b, display device 120, user device 130, encoder 210, U-Net system or architecture 215, loss function system 220, decoder 225, accuracy assessment system 230, and data augmentation system 235, etc.), as described above. Note that Figure 10 It is intended only to provide a general description of the various components, one or more of which (or none) may be used as appropriate. Figure 10 It is broadly shown how various system elements may be implemented in a relatively separate or relatively more integrated manner.

[0220] Computer or hardware system 1000 - which can represent an embodiment of a computer or hardware system (i.e., computing systems 105a, 105b, 205, and 205', AI systems 110a and 110b, display device 120, user device 130, encoder 210, U-Net system or architecture 215, loss function system 220, decoder 225, accuracy assessment system 230, and data augmentation system 235, etc.), as described above with respect to Figure 1 9 is shown to include hardware elements that may be electrically coupled (or may otherwise communicate, as appropriate) via a bus 1005. The hardware elements may include one or more processors 1010, including but not limited to one or more general-purpose processors and / or one or more special-purpose processors (e.g., microprocessors, digital signal processing chips, graphics acceleration processors, and / or the like); one or more input devices 1015, which may include but are not limited to a mouse, keyboard, and / or the like; and one or more output devices 1020, which may include but are not limited to a display device, a printer, and / or the like.

[0221] The computer or hardware system 1000 may also include (and / or communicate with) one or more storage devices 1025, which may include, but are not limited to, local and / or network accessible storage devices, and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices, such as random access memory ("RAM") and / or read-only memory ("ROM"), which may be programmable, flash-updatable, and / or the like. Such storage devices may be configured to implement any suitable data storage, including, but not limited to, various file systems, database structures, and / or the like.

[0222] The computer or hardware system 1000 may also include a communication subsystem 1030, which may include but is not limited to a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device and / or a chipset (e.g., Bluetooth TM Devices, 802.11 devices, WiFi devices, WiMax devices, WWAN devices, cellular communication facilities, etc.) and / or the like. The communication subsystem 1030 can allow data to be exchanged with a network (such as the network described below, to name just one example), other computer or hardware systems, and / or any other devices described herein. In many embodiments, the computer or hardware system 1000 will further include a working memory 1035, which can include a RAM or ROM device as described above.

[0223] The computer or hardware system 1000 may also include software elements, shown as currently located in working memory 1035, including an operating system 1040, device drivers, executable libraries, and / or other code, such as one or more application programs 1045, which may include computer programs provided by various embodiments (including but not limited to hypervisors, VMs, etc.), and / or may be designed to implement methods provided by other embodiments and / or configure systems provided by other embodiments, as described herein. By way of example only, one or more processes described with respect to the above methods may be implemented as code and / or instructions executable by a computer (and / or a processor within a computer); in one aspect, such code and / or instructions may be used to configure and / or adapt a general-purpose computer (or other device) to perform one or more operations according to the described methods.

[0224] A set of these instructions and / or codes may be encoded and / or stored on a non-transitory computer-readable storage medium, such as the storage device 1025 described above. In some cases, the storage medium may be incorporated into a computer system (e.g., system 1000). In other embodiments, the storage medium may be separate from the computer system (i.e., removable media, such as a compact disc, etc.), and / or provided in an installation package such that the storage medium can be used to program, configure, and / or adapt a general-purpose computer having the instructions / code stored thereon. These instructions may take the form of executable code, which may be executed by the computer or hardware system 1000, and / or may take the form of source code and / or installable code, which, when compiled and / or installed on the computer or hardware system 1000 (e.g., using any of a variety of commonly available compilers, installers, compression / decompression utilities, etc.), then takes the form of executable code.

[0225] It will be apparent to those skilled in the art that substantial variations can be made depending on specific requirements. For example, customized hardware (e.g., programmable logic controllers, field programmable gate arrays, application-specific integrated circuits, and / or the like) can also be used, and / or specific elements can be implemented in hardware, software (including portable software such as applets, etc.), or both. In addition, connections to other computing devices such as network input / output devices can be employed.

[0226] As described above, in one aspect, some embodiments may employ a computer or hardware system (e.g., computer or hardware system 1000) to perform methods according to various embodiments of the present invention. According to one set of embodiments, some or all of the processes of these methods are performed by computer or hardware system 1000 in response to processor 1010 executing one or more sequences of one or more instructions contained in working memory 1035 (which may be incorporated into operating system 1040 and / or other code, such as application programs 1045). These instructions may be read into working memory 1035 from another computer-readable medium (e.g., one or more storage devices 1025). By way of example only, execution of the sequences of instructions contained in working memory 1035 may cause processor 1010 to perform one or more processes of the methods described herein.

[0227] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any medium that participates in providing data that causes a machine to operate in a specific manner. In embodiments implemented using a computer or hardware system 1000, various computer-readable media may be involved in providing instructions / code to the processor 1010 for execution and / or may be used to store and / or carry such instructions / code (e.g., as signals). In many implementations, the computer-readable medium is a non-transitory, physical, and / or tangible storage medium. In some embodiments, the computer-readable medium may take a variety of forms, including but not limited to non-volatile media, volatile media, and the like. Non-volatile media include, for example, optical and / or magnetic disks, such as storage device 1025. Volatile media include, but are not limited to, dynamic memory, such as working memory 1035. In some alternative embodiments, the computer-readable medium may take the form of a transmission medium, including but not limited to coaxial cables, copper wire, and optical fiber, including the wires comprising bus 1005, and various components of the communication subsystem 1030 (and / or the medium through which the communication subsystem 1030 provides communication with other devices). In another set of embodiments, transmission media may also take the form of waves (including but not limited to radio waves, acoustic waves, and / or light waves, such as those generated during radio wave and infrared data communications).

[0228] Common forms of physical and / or tangible computer readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape or any other magnetic medium, CD-ROMs, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM and EPROM, FLASH-EPROM, any other memory chip or cartridge, a carrier wave as described below, or any other medium from which a computer can read instructions and / or code.

[0229] Various forms of computer-readable media may be involved in carrying one or more sequences of one or more instructions to processor 1010 for execution. By way of example only, the instructions may initially be carried on a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions as signals over a transmission medium for receipt and / or execution by computer or hardware system 1000. According to various embodiments of the present invention, these signals (which may be in the form of electromagnetic signals, acoustic signals, optical signals, and / or the like) are examples of carrier waves on which instructions may be encoded.

[0230] The communication subsystem 1030 (and / or its components) will typically receive the signal, and the bus 1005 may then transfer the signal (and / or the data, instructions, etc. carried by the signal) to the working memory 1035, from which the processor 1005 retrieves and executes the instructions. The instructions received by the working memory 1035 may optionally be stored on the storage device 1025 before or after execution by the processor 1010.

[0231] As described above, a set of embodiments includes methods and systems for implementing digital microscopy imaging, and more particularly, relates to methods, systems, and apparatus for implementing digital microscopy imaging using deep learning-based segmentation, implementing instance segmentation based on partial annotation, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within biological samples. Figure 11 1 shows a schematic diagram of a system 1100 that can be used according to one set of embodiments. The system 1100 may include one or more user computers, user devices, or client devices 1105. The user computers, user devices, or client devices 1105 may be running various commercial UNIX TM1105. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser. The system 1100 may be a computer or system that is configured to run a web application or a web browser.

[0232] Certain embodiments operate in a networked environment, which may include a network 1110. The network 1110 may be any type of network familiar to those skilled in the art, which may support the use of various commercial (and / or free or proprietary) protocols, including but not limited to TCP / IP, SNA, TM 、IPX TM 、AppleTalk TM For example only, network 1110 (similar to Figure 1 The network 155, etc.) may each include a local area network ("LAN"), including but not limited to fiber optic networks, Ethernet networks, token ring networks, TM network, etc.; wide area network ("WAN"); wireless wide area network ("WWAN"); virtual network, such as virtual private network ("VPN"); Internet; intranet; extranet; public switched telephone network ("PSTN"); infrared network; wireless network, including but not limited to any IEEE 802.11 protocol suite, Bluetooth as known in the art TM The network may include a network operating under the WLAN protocol and / or any other wireless protocol; and / or any combination of these and / or other networks. In a particular embodiment, the network may include an access network of a service provider, such as an Internet service provider ("ISP"). In another embodiment, the network may include a core network of the service provider and / or the Internet.

[0233] The embodiment may also include one or more server computers 1115. Each server computer 1115 may be configured with an operating system, including but not limited to any of the operating systems discussed above, as well as any commercially (or free) available server operating system. Each server 1115 may also run one or more applications that can be configured to provide services to one or more clients 1105 and / or other servers 1115.

[0234] By way of example only, as described above, one of the servers 1115 may be a data server, a web server, a cloud computing device, etc. The data server may include (or communicate with) a web server, which, by way of example only, may be used to process requests for web pages or other electronic documents from user computers 1105. The web server may also run various server applications, including HTTP servers, FTP servers, CGI servers, database servers, Java servers, etc. In some embodiments of the present invention, the web server may be configured to provide web pages that can be operated within a web browser on one or more user computers 1105 to perform the methods of the present invention.

[0235] In some embodiments, the server computers 1115 may include one or more application servers that may be configured with one or more applications that may be accessed by clients running on one or more of the client computers 1105 and / or other servers 1115. By way of example only, the servers 1115 may be one or more general-purpose computers capable of executing programs or scripts, including but not limited to web applications (which in some cases may be configured to perform the methods provided by various embodiments), in response to the client computers 1105 and / or other servers 1115. By way of example only, the web applications may be implemented in any suitable programming language, such as Java. TM , C, C# TM or C++) and / or any scripting language (such as Perl, Python, or TCL), and any combination of programming and / or scripting languages. The application server may also include a database server, including but not limited to Oracle TM 、Microsoft TM 、Sybase TM 、IBM TMCommercially available database servers such as , which can process requests from clients running on a user computer, user device, or client device 1105 and / or another server 1115 (including, depending on the configuration, dedicated database clients, API clients, web browsers, etc.). In some embodiments, the application server can execute one or more of the processes for implementing digital microscopy imaging, more specifically, methods, systems, and apparatus for implementing digital microscopy imaging using deep learning-based segmentation, implementing instance segmentation based on part annotation, and / or implementing a user interface configured to facilitate user annotation of instance segmentation within biological samples, as described in detail above. The data provided by the application server can be formatted as one or more web pages (e.g., including HTML, JavaScript, etc.) and / or can be forwarded to the user computer 1105 via a web server (e.g., as described above). Similarly, the web server can receive web page requests and / or input data from the user computer 1105 and / or forward web page requests and / or input data to the application server. In some cases, the web server can be integrated with the application server.

[0236] According to further embodiments, one or more servers 1115 may function as a file server and / or may include one or more files (e.g., application code, data files, etc.) necessary to implement the various disclosed methods and incorporated into an application running on a user computer 1105 and / or another server 1115. Alternatively, as will be appreciated by those skilled in the art, a file server may include all necessary files to allow such an application to be remotely called by a user computer, user device, or client device 1105 and / or server 1115.

[0237] It should be noted that the functions described with respect to the various servers herein (eg, application server, database server, web server, file server, etc.) can be performed by a single server and / or multiple dedicated servers, depending on implementation-specific requirements and parameters.

[0238] In certain embodiments, the system may include one or more databases 1120a-1120n (collectively, "databases 1120"). The location of each database 1120 is arbitrary: by way of example only, database 1120a may reside on a storage medium local to (and / or residing within) server 1115a (and / or user computer, user device, or client device 1105). Alternatively, database 1120n may be remote from any or all of computers 1105, 1115, as long as it can communicate with one or more of them (e.g., via network 1110). In a specific set of embodiments, database 1120 may reside in a storage area network ("SAN") familiar to those skilled in the art. (Similarly, any necessary files for performing the functions attributed to computers 1105, 1115 may be stored locally on the respective computers and / or remotely, as appropriate.) In one set of embodiments, database 1120 may be a relational database, such as an Oracle database, adapted to store, update, and retrieve data in response to SQL-formatted commands. For example, as described above, the database may be controlled and / or maintained by a database server.

[0239] According to some embodiments, the system 1100 may also include a computing system 1125 (similar to Figure 1 computing system 105a, etc.) and a corresponding database 1130 (similar to Figure 1 The system 1100 may also include a display device 1135 (similar to Figure 1 120, etc.), which is used to allow the user 1140 to view the optical view of the first biological sample displayed on the display device 1135 (e.g., as Figure 3A-Figure 3E ). User 1140 can use one or more user devices 1145 (similar to Figure 1 In some embodiments, the system 1100 may further include one or more audio sensors 1150 (optional; similar to Figure 1 Audio sensor 135, etc.; including but not limited to one or more microphones, one or more voice recorders, or one or more audio recorders and / or the like), camera 1155 (optional; similar to Figure 1 including but not limited to one or more eye tracking sensors, one or more motion sensors, or one or more tracking sensors and / or the like) and a microscope 1160 (optional; similar to Figure 1145, etc.). In some cases, audio sensor 1150 can be used to record the voice or verbal annotations of user 1140 while the user views the FOV of the first biological sample on display device 1135 or through the eyepiece of microscope 1160. When the user views the FOV of the first biological sample on display device 1135 or through the eyepiece of microscope 1160, camera 1155 can capture an image of user 1140 (in some cases, an image of at least one eye of user 1140) when user 1140 is within the field of view (“FOV”) 1155a of camera 1155. In some cases, two or more of computing system 1125, database 1130, display device 1135, user device 1145, audio sensor 1150 (optional), camera 1155 (optional), and / or microscope 1160 (optional) can be placed in a work environment 1165, which can include, but is not limited to, at least one of a laboratory, a clinic, a medical facility, a research facility, a healthcare facility or room, and / or the like.

[0240] As an alternative to or in addition to the computing system 1125 and the corresponding database 1130, the system 1100 may also include a remote computing system 1170 (similar to Figure 1 remote computing system 105b, etc.) and a corresponding database 1175 (similar to Figure 1 In some embodiments, the system 1100 may further include an artificial intelligence ("AI") system 1180. In some embodiments, the computing systems 1125 and / or 1170 may include, but are not limited to, a computing system located within the work environment, a remote computing system located outside the work environment and accessible via a network, a network server, a network browser, a cloud computing system, and / or the like. According to some embodiments, the AI ​​system 1180 may include, but is not limited to, at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network ("CNN"), a fully convolutional neural network ("FCN"), and / or the like.

[0241] In operation, computing system 1125, remote computing system 1170, and / or AI system 1180 (collectively, "computing systems," etc.) may perform data augmentation on a first image and a second image (optional), the first image comprising a field of view ("FOV") of a first biological sample and the second image comprising a marker of an instance of an object of interest in the first biological sample. In some cases, the first biological sample may include, but is not limited to, one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and / or the like, wherein the object of interest may include, but is not limited to, at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, or an organ structure, and / or the like. In some embodiments, data augmentation of the first image and the second image may include, but is not limited to, at least one of elasticity enhancement or color enhancement and / or the like (configured in some cases to facilitate instance segmentation).

[0242] The computing system may receive an (enhanced) first image and an (enhanced) second image. The computing system may train the AI ​​system 1180 to generate or update an AI model to predict instances of an object of interest based at least in part on a plurality of sets of at least two images generated from the second image, each of at least two images in the plurality of sets of at least two images being different from one another. In some embodiments, the at least two images may include, but are not limited to, a centroid layer image that at least highlights the centroid of each labeled instance of the object of interest in the second image and a boundary layer image that highlights the edge or boundary of each labeled instance of the object of interest in the second image. Alternatively, the at least two images may include, but are not limited to, at least a centroid layer image that highlights the centroid of each labeled instance of the object of interest in the second image, a boundary layer image that highlights the edge or boundary of each labeled instance of the object of interest in the second image, and a semantic segmentation layer image that includes semantic segmentation data for each labeled instance of the object of interest in the second image. In other alternative embodiments, the at least two images may include any number of images or surfaces that highlight different aspects of the instances of the object of interest in the first biological sample.

[0243] In some embodiments, as part of training an AI system to generate or update an AI model to predict instances of an object of interest based at least in part on multiple sets of at least two images generated from a second image, etc., the computing system may use an encoder (which may be part of the computing system's software and / or hardware, or may be a separate device in communication with the computing system, etc. (in some cases, a dedicated encoder, etc.)) to encode the (augmented) second image to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image. In some embodiments, encoding the second image to generate the third encoded image may include computing, with the computing system, a center of mass of each labeled instance of the object of interest in the second image; and generating, with the computing system, the third encoded image, the third encoded image including a highlighting of the center of mass of each labeled instance of the object of interest. In some cases, encoding the second image to generate the fourth encoded image may include computing, with the computing system, an edge or boundary of each labeled instance of the object of interest in the second image; and generating, with the computing system, a fourth encoded image, the fourth encoded image including a highlighting of the edge or boundary of each labeled instance of the object of interest.

[0244] According to some embodiments, encoding the second image to generate a third encoded image may further include the computing system calculating: a first distance metric between each pixel in the third encoded image and each centroid of each labeled instance of the object of interest; and a first function that generates a first proximity map, the first function being a function of the first distance metric, the third encoded image comprising the first proximity map. Similarly, encoding the second image to generate a fourth encoded image may further include the computing system calculating: a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of an edge or boundary of each labeled instance of the object of interest; and a second function that generates a second proximity map, the second function being a function of the second distance metric, the fourth encoded image comprising the second proximity map. In some cases, the computing system may assign a first weighted pixel value to each pixel in the third encoded image based at least in part on at least one of the first distance metric, the first function, or the first proximity map calculated for each pixel; and may assign a second weighted pixel value to each pixel in the fourth encoded image based at least in part on at least one of the second distance metric, the second function, or the second proximity map calculated for each pixel.

[0245] In some embodiments, the computing system may determine a first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image; and may determine a second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image. The computing system may use a loss function to calculate the loss value based on a first weighted pixel value of each pixel in the third encoded image multiplied by the product of the first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image, and a second weighted pixel value of each pixel in the fourth encoded image multiplied by the product of the second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image. In some cases, the loss function may include, but is not limited to, one of a mean square error loss function, a mean square logarithmic error loss function, a mean absolute error loss function, a Huber loss function, a weighted sum of squared differences loss function, and / or the like.

[0246] In some embodiments, the AI ​​system may update the AI ​​model by updating one or more parameters of the AI ​​model based on the calculated loss value. In some cases, the one or more parameters may include, but are not limited to, a single parameter, a plurality of parameters between two and one hundred (inclusive), a plurality of parameters between one hundred and one thousand (inclusive), a plurality of parameters between one thousand and one million (inclusive), or more. The computing system may use the updated AI model to generate a fifth image and a sixth image based on the first image.

[0247] In some cases, the labeling of the instance of the object of interest in the second image may include, but is not limited to, at least one of a complete annotation of the first instance of the object of interest identifying a centroid and an edge of the first instance of the object of interest, a partial annotation of the second instance of the object of interest identifying only a centroid of the second instance of the object of interest, or an unknown annotation (i.e., otherwise indicated as unknown) of the third instance of the object of interest identifying neither a centroid nor an edge, and / or the like. In some embodiments, the computing system may mask the second instance of the object of interest with the partial annotation in corresponding pixels in the fourth and sixth images before calculating the loss value, but not with the partial annotation in the third or fifth images, and may mask the third instance of the object of interest with the unknown annotation in corresponding pixels in the third and fifth images and in corresponding pixels in the fourth and sixth images before calculating the loss value. In some cases, for the partial annotation or the unknown annotation, masking at least a portion of the second or third instance of the object of interest may include masking out a circle in the third and / or fourth encoded images, the circle representing a distance from the centroid or a point within the partially annotated object indicated by a user input (e.g., a mouse click, etc.). In some cases, the circle radius may be predefined, or may be calculated "on the fly" based on information from a complete annotation of objects in the same area. Although a circular mask is described, other polygonal or geometric shapes may be used as appropriate or desired. Alternatively, the mask may include changing the weight of a particular pixel in the third encoded image and the corresponding pixel in the fifth image (or a particular pixel in the fourth encoded image and the corresponding pixel in the sixth image) to the same value so that they cancel each other out when compared pixel by pixel.

[0248] The computing system may decode the fifth and sixth images using a decoder (which may be part of the computing system's software and / or hardware, or may be a separate device (in some cases, a dedicated decoder, etc.) in communication with the computing system, etc.) to generate a seventh image, the seventh image including predicted markers of instances of the object of interest in the first biological sample, in some cases by applying at least one of one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or one or more machine learning operations that directly decode the fifth and sixth images to generate the seventh image. In some cases, applying at least one of the one or more morphological operations or the one or more machine learning operations may include applying one or more morphological operations, wherein after decoding the fifth and sixth images by applying the one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images, the computing system may apply a watershed algorithm to generate the seventh image. In some cases, the one or more morphological operations may include, but are not limited to, at least one of a reconstruction open transform or a regional H-minimum transform, and / or the like.

[0249] According to some embodiments, the computing system may compare the seventh image with the second image to generate an instance segmentation evaluation result. In some cases, generating the instance segmentation evaluation result may include evaluating the instance segmentation performance using one or more indicators, which may include but are not limited to at least one of the aggregated Jaccard index ("AJI") indicator, the F1 indicator, the dice indicator, the average dice indicator, the joint dice indicator, and / or the like. In some cases, the instance segmentation evaluation result may include but are not limited to at least one of the instance segmentation evaluation indicator, one or more numerical instance segmentation evaluation scores, or instance segmentation classification (including but not limited to true positive ("TP"), true negative ("TN"), false positive ("FP"), false negative ("FN"), over-segmentation or under-segmentation, etc.) and / or the like. The computing system may display the generated instance segmentation evaluation result on a display screen.

[0250] In some cases, training the AI ​​system to generate or update the AI ​​model to predict an instance of the object of interest based at least in part on multiple sets of at least two images generated from the second image may include encoding at least the second image to generate a third encoded image and a fourth encoded image, training the AI ​​system to generate or update the AI ​​model to predict an instance of the object of interest based at least in part on the third encoded image and the fourth encoded image, generating a fifth image and a sixth image, decoding the fifth image and the sixth image to generate a seventh image, and comparing the seventh image to the second image, etc. Although the AI ​​system is trained using two images (in this case, the third encoded image and the fourth encoded image), various embodiments are not limited thereto, and more than two images (or surfaces) may be used.

[0251] According to some embodiments, a computing system may receive an eighth image, the eighth image including a FOV of a second biological sample that is different from the first biological sample; may use an AI model generated or updated by a trained AI system to generate two or more images based on the eighth image, the two or more images being different from each other; and may use a decoder to decode the two or more images to generate a ninth image, the ninth image including a predicted label of an instance of an object of interest in the second biological sample. In this way, the trained AI system and / or AI model can be used to predict labels for instances of an object of interest in a new biological sample - in some cases where there is no ground truth image (or previously user-annotated image) corresponding to the new biological sample.

[0252] Alternatively or additionally, the computing system may generate a user interface configured to collect training data using at least one of complete or partial annotation of objects of interest within an image of a biological sample, and may display a first image of a FOV including a first biological sample within a display portion of the user interface. The computing system may receive a first user input from a user (e.g., a pathologist, clinician, doctor, nurse, or laboratory technician, etc.) via the user interface, the first user input indicating the presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface. The computing system may generate a boundary around each of the first plurality of objects of interest based at least in part on the location of each of the first plurality of objects within the first image identified by the first user input and at least in part on analyzing pixels in or around the corresponding location using an algorithm (which may include, but is not limited to, an object detection algorithm, a pixel recognition algorithm, an edge detection algorithm, and / or the like).

[0253] In some cases, the computing system may receive a second user input from a user via a user interface indicating a movement of a point within one of the first plurality of objects of interest from a previous location within the first image to a new location, and may generate a new boundary around the one of the first plurality of objects of interest contained within the first image displayed in the display portion of the user interface based at least in part on the new location of the point within the one of the first plurality of objects of interest within the first image indicated by the second user input and based at least in part on analyzing pixels in or around the new location of the point within the one of the first plurality of objects of interest using an algorithm, the new boundary replacing a boundary previously generated around the one of the first plurality of objects of interest. In some cases, the computing system may receive a third user input from a user via the user interface indicating a partial annotation of one of the second plurality of objects of interest contained within the first image displayed in the display portion of the user interface, and may generate a partial annotation symbol in the first image that identifies a location of a centroid of the one of the second plurality of objects of interest without a boundary based at least in part on the location of the third user input within the first image. In some cases, the computing system may receive a fourth user input from the user via the user interface, indicating that one of the third plurality of objects of interest is unknown or that the instance class of one of the third plurality of objects of interest should be switched to another instance class (e.g., cancer, benign, etc.), and may generate an unknown annotation symbol (i.e., a symbol or annotation, etc. representing an unknown instance or object) in the first image that identifies the location of the unknown object represented by the fourth user input based at least in part on the location of the fourth user input within the first image, or may switch the instance class of a selected object from the third plurality of objects of interest to another instance class selected by the fourth user input (e.g., switching between cancer and benign, switching between complete annotation and partial annotation, switching between partial annotation and unknown annotation, switching between complete annotation and unknown annotation, etc.).

[0254] According to some embodiments, the first user input may include, but is not limited to, one of a click input or a boundary area input. In some cases, the click input may define the location of the center of mass of a first object in the first plurality of objects of interest identified by the click input, while the boundary area input may define an area within the first image that marks the outer limits of the boundary of a second object in the first plurality of objects of interest identified by the boundary area input. In some cases, the boundary area input may include, but is not limited to, one of a rectangular boundary area input, a circular boundary area input, a polygon placement input, a line placement input, and / or the like. In some embodiments, the second user input may include, but is not limited to, a click and drag input. In some cases, the third user input may include, but is not limited to, a double-click input, wherein the third user input selects or deselects a boundary around one of the second plurality of objects of interest. In some cases, the fourth user input may include, but is not limited to, one of a shift-mouse click input or a key-press-mouse click input, wherein the fourth user input may include, but is not limited to, switching between a complete annotation and an unknown annotation, or switching between instance classes from a list of instance classes, etc. However, various embodiments are not limited to these specific inputs, and these inputs may be any suitable input for indicating a complete annotation, a partial annotation, and / or an unknown annotation, etc.

[0255] The computing system can generate at least one of a second image or an annotation dataset based on the first image, the second image including data regarding the location of each of the first plurality of objects of interest within the first image based on a received first user input and a generated boundary around each of the first plurality of objects of interest identified by the received first user input, and the annotation dataset including at least one of pixel location data or coordinate data for each of the first plurality of objects within the first image based on the first user input and the generated boundary around each of the first plurality of objects of interest identified by the received first user input. In this manner, the system provides a fast and efficient UI that allows users (or annotators) to generate annotations in an efficient manner. In particular, the user does not need to open any menus or follow a complex set of operations to interact with the UI of the annotation system. A complete annotation (i.e., generating a boundary around the location marked by the click input or boundary area input, etc.) can be generated through a single operation (i.e., using a point-and-click input or a boundary area input, etc.). To change the automatically generated boundary, the user simply moves a point within the instance or object using a single operation (i.e., using a click-and-drag input, etc.), causing the system to redraw or regenerate a new boundary around the instance or object. Thus, users don't need to waste time manually drawing around the edges or boundaries of an instance or object to achieve a complete annotation. Similarly, with a single action (i.e., shift-mouse click input, key-mouse click input, or mouse / keyboard combination), a complete annotation can be changed to a partial annotation, or the class of an instance or object can be changed. Actions are not tied to specific mouse / keyboard actions; rather, any combination can be used or customized as appropriate or needed.

[0256] In some embodiments, the computing system can train the AI ​​system 1180 to generate or update an AI model to predict instances of an object of interest in a first biological sample based at least in part on multiple sets of at least two images generated from at least one of the second image or annotation dataset, each of at least two images in the multiple sets of at least two images being different from each other. In some cases, training an AI system to generate or update an AI model to predict an instance of an object of interest based at least in part on at least two images can include: encoding a second image or at least one of the annotation datasets with a computing system and using an encoder (which can be part of the computing system's software and / or hardware, or can be a separate device in communication with the computing system, etc. (in some cases, a dedicated encoder, etc.)) to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image; training the AI ​​system to generate or update the AI ​​model to predict an instance of the object of interest based at least in part on the third encoded image and the fourth encoded image; generating a fifth image and a sixth image based on the first image and based on the training using the AI ​​model generated or updated by the AI ​​system, the sixth image being different from the fifth image; decoding the fifth image and the sixth image with the computing system and using a decoder (which can be part of the computing system's software and / or hardware, or can be a separate device in communication with the computing system, etc. (in some cases, a dedicated decoder, etc.)) to generate a seventh image, the seventh image including predicted markers for the instance of the object of interest in the first biological sample; and (optionally) comparing the seventh image with the second image with the computing system to generate an instance segmentation evaluation result. The encoding of the second image and the training of the AI ​​system 1180 can also be as described above with respect to Figure 9B etc. as described.

[0257] These and other functions of the system 1100 (and its components) are described above with respect to Figure 1 - Figure 9 describes in more detail.

[0258] Although certain features and aspects have been described with respect to exemplary embodiments, those skilled in the art will recognize that many modifications are possible. For example, the methods and processes described herein may be implemented using hardware components, software components, and / or any combination thereof. Furthermore, although the various methods and processes described herein may be described with respect to specific structural and / or functional components for ease of description, the methods provided by the various embodiments are not limited to any specific structural and / or functional architecture, but may be implemented on any suitable hardware, firmware, and / or software configuration. Similarly, although certain functions are attributed to certain system components, unless the context indicates otherwise, the functions may be distributed among various other system components according to several embodiments.

[0259] Furthermore, although the processes of the methods and processes described herein are described in a particular order for ease of description, various processes may be reordered, added, and / or omitted according to various embodiments, unless the context dictates otherwise. Furthermore, processes described with respect to one method or process may be incorporated into other described methods or processes; similarly, system components described according to a particular structural architecture and / or with respect to one system may be organized and / or incorporated into other described systems in alternative structural architectures. Thus, although various embodiments with or without certain features are described for ease of description and illustration of exemplary aspects of those embodiments, various components and / or features described herein with respect to particular embodiments may be replaced, added, and / or subtracted from other described embodiments, unless the context dictates otherwise. Thus, although several exemplary embodiments are described above, it should be understood that the invention is intended to encompass all modifications and equivalents within the scope of the appended claims.

Claims

1. A method for implementing digital microscopic imaging, comprising: receiving, with a computing system, a first image comprising a field of view ("FOV") of a first biological sample; receiving, with the computing system, a second image including markings of instances of an object of interest in the first biological sample; Encoding the second image with the computing system to generate a third encoded image includes: computing, with the computing system, a center of mass for each labeled instance of an object of interest in the second image; and generating, with the computing system, the third encoded image, the third encoded image including a highlighting of a centroid of each labeled instance of an object of interest; Encoding the second image with the computing system to generate a fourth encoded image includes: computing, with the computing system, an edge or boundary of each labeled instance of an object of interest in the second image; and generating, with the computing system, the fourth encoded image, the fourth encoded image including a highlighting of an edge or boundary of each labeled instance of an object of interest; and An artificial intelligence ("AI") system is trained to generate or update an AI model to predict instances of an object of interest, wherein an input to the training of the AI ​​system includes the third encoded image and the fourth encoded image.

2. The method of claim 1 , wherein the computing system comprises one of a computing system disposed in a work environment, a remote computing system disposed outside the work environment and accessible via a network, a network server, a network browser, or a cloud computing system, wherein the work environment comprises at least one of a laboratory, a clinic, a medical facility, a research facility, a healthcare facility, or a room.

3. The method of claim 1 , wherein the AI ​​system comprises at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network (“CNN”), or a fully convolutional network (“FCN”).

4. The method of claim 1 , wherein the first biological sample comprises one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and wherein the object of interest comprises at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, or an organ structure.

5. The method according to claim 1, wherein Examples of training an AI system to generate or update an AI model to predict objects of interest include: generating a fifth image and a sixth image based on the first image using an AI model generated or updated by the AI ​​system, the fifth image including a highlighting of a center of mass of each predicted instance of the object of interest and the sixth image including a highlighting of an edge or boundary of each predicted instance of the object of interest; and The fifth image and the sixth image are decoded with the computing system using a decoder to generate a seventh image that includes predicted labels for instances of objects of interest in the first biological sample.

6. The method of claim 5, wherein training the AI ​​system to generate or update the AI ​​model to predict instances of the object of interest further comprises: The seventh image is compared with the second image with the computing system to generate an instance segmentation assessment result.

7. The method according to claim 5, wherein: Encoding the second image to generate the third encoded image further comprises: computing, with the computing system, a first distance metric between each pixel in the third encoded image and each centroid of each labeled instance of an object of interest; and computing a first function with the computing system to generate a first proximity map, the first function being a function of the first distance metric, the third encoded image comprising the first proximity map; and Encoding the second image to generate the fourth encoded image further comprises: computing, with the computing system, a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of an edge or boundary of each labeled instance of an object of interest; and A second function is calculated with the computing system to generate a second proximity map, the second function being a function of the second distance metric, the fourth encoded image including the second proximity map.

8. The method according to claim 7, further comprising: assigning, with the computing system, a first weighted pixel value to each pixel in the third encoded image based at least in part on at least one of the first distance metric, the first function, or the first proximity map calculated for each pixel; and Assigning, with the computing system, a second weighted pixel value to each pixel in the fourth encoded image based at least in part on at least one of the second distance metric, the second function, or the second proximity map calculated for each pixel.

9. The method according to claim 7, further comprising: determining, with the computing system, a first pixel loss value between each pixel in the third encoded image and a corresponding pixel in the fifth image; determining, with the computing system, a second pixel loss value between each pixel in the fourth encoded image and a corresponding pixel in the sixth image; calculating, with the computing system, a loss value based on a first weighted pixel value of each pixel in the third encoded image multiplied by a first pixel loss value between each pixel in the third encoded image and a corresponding pixel in the fifth image, and a second weighted pixel value of each pixel in the fourth encoded image multiplied by a second pixel loss value between each pixel in the fourth encoded image and a corresponding pixel in the sixth image using a loss function, wherein the loss function comprises one of a mean square error loss function, a mean square logarithmic error loss function, a mean absolute error loss function, a Huber loss function, or a weighted sum of squared differences loss function; and updating, with the AI ​​system, the AI ​​model by updating one or more parameters of the AI ​​model based on the calculated loss value; Generating the fifth image and the sixth image includes using the updated AI model to generate the fifth image and the sixth image based on the first image.

10. The method of claim 9, wherein the labeling of the instance of the object of interest in the second image comprises at least one of a complete annotation of the first instance of the object of interest identifying a centroid and edges of the first instance of the object of interest, a partial annotation of the second instance of the object of interest identifying only a centroid of the second instance of the object of interest, or an unknown annotation of the third instance of the object of interest not identifying a centroid and edges.

11. The method according to claim 10, further comprising: prior to calculating the loss value, masking, with the computing system, a second instance of the object of interest using partial annotations in corresponding pixels in the fourth encoded image and the sixth image without masking the second instance of the object of interest using partial annotations in the third encoded image or the fifth image; and Prior to calculating the loss value, masking, with the computing system, a third instance of the object of interest using unknown annotations in corresponding pixels in the third encoded image and the fifth image and in corresponding pixels in the fourth encoded image and the sixth image.

12. The method of claim 5 , wherein decoding the fifth and sixth images to generate the seventh image comprises decoding, with the computing system and using the decoder, the fifth and sixth images to generate the seventh image by applying at least one of one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or one or more machine learning operations that directly decode the fifth and sixth images to generate the seventh image.

13. The method of claim 12, wherein applying at least one of the one or more morphological operations or the one or more machine learning operations comprises applying the one or more morphological operations, wherein the method further comprises: After decoding the fifth and sixth images by applying the one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images, a watershed algorithm is applied to generate the seventh image.

14. The method of claim 13, wherein the one or more morphological operations include at least one of a reconstruction open transform or a regional H-minimum transform.

15. The method according to claim 5, further comprising: receiving, with the computing system, an eighth image, the eighth image including a FOV of a second biological sample different from the first biological sample; generating two or more images based on the eighth image using an AI model generated or updated by a trained AI system, the two or more images being different from each other; and The two or more images are decoded with the computing system and using the decoder to generate a ninth image that includes predicted signatures of instances of objects of interest in the second biological sample.

16. The method of claim 1, wherein the first image and the second image are data augmented before being received by the computing system, wherein the data augmentation of the first image and the second image comprises at least one of elastic augmentation or color augmentation configured to facilitate instance segmentation.

17. A system for implementing digital microscopic imaging, comprising: Computing systems, including: at least one first processor; and a first non-transitory computer-readable medium communicatively coupled to the at least one first processor, the first non-transitory computer-readable medium having computer software stored thereon, the computer software comprising a first set of instructions that, when executed by the at least one first processor, cause the computing system to: receiving a first image comprising a field of view ("FOV") of a first biological sample; receiving a second image comprising markings of instances of an object of interest in the first biological sample; Encoding the second image with the computing system to generate a third encoded image includes: computing, with the computing system, a center of mass for each labeled instance of an object of interest in the second image; and generating, with the computing system, the third encoded image, the third encoded image including a highlighting of a centroid of each labeled instance of an object of interest; Encoding the second image with the computing system to generate a fourth encoded image includes: computing, with the computing system, an edge or boundary of each labeled instance of an object of interest in the second image; and generating, with the computing system, the fourth encoded image, the fourth encoded image including a highlighting of an edge or boundary of each labeled instance of an object of interest; and An artificial intelligence ("AI") system is trained to generate or update an AI model to predict instances of an object of interest, wherein an input to the training of the AI ​​system includes the third encoded image and the fourth encoded image.

Images