User interface configured for easy instance segmentation within biological samples and user annotation

Abstract

Novel tools and techniques are provided for digital microscopy imaging using deep learning-based segmentation via multiple regression layers, instance segmentation based on partial annotation, and / or implementing user interfaces configured to facilitate instance segmentation. In various embodiments, the computational system can generate a user interface configured to collect training data for predicting instance segmentation within a biological sample, and can display a first image including the field of view of the biological sample within a display portion of the user interface. The computational system can receive, via the user interface, a first user input indicating the centroid of each of a first plurality of objects of interest and a second user input indicating the boundary surrounding each of the first plurality of objects of interest. The computational system can train an AI system to predict instance segmentation of objects of interest in an image of a biological sample.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application Serial No. 62 / 832,880 (“'880 Application”) entitled “DL Based Segmentation via Regression Layers” (Attorney’s File No. 20190110-01), filed April 12, 2019, by Elad Arbel et al., and U.S. Patent Application Serial No. 62 / 832,877 (“'877 Application”) entitled “Nuclei Segmentation Using Partial Annotation” (Attorney’s File No. 20190111-01), filed April 11, 2019, by Elad Arbel et al., the disclosure of each of which is incorporated herein by reference in its entirety for all purposes.

[0003] This application may relate to U.S. Patent Application Serial No. __ / __ (“'__ Application”) (Attorney’s File No. 20190111-02) entitled “User Interface Configured to Facilitate User Annotation for Instance Segmentation Within Biological Samples”, filed April 10, 2020 by Elad Arbel et al., which claims priority to applications '880 and '877, the disclosure of which is incorporated herein by reference in its entirety for all purposes.

[0004] The disclosures of these applications / patents (collectively referred to in this document as “Related Applications”) are incorporated herein by reference in their entirety for all purposes.

[0005] Copyright Notice

[0006] This patent document contains copyrighted material in part. The copyright holder does not object to anyone copying the patent document or patent disclosure appearing in the Patent and Trademark Office's patent documents or records by fax, but retains all copyrights in any event. Technical Field

[0007] This disclosure generally relates to methods, systems, and apparatuses for implementing digital microscopy (e.g., digital pathology or live-cell imaging), and more specifically, to methods, systems, and apparatuses for implementing digital microscopy using deep learning-based segmentation, implementing instance segmentation based on partial annotation, and / or implementing user interfaces configured to facilitate instance segmentation within biological samples. Background Technology

[0008] In recent years, digital pathology has become increasingly popular because many stained tissue slides are digitally scanned at high resolution (e.g., 40X) and viewed as whole slide images (“WSI”) using digital devices (e.g., PCs, tablets, etc.) instead of standard microscopes. Having information in digital format enables digital analysis that can be applied to WSI for easier diagnosis.

[0009] Given an image containing many instances of a particular type of object, instance segmentation is the problem of identifying and delineating the different instances in the image (e.g., which cells may contact or partially overlap other cells). An example of this task is nuclear segmentation in microscopic images, where all cell nuclei need to be segmented. This task is an important step in many digital pathology analyses, such as nuclear classification and various cancer grading tasks. Developing robust nuclear segmentation methods is particularly challenging due to the vast diversity of nuclear shape, color, orientation, and density in different tissues and staining types (e.g., for multi-organ nuclear segmentation or MoNuSeg).

[0010] The performance of cell nucleus segmentation algorithms depends on the size and quality of the available ground truth data used to train the model. For a field of view I of size N x M x 3, the ground truth data used for cell nucleus segmentation can be specified by an integer-valued label mat L of size N x M, where all pixels belonging to a specific cell nucleus are assigned a unique positive ID, and all background pixels are assigned a zero value. Collecting this ground truth data is very challenging because the exact boundaries of each cell nucleus in the FOV need to be specified. This tedious annotation task is performed by domain experts for thousands of cell nuclei. Therefore, the current difficulty in obtaining large amounts of training data is a factor limiting the broader applicability of deep learning models to cell nucleus segmentation. Furthermore, while deep learning (“DL”)-based models may tend to perform very well for the specific tasks they were developed for (e.g., cell nucleus segmentation in a specific tissue type, or a specific staining protocol, etc.), they often perform poorly when naively applied to different tissue types, requiring non-trivial additional annotation work in transfer learning scenarios.

[0011] In many cases, WSI analysis pipelines require robust nucleus segmentation as a fundamental building block. High-performance nucleus segmentation includes a training phase that utilizes high-quality training data annotated by domain experts (e.g., pathologists) to process multiple nucleus contours. This annotation task is difficult, time-consuming, and cumbersome, limiting the applicability of deep learning models in nucleus segmentation. When naively applied to different segmentation scenarios, deep learning-based models often perform poorly.

[0012] Therefore, there is a need for more robust and scalable solutions for realizing digital microscopy, and more specifically, for methods, systems, and apparatuses for realizing digital microscopy using deep learning-based segmentation, instance segmentation based on partial annotation, and / or implementing user interfaces configured to facilitate instance segmentation within biological samples. Attached Figure Description

[0013] A further understanding of the nature and advantages of particular embodiments can be achieved by referring to the remainder of the specification and the accompanying drawings, wherein similar reference numerals are used to refer to similar components. In some cases, sublabels are associated with reference numerals to indicate one of a plurality of similar components. When reference is made to reference numerals without specifying existing sublabels, it is intended to refer to all such plurality of similar components.

[0014] Figure 1 This is a schematic diagram illustrating a system according to various embodiments for implementing digital microscopy using deep learning-based segmentation, instance segmentation based on partial annotation, and / or implementing user interfaces configured to facilitate instance segmentation within biological samples.

[0015] Figures 2A-2C This is a system flowchart illustrating various systems according to various embodiments for implementing digital microscopy using deep learning-based segmentation, instance segmentation based on partial annotation, and / or implementing user interfaces configured to facilitate instance segmentation within biological samples.

[0016] Figures 3A-3E This is a schematic diagram illustrating various embodiments of a user interface for facilitating instance segmentation within biological samples, according to various embodiments.

[0017] Figure 4 Annotations depicting objects of interest in original images of first biological samples according to various embodiments are provided, and examples of various images showing predictions of objects of interest by an artificial intelligence (“AI”) system are shown.

[0018] Figure 5 Examples of various images depicting elastic enhancements of original images of first biological samples and elastic enhancements of annotated images of the original images, according to various embodiments.

[0019] Figure 6 Examples of various color-enhanced images depicting original images of first biological samples according to various embodiments are provided.

[0020] Figure 7 Examples of various images illustrating the predictive power of objects of interest based on complete and partial segmentation according to various embodiments are depicted.

[0021] Figures 8A-8D This is a flowchart illustrating methods for achieving digital microscopy using deep learning-based segmentation and / or instance segmentation based on partial annotations, according to various embodiments.

[0022] Figures 9A-9D This is a flowchart illustrating a method, according to various embodiments, for implementing digital microscopy using deep learning-based segmentation, instance segmentation based on partial annotation, and / or implementing a user interface configured to facilitate instance segmentation within biological samples.

[0023] Figure 10 This is a block diagram illustrating an exemplary computer or system hardware architecture according to various embodiments.

[0024] Figure 11 This is a block diagram illustrating a networked system of a computer, computing system, or system hardware architecture that can be used according to various embodiments. Detailed Implementation

[0025] Overview

[0026] Various embodiments provide tools and techniques for implementing digital microscopy (e.g., digital pathology or live-cell imaging), and more specifically, provide methods, systems, and apparatus for implementing digital microscopy using deep learning-based segmentation, instance segmentation based on partial annotation, and / or implementing user interfaces configured to facilitate instance segmentation within biological samples.

[0027] In various embodiments, the computing system may receive a first image and a second image, the first image including the field of view (“FOV”) of the first biological sample, and the second image including markers of instances of objects of interest within the first biological sample. The computing system may use an encoder to encode the second image to generate a third coded image and a fourth coded image, the fourth coded image being different from the third coded image.

[0028] In some embodiments, the first biological sample may include, but is not limited to, one of human tissue samples, animal tissue samples, or plant tissue samples and / or the like, wherein the object of interest may include, but is not limited to, at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures and / or the like. In some cases, the labeling of instances of the object of interest in the second image may include, but is not limited to, at least one of the following: a complete annotation of the first instance of the object of interest identifying the centroid and edges of the first instance of the object of interest, or a partial annotation of the second instance of the object of interest identifying only the centroid of the second instance of the object of interest.

[0029] In some embodiments, encoding the second image to generate a third encoded image may include calculating the centroid of each marker instance of the object of interest in the second image using a computing system; and generating the third encoded image using a computing system, the third encoded image including a highlight of the centroid of each marker instance of the object of interest. In some cases, encoding the second image to generate a fourth encoded image may include calculating the edges or boundaries of each marker instance of the object of interest in the second image using a computing system; and generating the fourth encoded image using a computing system, the fourth encoded image including a highlight of the edges or boundaries of each marker instance of the object of interest.

[0030] According to some embodiments, the computing system can train the AI ​​system to generate or update the AI ​​model to predict instances of objects of interest, at least in part, based on the third and fourth coded images. The computing system can use the regression layers of the AI ​​system or the (updated) AI model to generate a fifth and a sixth image, the sixth image being different from the fifth image.

[0031] The computational system can use a decoder to decode a fifth and a sixth image to generate a seventh image, which includes predicted labels of instances of objects of interest in the first biological sample. In some cases, this can be achieved by applying at least one of one morphological operations that identify foreground and background labels in each of the fifth and sixth images before generating the seventh image, or by directly decoding the fifth and sixth images to generate the seventh image. In some cases, applying at least one of one morphological operations or one of one or more machine learning operations may include applying one or more morphological operations, wherein after decoding the fifth and sixth images by applying one or more morphological operations that identify foreground and background labels in each of the fifth and sixth images, the computational system may apply a watershed algorithm to generate the seventh image. In some cases, the one or more morphological operations may include, but are not limited to, at least one of the following: reconstruction open transformation or region H-minimum transformation and / or the like.

[0032] According to some embodiments, the computing system can compare a seventh image with a second image to generate an instance segmentation evaluation result. In some cases, generating the instance segmentation evaluation result may include evaluating instance segmentation performance using one or more metrics, which may include, but are not limited to, at least one of the following: aggregated Jaccard index (“AJI”), F1 score, DICE index, average DICE index, or joint DICE index and / or the like. In some cases, the instance segmentation evaluation result may include, but is not limited to, instance segmentation evaluation metrics, one or more numerical instance segmentation evaluation scores, or instance segmentation classifications (including, but not limited to, true positives (“TP”), true negatives (“TN”), false positives (“FP”), false negatives (“FN”), oversegmentation or undersegmentation, etc.) and / or the like. The computing system may display the generated instance segmentation evaluation result on a display screen. In some cases, the seventh image may be generated by a label-controlled watershed algorithm using a regression layer (which may include an edge surface regression layer, etc.). In some cases, parameters for morphological operations may be set using the instance segmentation evaluation result (e.g., AJI score, etc.) as the objective function after applying Bayesian optimization.

[0033] In some cases, training an AI system to generate or update an AI model to predict instances of an object of interest, at least in part, based on multiple sets of at least two images generated from a second image, may include at least encoding the second image to generate a third and fourth coded image, training the AI ​​system to generate or update the AI ​​model to predict instances of an object of interest, at least in part, based on the third and fourth coded images, generating a fifth and sixth image, decoding the fifth and sixth images to generate a seventh image, and comparing the seventh image with the second image, etc. Although two images (in this case, the third and fourth coded images) are used to train the AI ​​system, various embodiments are not limited to this, and more than two images (or surfaces) may be used.

[0034] According to some embodiments, the computing system can receive an eighth image, which includes a field of view (FOV) of a second biological sample different from that of the first biological sample; it can generate two or more images based on the eighth image using an AI model generated or updated by a trained AI system, which are different from each other; and it can decode the two or more images using a decoder to generate a ninth image, which includes predicted labels of instances of objects of interest in the second biological sample. Similar to decoding the fifth and sixth images, decoding two or more images to generate the ninth image can include using the computing system and a decoder to decode the two or more images by applying at least one of one or more morphological operations that identify foreground and background markers in each of the two or more images before generating the ninth image, or by directly decoding the two or more images to generate the ninth image. In the case of applying one or more morphological operations, after decoding the two or more images by applying one or more morphological operations that identify foreground and background markers in each of the two or more images, the computing system can apply a watershed algorithm to generate the ninth image. In this way, trained AI systems and / or AI models can be used to predict the labeling of instances of objects of interest in new biological samples—in some cases where there is no ground-based imagery (or previously annotated imagery) corresponding to the new biological sample.

[0035] Alternatively or additionally, the computational system may generate a user interface configured to collect training data using at least one of full or partial annotations of objects of interest within a biological sample image, and may display a first image including the FOV of the first biological sample within a display portion of the user interface. The computational system may receive first user input from a user (e.g., a pathologist, clinician, physician, nurse, or laboratory technician) via the user interface, indicating the presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface. The computational system may generate a boundary around each of the first plurality of objects of interest based at least in part on the location of each of the first plurality of objects within the first image identified by the first user input and at least in part on the analysis of pixels in or around the corresponding location using an algorithm (which may include, but is not limited to, object detection algorithms, pixel recognition algorithms, edge detection algorithms, and / or the like).

[0036] In some cases, the computing system may receive second user input from a user via a user interface, indicating a movement of a point within one of a first plurality of objects of interest (OPIs) from a previous position to a new position within the first image. The system may generate a new boundary around the one of the first plurality of OPIs displayed in the display portion of the user interface, based at least in part on the new position of the point within the one of the first plurality of OPIs in the first image as indicated by the second user input, and at least in part on the analysis of pixels in or around the new position of the point within the one of the first plurality of OPIs using an algorithm. This new boundary replaces the boundary previously generated around the one of the first plurality of OPIs. In some cases, the computing system may receive third user input from a user via a user interface, indicating a partial annotation of one of a second plurality of OPIs contained in the first image displayed in the display portion of the user interface. The system may generate a partial annotation symbol in the first image, based at least in part on the position of the third user input within the first image. This partial annotation symbol identifies the location of the unbounded centroid of one of the second plurality of OPIs. In some cases, the computing system may receive a fourth user input from a user via a user interface, which indicates that one of the third or more objects of interest is unknown or that the instance class of one of the third or more objects of interest should be switched to another instance class (e.g., cancer, benign, etc.). The system may also generate an unknown annotation symbol (i.e., a symbol or annotation representing an unknown instance or object, etc.) in the first image based at least in part on the position of the fourth user input within the first image, identifying the location of the unknown object represented by the fourth user input. Alternatively, the system may switch the instance class of one of the third or more objects of interest to another instance class selected by the fourth user input (e.g., switching between cancer and benign, switching between full annotation and partial annotation, switching between partial annotation and unknown annotation, switching between full annotation and unknown annotation, etc.).

[0037] According to some embodiments, the first user input may include, but is not limited to, a click input or a boundary region input. In some cases, the click input may define the location of the centroid of a first object among a first plurality of objects of interest identified by the click input, while the boundary region input may define a region within a first image that marks the outer boundary of a second object among a first plurality of objects of interest identified by the boundary region input. In some cases, the boundary region input may include, but is not limited to, a rectangular boundary region input, a circular boundary region input, a polygon placement input, or a line placement input and / or the like. In some embodiments, the second user input may include, but is not limited to, click and drag input. In some cases, the third user input may include, but is not limited to, a double-click input, wherein the third user input selects or deselects the boundary around one of the second plurality of objects of interest. In some cases, the fourth user input may include, but is not limited to, a shift plus mouse click input or a key plus mouse click input, wherein the fourth user input may include, but is not limited to, a conversion between full annotations and unknown annotations or a switching between instance classes in an instance class list. However, various embodiments are not limited to these specific inputs, and these inputs may be any suitable input for indicating full annotations, partial annotations, and / or unknown annotations, etc.

[0038] The computing system can generate at least one of a second image or an annotation dataset based on a first image. The second image includes data about the location of each of the first plurality of objects of interest within the first image, which are the boundaries around each of the first plurality of objects of interest identified by the received first user input and the generated data. The annotation dataset includes at least one of pixel location data or coordinate data for each of the first plurality of objects of interest within the first image, which are the boundaries around each of the first plurality of objects of interest identified by the received first user input and the generated data.

[0039] In this way, the system provides a fast and efficient UI that allows users (or annotators) to generate annotations effectively. Specifically, users do not need to open any menus or follow a complex set of operations to interact with the annotation system's UI. A complete annotation (i.e., generating a boundary around the location marked by the click input or boundary area input) can be generated with a single operation (i.e., using click input or boundary area input, etc.). To change the automatically generated boundary, the user simply moves a point within the instance or object using a single operation (i.e., using click-drag input, etc.), causing the system to redraw or regenerate a new boundary around the instance or object. Therefore, users do not need to waste time manually drawing around the edges or boundaries of instances or objects to obtain a complete annotation. Similarly, a complete annotation can be changed to a partial annotation, or the class of an instance or object can be changed, using a single operation (i.e., shift plus mouse click input, key plus mouse click input, or mouse / keyboard combination, etc.). Operations are not bound to specific mouse / keyboard actions; instead, any combination can be used or customized appropriately or as needed.

[0040] These and other aspects, including the use of deep learning-based segmentation (in some cases, through multiple regression layers or other machine learning or deep learning architectures, etc.) for digital microscopy imaging (e.g., digital pathology or live-cell imaging), instance segmentation based on partial annotations, and / or the implementation of user-defined annotations configured to facilitate instance segmentation within biological samples, are described in more detail with reference to the accompanying drawings. While the focus is on biological samples as depicted in the figures below, various embodiments are not limited thereto, and instance segmentation, training systems to generate or update AI models to predict instance segmentation, and / or user-defined annotations configured to facilitate instance segmentation, may be adapted to non-biological samples, including but not limited to chemical samples, humans, animals, plants, insects, tools, vehicles, structures, landmarks, planets, stars, specific living objects, or specific inanimate objects, and / or the like.

[0041] The following detailed description illustrates several exemplary embodiments in more detail to enable those skilled in the art to practice such embodiments. The described embodiments are provided for illustrative purposes and are not intended to limit the scope of the invention.

[0042] In the following description, numerous specific details are set forth for purposes of explanation to provide a thorough understanding of the described embodiments. However, it will be apparent to those skilled in the art that other embodiments of the invention can be practiced without some of these specific details. In other instances, certain structures and devices are shown in block diagram form. Several embodiments are described herein, and while various features are attributed to different embodiments, it should be understood that features described with respect to one embodiment may also be incorporated into other embodiments. However, for the same reason, a single feature or multiple features of any described embodiment should not be considered essential to every embodiment of the invention, as such features may be omitted in other embodiments of the invention.

[0043] Unless otherwise stated, all figures used herein to indicate quantity, size, etc., should be understood to be modified by the term "about" in all cases. In this application, unless otherwise stated, the use of the singular includes the plural, and unless otherwise stated, the use of the terms "and" and "or" means "and / or". Furthermore, the use of the term "comprising" and other forms such as "including" and "included" should be considered non-exclusive. Additionally, unless otherwise specifically stated, terms such as "element" or "component" cover elements and components comprising one unit as well as elements and components comprising more than one unit.

[0044] While the various embodiments described herein embody (in some cases) software products, computer-executed methods, and / or computer systems, they represent tangible and specific improvements to existing technologies, including but not limited to digital pathology techniques, live-cell imaging techniques, digital microscopy imaging techniques, instance segmentation techniques, cell nucleus segmentation techniques, user interface techniques, and / or the like. In other respects, certain embodiments may improve the functionality of the user device or system itself (e.g., digital pathology systems, live-cell imaging systems, digital microscopy systems, instance segmentation systems, cell nucleus segmentation systems, user interface systems, etc.), for example by receiving a first image using a computing system, the first image including the field of view (“FOV”) of a first biological sample; receiving a second image using a computing system, the second image including labels of instances of objects of interest in the first biological sample; encoding the second image using a computing system and an encoder to generate a third and a fourth encoded image, the fourth encoded image being different from the third encoded image; and training an artificial intelligence (“AI”) system to generate or update an AI model to predict instances of objects of interest at least in part based on the third and fourth encoded images; using a regression layer of the AI ​​system to generate a fifth and a sixth image based on the first image and based on training, the sixth image being different from the fifth image; decoding the fifth and sixth images using a computing system and a decoder to generate a seventh image, the seventh image including predicted labels of instances of objects of interest in the first biological sample; and comparing the seventh image with the second image using a computing system to generate an instance segmentation evaluation result; and / or the like.

[0045] Alternatively or additionally, certain embodiments may improve the functionality of the user device or system itself (e.g., digital pathology systems, live-cell imaging systems, digital microscopy imaging systems, instance segmentation systems, cell nucleus segmentation systems, user interface systems, etc.), for example, by generating a user interface using a computing system configured to collect training data using at least one of complete or partial annotations of objects of interest within a biological sample image; displaying a first image including the field of view (“FOV”) of the first biological sample within a display portion of the user interface using a computing system; receiving first user input from a user using a computing system and via the user interface, the first user input indicating the presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface; and using a computing system to identify, at least in part, objects of interest based on the first user input. The location of each of the first plurality of objects within a first image and, at least in part, based on the analysis of pixels in or around the corresponding location using an algorithm, generates a boundary around each of the first plurality of objects of interest; a computing system generates at least one of a second image or an annotation dataset based on the first image, the second image including data about the location of each of the first plurality of objects of interest within the first image, based on the boundaries around each of the first plurality of objects of interest identified by the received first user input, and the annotation dataset including at least one of pixel location data or coordinate data of each of the first plurality of objects within the first image, based on the boundaries around each of the first plurality of objects of interest identified by the received first user input.

[0046] In particular, with regard to any abstract concepts present in the various embodiments, these concepts can be implemented as described herein by devices, software, systems, and methods involving specific novel functions (e.g., steps or operations), such as providing a user interface configured to receive user input indicating at least one of the following: a complete annotation of a first instance of an object of interest identifying the centroid and edges of a first instance of an object of interest; a partial annotation of a second instance of an object of interest identifying only the centroid of a second instance of an object of interest; or an annotation of an unknown instance of an object of interest; and / or training an AI system (including, but not limited to, at least one of, machine learning systems, deep learning systems, neural networks, convolutional neural networks (“CNN”) or fully convolutional networks (“FCN”) and / or the like) to predict instances of objects of interest in biological sample images based on complete and / or partial annotations; and / or the like, to name just a few examples, which go beyond mere conventional computer processing operations. These capabilities can produce tangible results outside of computer systems, and by way of example only, include optimized presentation and tracking of user input indicating annotations of complete, partial, and / or unknown objects, and / or optimized training of AI systems to generate or update AI models to predict instances of objects of interest in biological sample images, and / or similar objects, based on complete and / or partial annotations, at least some of which can be observed or measured by users (including, but not limited to, pathologists, clinicians, doctors, nurses, or laboratory technicians).

[0047] In one aspect, a method may include receiving a first image using a computing system, the first image including a field of view (“FOV”) of a first biological sample; receiving a second image using the computing system, the second image including labels of instances of objects of interest in the first biological sample; and training an artificial intelligence (“AI”) system to generate or update an AI model to predict instances of objects of interest based at least in part on a plurality of sets of at least two images generated from the second image, each of at least two of the plurality of sets of at least two images being distinct from one another.

[0048] In some embodiments, the computing system may include one of the following: a computing system located in the work environment, a remote computing system located outside the work environment and accessible via a network, a web server, a web browser, or a cloud computing system and / or the like. The work environment may include at least one of the following: a laboratory, clinic, medical facility, research facility, healthcare facility, or room and / or the like. In some cases, the AI ​​system may include at least one of the following: a machine learning system, a deep learning system, a neural network, a convolutional neural network (“CNN”), or a fully convolutional network (“FCN”), and / or the like. In some cases, the first biological sample may include one of the following: a human tissue sample, an animal tissue sample, or a plant tissue sample and / or the like. The object of interest may include at least one of the following: normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures and / or the like.

[0049] According to some embodiments, training an AI system to generate or update an AI model to predict instances of an object of interest, at least in part, based on multiple sets of at least two images generated from a second image, may include encoding the second image using a computing system and an encoder to generate a third and fourth encoded image, the fourth encoded image being different from the third encoded image; training the AI ​​system to generate or update an AI model to predict instances of an object of interest, at least in part, based on the third and fourth encoded images; generating a fifth and sixth image, different from the fifth image, based on the first image using the AI ​​model generated or updated by the AI ​​system; and decoding the fifth and sixth images using a computing system and a decoder to generate a seventh image, the seventh image including a predicted label of an instance of an object of interest in the first biological sample. In some cases, training the AI ​​system to generate or update an AI model to predict instances of an object of interest, at least in part, based on multiple sets of at least two images generated from the second image, may further include comparing the seventh image with the second image using a computing system to generate an instance segmentation evaluation result.

[0050] In some embodiments, encoding the second image to generate a third encoded image may include: calculating the centroid of each marker instance of an object of interest in the second image using a computing system; and generating the third encoded image using a computing system, the third encoded image including a highlight of the centroid of each marker instance of the object of interest. In some embodiments, encoding the second image to generate a fourth encoded image may include: calculating the edges or boundaries of each marker instance of an object of interest in the second image using a computing system; and generating the fourth encoded image using a computing system, the fourth encoded image including a highlight of the edges or boundaries of each marker instance of an object of interest.

[0051] As an example only, in some cases, encoding a second image to generate a third encoded image may further include using a computing system to calculate a first distance metric between each pixel in the third encoded image and each centroid of each marker instance of the object of interest; and using a computing system to calculate a first function to generate a first proximity map, the first function being a function of the first distance metric, the third encoded image including the first proximity map. Similarly, encoding a second image to generate a fourth encoded image may further include using a computing system to calculate a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of the edge or boundary of each marker instance of the object of interest; and using a computing system to calculate a second function to generate a second proximity map, the second function being a function of the second distance metric, the fourth encoded image including the second proximity map.

[0052] According to some embodiments, the method may further include assigning a first weighted pixel value to each pixel in a third coded image using a computing system based at least in part on at least one of a first distance metric, a first function, or a first proximity map for each pixel; and assigning a second weighted pixel value to each pixel in a fourth coded image using a computing system based at least in part on at least one of a second distance metric, a second function, or a second proximity map for each pixel.

[0053] In some embodiments, the method may further include: determining a first pixel loss value between each pixel in the third coded image and a corresponding pixel in the fifth image using a computing system; determining a second pixel loss value between each pixel in the fourth coded image and a corresponding pixel in the sixth image using a computing system; calculating a loss value using a loss function based on a first weighted pixel value of each pixel in the third coded image multiplied by the product of the first pixel loss value between each pixel in the third coded image and the corresponding pixel in the fifth image, and a second weighted pixel value of each pixel in the fourth coded image multiplied by the product of the second pixel loss value between each pixel in the fourth coded image and the corresponding pixel in the sixth image using a computing system; and updating the AI ​​model using an AI system by updating one or more parameters of the AI ​​model based on the calculated loss value. In some cases, the loss function may include one of the following: mean squared error loss function, mean logarithmic error loss function, mean absolute error loss function, Huber loss function, or weighted sum of squared differences loss function and / or the like. In this case, generating the fifth and sixth images may include generating the fifth and sixth images based on the first image using the updated AI model.

[0054] According to some embodiments, the labeling of instances of objects of interest in the second image includes at least one of the following: a complete annotation of the first instance of the object of interest that identifies the centroid and edges of the first instance of the object of interest; a partial annotation of the second instance of the object of interest that only identifies the centroid of the second instance of the object of interest; or an unknown annotation of the third instance of the object of interest that neither identifies the centroid nor the edges. In some cases, the method may further include, before calculating the loss value, using a computing system to mask the second instance of the object of interest using partial annotations in corresponding pixels of the fourth and sixth images, without using partial annotations in the third or fifth image; and before calculating the loss value, using a computing system to mask the third instance of the object of interest using unknown annotations in corresponding pixels of the third and fifth images and corresponding pixels of the fourth and sixth images.

[0055] In some embodiments, decoding the fifth and sixth images to generate the seventh image may include using a computing system and a decoder to decode the fifth and sixth images to generate the seventh image by applying at least one of one morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or by directly decoding the fifth and sixth images to generate the seventh image. In some cases, applying at least one of one morphological operations or one of one or more machine learning operations may include applying one or more morphological operations, and the method may further include applying a watershed algorithm to generate the seventh image after decoding the fifth and sixth images by applying one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images. In some cases, one or more morphological operations may include at least one of a reconstruction open transform or a region H-minimum transform and / or the like.

[0056] According to some embodiments, the method may further include receiving an eighth image using a computing system, the eighth image including a FOV of a second biological sample that is different from that of the first biological sample; generating two or more images based on the eighth image using an AI model generated or updated by a trained AI system, the two or more images being different from each other; and decoding the two or more images using a computing system and a decoder to generate a ninth image, the ninth image including predicted labels of instances of objects of interest in the second biological sample.

[0057] In some cases, the first and second images may be data-enhanced before being received by the computing system, wherein the data enhancement of the first and second images may include at least one of elastic enhancement or color enhancement and / or the like configured to facilitate instance segmentation. In some cases, at least two images include at least a centroid layer image, a boundary layer image, and a semantic segmentation layer image, wherein the centroid layer image highlights the centroid of each labeled instance of the object of interest in the second image, the boundary layer image highlights the edges or boundaries of each labeled instance of the object of interest in the second image, and the semantic segmentation layer image includes semantic segmentation data for each labeled instance of the object of interest in the second image.

[0058] In another aspect, a system may include a computing system that may include at least one first processor and a first non-transitory computer-readable medium communicatively coupled to the at least one first processor. The first non-transitory computer-readable medium may store computer software thereon including a first set of instructions, which, when executed by the at least one first processor, cause the computing system to: receive a first image including a field of view (“FOV”) of a first biological sample; receive a second image including labels of instances of objects of interest in the first biological sample; and train an artificial intelligence (“AI”) system to generate or update an AI model to predict instances of objects of interest, at least in part, based on at least two images generated from a plurality of sets of at least two images, each of the plurality of sets of at least two images being distinct from one another.

[0059] In another aspect, a method may include receiving a first image using a computing system, the first image including a field of view (“FOV”) of a first biological sample; generating two or more images based on the first image using an AI model generated or updated by a trained artificial intelligence (“AI”) system, each of the two or more images being distinct from each other, wherein training the AI ​​system includes training the AI ​​system to generate or update the AI ​​model to predict instances of objects of interest based at least in part on multiple sets of at least two images generated according to user-annotated images, each of at least two images in the multiple sets of at least two images being distinct from each other; and decoding the two or more images using a computing system and a decoder to generate a second image including predicted labels of instances of objects of interest in the first biological sample.

[0060] In one aspect, a method may include generating a user interface using a computing system, the user interface being configured to collect training data using at least one of full or partial annotations of objects of interest within an image of a biological sample; displaying, using a computing system, a first image including a field of view (“FOV”) of a first biological sample within a display portion of the user interface; receiving, using a computing system and via the user interface, first user input from a user indicating the presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface; and using a computing system to identify, at least in part, the location of each of the first plurality of objects within the first image based on the first user input and at least in part based on the use of a computing system... The method analyzes pixels within or around a corresponding location to generate a boundary around each of a first plurality of objects of interest; and uses a computing system to generate at least one of a second image or an annotation dataset based on a first image, the second image including data about the location of each of the first plurality of objects of interest within the first image, the boundary around each of the first plurality of objects of interest identified by the received first user input, and the annotation dataset including at least one of pixel location data or coordinate data of each of the first plurality of objects within the first image, the boundary of each of the first plurality of objects of interest identified by the received first user input, and the boundary of each of the first plurality of objects of interest identified by the received first user input.

[0061] In some embodiments, the computing system may include one of the following: a computing system located in the work environment, a remote computing system located outside the work environment and accessible via a network, a web server, a web browser, or a cloud computing system and / or the like. In some cases, the work environment may include at least one of a laboratory, clinic, medical facility, research facility, healthcare facility, or room and / or the like. In some cases, the first biological sample may include one of a human tissue sample, an animal tissue sample, or a plant tissue sample and / or the like. In some cases, the object of interest may include at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures.

[0062] In some embodiments, the method may further include receiving second user input from a user using a computing system and via a user interface, the second user input instructing a point within one of a first plurality of objects of interest to move from a previous position within a first image to a new position; and generating, using a computing system, at least partially based on the new position of the point within the one of the first plurality of objects of interest in the first image represented by the second user input, and at least partially based on analyzing pixels in or around the new position of the point within the one of the first plurality of objects of interest using an algorithm, a new boundary around the one of the first plurality of objects of interest contained in the first image displayed in a display portion of the user interface, the new boundary replacing a boundary previously generated around the one of the first plurality of objects of interest.

[0063] According to some embodiments, the method may further include receiving third user input from a user using a computing system and via a user interface, the third user input indicating a partial annotation of one of a second plurality of objects of interest contained within a first image displayed in a display portion of the user interface; and generating a partial annotation symbol in the first image using a computing system based at least in part on the location of the third user input within the first image, the partial annotation symbol identifying the location of the unbounded centroid of one of the second plurality of objects of interest.

[0064] In some embodiments, the method may further include receiving a fourth user input from a user using a computing system and via a user interface, the fourth user input indicating that one of the third plurality of objects of interest is unknown or that an instance class of one of the third plurality of objects of interest should be switched to another instance class; and generating an unknown annotation symbol in the first image, at least in part based on the position of the fourth user input within the first image, to identify the location of the unknown object represented by the fourth user input, or switching an instance class of a selected one of the third plurality of objects of interest to another instance class selected by the fourth user input.

[0065] By way of example only, in some cases, the first user input may include one of a click input or a boundary region input, wherein the click input defines the location of the centroid of each of at least one of a first plurality of objects of interest identified by the click input, wherein the boundary region input defines a region within a first image that marks the outer boundary of a second object among the first plurality of objects of interest identified by the boundary region input, wherein the boundary region input may include one of a rectangular boundary region input, a circular boundary region input, a polygon placement input, or a line placement input and / or the like. The second user input may include click and drag input. The third user input may include a double-click input, wherein the third user input may include selecting or deselecting a boundary surrounding one of the second plurality of objects of interest. The fourth user input may include one of a shift plus mouse click input or a key plus mouse click input and / or the like. The fourth user input may include one of switching between full annotations and unknown annotations or switching between instance classes from a list of instance classes.

[0066] According to some embodiments, the method may further include training an artificial intelligence (“AI”) system to generate or update an AI model to predict instances of objects of interest in a first biological sample, at least in part, based on at least two images generated from at least one of a plurality of sets of at least two images, each of which is distinct from the others. In some cases, training the AI ​​system to generate or update the AI ​​model to predict instances of objects of interest, at least in part, based on at least two images from a plurality of sets of at least two images may include: encoding at least one of the second image or annotation datasets using a computing system and an encoder to generate a third and a fourth encoded image, the fourth encoded image being distinct from the third encoded image; training the AI ​​system to generate or update the AI ​​model to predict instances of objects of interest, at least in part, based on the third and fourth encoded images; generating a fifth and a sixth image based on the first image using the AI ​​model generated or updated by the AI ​​system, the sixth image being distinct from the fifth image; and decoding the fifth and sixth images using a computing system and a decoder to generate a seventh image, the seventh image including a predicted label of an instance of an object of interest in the first biological sample. In some cases, the AI ​​system may include at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network (“CNN”) or a fully convolutional network (“FCN”) and / or the like. In some cases, training the AI ​​system to generate or update the AI ​​model to predict instances of an object of interest based at least in part on multiple sets of at least two images may also include comparing a seventh image with the second image using a computational system to generate an instance segmentation evaluation result.

[0067] In some embodiments, the third coded image may include the centroid of each of a first plurality of objects of interest based on a first user input, wherein the fourth coded image may include the boundary of each of the first plurality of objects of interest generated. According to some embodiments, encoding a second image to generate a third coded image may include: calculating a first distance metric between each pixel in the third coded image and each centroid of each of the first plurality of objects of interest using a computing system; and calculating a first function using a computing system to generate a first proximity map, the first function being a function of the first distance metric, the third coded image including the first proximity map. Similarly, encoding a second image to generate a fourth coded image may include: calculating a second distance metric between each pixel in the fourth coded image and the nearest edge pixel of an edge or boundary of each of the first plurality of objects of interest using a computing system; and calculating a second function using a computing system to generate a second proximity map, the second function being a function of the second distance metric, the fourth coded image including the second proximity map.

[0068] According to some embodiments, the method may further include assigning a first weighted pixel value to each pixel in a third coded image using a computing system based at least in part on at least one of a first distance metric, a first function, or a first proximity map calculated for each pixel; and assigning a second weighted pixel value to each pixel in a fourth coded image using a computing system based at least in part on at least one of a second distance metric, a second function, or a second proximity map calculated for each pixel.

[0069] In some embodiments, the method may further include: determining a first pixel loss value between each pixel in the third coded image and a corresponding pixel in the fifth image using a computing system; determining a second pixel loss value between each pixel in the fourth coded image and a corresponding pixel in the sixth image using a computing system; calculating a loss value using a loss function based on a first weighted pixel value of each pixel in the third coded image multiplied by the product of the first pixel loss value between each pixel in the third coded image and the corresponding pixel in the fifth image, and a second weighted pixel value of each pixel in the fourth coded image multiplied by the product of the second pixel loss value between each pixel in the fourth coded image and the corresponding pixel in the sixth image using a computing system; and updating the AI ​​model using an AI system by updating one or more parameters of the AI ​​model based on the calculated loss value. In some cases, the loss function may include one of the following: mean squared error loss function, mean logarithmic error loss function, mean absolute error loss function, Huber loss function, or weighted sum of squared differences loss function and / or the like. In this case, generating the fifth and sixth images may include generating the fifth and sixth images based on the first image using the updated AI model.

[0070] According to some embodiments, decoding a fifth and sixth image to generate a seventh image may include using a computing system and a decoder to decode the fifth and sixth images to generate the seventh image by applying at least one of one morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or by directly decoding the fifth and sixth images to generate the seventh image. In some cases, applying at least one of one morphological operations or one of one or more machine learning operations may include applying one or more morphological operations, wherein the method may further include applying a watershed algorithm to generate the seventh image after decoding the fifth and sixth images by applying one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images.

[0071] In another aspect, a system may include a computing system that may include at least one first processor and a first non-transitory computer-readable medium communicatively coupled to the at least one first processor. The first non-transitory computer-readable medium may store computer software thereon comprising a first set of instructions that, when executed by the at least one first processor, cause the computing system to: generate a user interface configured to collect training data using at least one of complete or partial annotations of objects of interest within an image of a biological sample; display a first image including a field of view (“FOV”) of the first biological sample within a display portion of the user interface; receive first user input from a user via the user interface, the first user input indicating the presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface; and, at least in part based on the location of each of the first plurality of objects identified by the first user input and at least... The method involves generating a boundary around each of a first plurality of objects of interest (OPIs) based in part on the analysis of pixels in or around a corresponding location using an algorithm; generating at least one of a second image or an annotation dataset based on a first image, wherein the second image includes data about the location of each of the first plurality of OPIs within the first image, based on the boundary around each of the first plurality of OPIs identified by the received first user input and the generated data, and the annotation dataset includes at least one of pixel location data or coordinate data about the boundary around each of the first plurality of OPIs identified by the received first user input and the generated data, based on the first user input and the generated data, about the boundary around each of the first plurality of OPIs within the first image.

[0072] According to some embodiments, the computing system may include one of the following: a computing system located in a working environment, a remote computing system located outside the working environment and accessible via a network, a web server, a web browser, or a cloud computing system and / or the like. In some cases, the working environment may include at least one of the following: a laboratory, clinic, medical facility, research facility, healthcare facility, or room. In some cases, the first biological sample may include one of the following: a human tissue sample, an animal tissue sample, or a plant tissue sample and / or the like. In some cases, the object of interest may include at least one of the following: normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures and / or the like. In some cases, the first user input may include one of the following: a click input or a boundary region input, wherein the click input defines the position of the centroid of each of at least one of a first plurality of objects of interest identified by the click input, wherein the boundary region input defines a region within a first image that marks the outer boundary of the perimeter of at least one of a first plurality of objects of interest identified by the boundary region input, wherein the boundary region input includes one of the following: a rectangular boundary region input, a circular boundary region input, a polygonal placement input, or a line placement input.

[0073] Various modifications and additions can be made to the discussed embodiments without departing from the scope of the invention. For example, while the above embodiments refer to specific features, the scope of the invention also includes embodiments with different combinations of features and embodiments that do not include all of the above features.

[0074] Specific exemplary embodiments

[0075] I. Segmentation via regression layers based on deep learning:

[0076] In images containing many instances of similar or related objects / features, or objects / features of a particular type (which may be in contact with or partially overlap with other objects / features), accurate identification of objects / features within an image can be challenging. Instance segmentation reflects the challenge of identifying all instances of an object and its corresponding features (e.g., shape / contour). An example of this task is cell nucleus segmentation in microscope images, a primary task in many digital pathology procedures such as cell nucleus counting, cell nucleus classification, and various cancer grading tasks. In such applications, manual segmentation can be difficult or impractical due to the large number of cell nuclei within the entire slide image (“WSI”), necessitating automated cell nucleus segmentation. However, robust automated cell nucleus segmentation remains an extremely challenging task due to the diversity of cell nucleus shape, color, orientation, density, and other features, as well as other factors such as image quality, resolution, tissue and staining type differences, and the large size of the WSI.

[0077] In one embodiment, for robust encoding of cell nucleus morphology—rather than encoding each cell as a different class of object (as done in semantic segmentation)—the morphology of the cell nucleus (i.e., the object to be segmented) can be encoded using two surfaces: the distance to the center of the cell nucleus and the cell nucleus outline. This encoding is robust because the morphology is encoded using many pixels and is not affected by errors in a single pixel. Using two surfaces to encode morphology (instead of just a distance) is novel and potentially more robust than existing methods.

[0078] In one embodiment, a weighted scheme is provided where higher weights are assigned to “important” pixels (e.g., pixels belonging to a shared boundary between two cell nuclei), thus helping the network focus on “risky” pixels (where errors could lead to oversegmentation or undersegmentation).

[0079] In one embodiment, a method can be provided that minimizes the effort required to generate a training dataset for an instance segmentation task, built on two main components: (A) a novel scheme for instance segmentation ground truth data comprising a hybrid of two types of cell annotations—full annotations (where cell nuclei fully specify the center and complete contour) and partial annotations (where cell nuclei specify only the center); and (B) a novel approach to solving the instance segmentation problem comprising (i) encoding ground truth data using two surfaces that can be robustly modeled by a fully convolutional biregressive neural network (trained with a hybrid of full and partial annotations) and (ii) decoding the network-predicted surfaces (for test images) into instance segments (based on a label-controlled watershed algorithm).

[0080] Example 1:

[0081] 1. Preprocessing:

[0082] 1.1 Dataset Preparation: The dataset used to train the model was provided as part of the MoNuSeg H&E staining multi-organ nucleus segmentation exercise in the Digital Pathology Challenge held at MICCAI 2018. The training dataset consisted of 30 1000x1000 image patches cropped from WSI (captured at 40x magnification) and downloaded from the TCGA archive. To ensure dataset diversity, each image corresponds to one patient, with images taken from 18 hospitals and covering 7 organs. Nucleus segmentation (ground-based) was provided for each image patch. For training purposes, 11 images were selected for validation since no validation set was provided (these images were not used during the training phase).

[0083] 1.2 Dataset Augmentation: Due to the small number and diversity of images used for training, we applied extensive data augmentation to both the RGB and labeled image sets, including standard augmentation procedures such as rotation, mirroring, and minor resizing, as well as techniques such as... Figure 5 The elastic image transformation is shown. Elastic enhancement is applied by sampling the RGB image and the annotation label image using a random displacement field. The distortion level is controlled by convolving the displacement field with a Gaussian kernel with a predefined standard deviation and scaling factor. Finally, the RGB image and label image are sampled separately using a displacement field with bilinear interpolation and nearest neighbor. Furthermore, RGB color transformation is performed directly in the chromatic channel optical density space via color deconvolution. The chromatic optical density is randomly scaled and biased, and then projected back into the RGB space, as shown. Figure 6 As shown.

[0084] 2. The proposed model

[0085] Our method consists of three main steps (detailed below): First, the ground reality is encoded as a set of two surfaces (see Section 2.1 below); second, a fully convolutional neural network (“FCN”) is trained based on the UNet or U-Net architecture proposed by Ronneberger et al. in 2015 to predict these surfaces; and finally, in post-processing, the predicted surfaces are used to perform constrained watershed segmentation and predicted cell nucleus segmentation (see Section 3 below).

[0086] 2.1 Ground Plot Encoding: For each training image, we segment the pixel-associated ground plot into non-overlapping objects (e.g., cell nuclei). We further compute the centroid of each nucleus (see...). Figure 4 and Figure 7 We now compute two distance metrics for each pixel: (a) the distance to the cell nucleus centroid (in pixels); and (b) the distance to the nearest edge pixel of the cell nucleus. Following the approach of Philipp Kainz et al. (Miccai 2015), we transform these distances from the cell nucleus center and edge. Furthermore, we assign weights to each pixel. Intuitively, we want to assign higher weights to “critical” pixels (where misprediction would lead to oversegmentation). Specifically, we follow a weighting scheme similar to U-Net and assign higher weights to pixels closer to two different cell nuclei.

[0087] 2.2 Network Architecture: We replace the final U-Net layer (classification layer for semantic classification) with a regression layer that outputs two surface maps. As the loss function, we use a weighted sum of the squared differences between the encoded ground reality and the model output, as shown in Figure 2.

[0088] 3. Post-processing:

[0089] 3.1 Decoding the Network Predicted Surface into Instance Segmentation: To convert the output network surface into a nucleus segmentation label map, we first apply several morphological operations, such as reconstruction open and region H-minimum transform, to find foreground and background labels from the centroid surface. Finally, an edge surface regression layer is used to generate the predicted label map via a label-controlled watershed algorithm. The parameters of the morphological operations are set after applying Bayesian optimization and aggregating Jaccard index (“AJI”) scores as the objective function.

[0090] 3.2 Performance Evaluation: Instance segmentation performance was evaluated using two metrics that provide an accuracy measure for instance segmentation—AJI and Joint-Dice. Both metrics take into account various errors in instance segmentation, such as incorrectly detected pixels (false negatives or false positives), oversegmented instances (where one instance is predicted and split into two or more instances), and undersegmented instances (where two or more instances are merged and predicted as a single instance).

[0091] B. Example 2:

[0092] Biregressive Deep Neural Networks for Nucleus Segmentation: Our approach consists of three main steps (detailed below): First, the ground reality is encoded as a set of two surfaces (see Section 2.1 below); second, a fully convolutional neural network (“FCN”) is trained based on the UNet or U-Net architecture proposed by Ronneberger et al. in 2015 to predict these surfaces; finally, in post-processing, the predicted surfaces are used to perform constrained watershed segmentation and predict nucleus segmentation (see Section 3 below).

[0093] Ground-based encoding: For each training image, we segment the pixel-associated ground-based reality into non-overlapping objects (e.g., cell nuclei). We further compute the centroid of each nucleus (see...). Figure 4 and Figure 7 We now compute two distance metrics for each pixel: (a) the distance to the cell nucleus centroid (in pixels); and (b) the distance to the nearest edge pixel of the cell nucleus. Following the method of Philipp Kainz et al. (Miccai 2015), we transform these distances from the cell nucleus center and edge.

[0094] Network architecture: We replace the final U-Net layer (classification layer for semantic classification) with two regression layers that attempt to predict the surface map. As the loss function, we use a weighted sum of the squared differences between the encoded ground reality and the model output.

[0095] Post-processing: To convert the output network surface into a nucleus segmentation label map, we first apply several morphological operations, such as reconstructed open and region H-minimum transforms, to find foreground and background labels from the centroid surface. Finally, a predicted label map is generated using a label-controlled watershed algorithm with an edge surface regression layer. The parameters of the morphological operations are set after applying Bayesian optimization and aggregating Jaccard index (“AJI”) scores as the objective function.

[0096] Adjusting the partial annotations used to train the deep neural network: We leverage the inherent separation between the channels for detecting cell nuclei and tracking their contours, and during training, we mask out partially annotated instances in the second channel. Specifically, we use boundary masking around the partially annotated cell nuclei, so for those particular nuclei, the network does not score based on the nucleus boundary prediction, but only on the detection accuracy, such as... Figure 3D As shown.

[0097] Results: To train the model, we used the MoNuSeg H&E-stained multi-organ nucleus segmentation dataset (the nucleus segmentation challenge held at MICCAI 2018). The competition dataset consists of 30 1000x1000 images, each cropped from WSI (captured at 40x magnification). To ensure diversity, the dataset covers 7 organs collected from different patients in 18 hospitals. Nucleus annotations (ground truth) are provided for each image. We used the above method as part of the MoNuSeg competition, achieving an AJI score of 0.62 on the competition test set. For the evaluation described below, since the competition test set has not yet been released, we selected 14 images from the 30 images for use as the test set (these images were not used during training). To simulate partially annotated data, we conducted a series of experiments at different ratios: 10%–100% (3 cross-validations), where we randomly replaced fully annotated nuclei with only the nucleus center.

[0098] Preliminary results indicate that accessing only 50% of the fully segmented kernel (while using only an approximate location at the center of the cell nucleus for the other 50%) reduces algorithm performance by only 2%–4%.

[0099] II. User Interface and Cell Nucleus Segmentation Notes:

[0100] This disclosure teaches a method that significantly reduces the effort required to generate training datasets for instance segmentation tasks. It includes a novel deep learning training framework specifically developed to address the challenge of efficiently collecting segmentation training data and using it to train deep learning-based cell kernel segmentation models that can also benefit from partially annotated training data.

[0101] This disclosure teaches a method that greatly minimizes the effort required to generate a training dataset for cell nucleus segmentation by using a novel encoding method for training a convolutional deep neural network (“CNN”) that allows a combination of two types of cell nucleus annotations (i.e., fully segmented cell nuclei and only detected cell nuclei) and a novel user interface that facilitates the collection of these two types of annotations.

[0102] This disclosure teaches a novel scheme for instance segmentation of ground fact data, comprising a combination of two types of nucleus annotation: full annotation (where nuclei fully specify the center and complete outline); and partial annotation (where nuclei specify only the center).

[0103] This disclosure teaches a novel approach to solving the instance segmentation problem: (i) using two surfaces to encode ground reality data, which can be robustly modeled by a fully convolutional biregressive neural network (trained using a mixture of full and partial annotations) and (ii) decoding the network-predicted surfaces (for test images) into instance segments (based on a label-controlled watershed algorithm).

[0104] This disclosure comprises at least two parts: a user interface for facilitating the collection of complete and partial annotations, and a method for combining the two types of annotations during the training process of the novel cell kernel segmentation algorithm. This disclosure further teaches the following:

[0105] (a) Ease or speed of generating training annotations: In one embodiment, a novel user interface was developed that allows domain experts to quickly (by clicking the mouse) generate a complete cell nucleus outline or simply label the approximate center of the cell nucleus.

[0106] (b) Utilizing partial annotations: Some cell nuclei are difficult to segment precisely, so allowing users to label only the cell nuclei for detection increases the number of cell nuclei available for training. In one embodiment, the training method labels contours and / or cell nucleus localizations to provide a cell nucleus segmentation model.

[0107] (c) Simpler Training Process: This disclosure teaches a method for rapidly labeling and / or segmenting all cells in the field of view, allowing for a simple training process. Therefore, no data preprocessing or cropping is required before training.

[0108] (d) Fully utilize the field of view (“FOV”): All data within a specific FOV is annotated to some extent (either fully segmented or partially segmented), and the training method teaches the partial annotations. This method does not require preprocessing or cropping the data before it becomes useful.

[0109] (e) Improved performance: Incomplete annotations are useful for the training process and increase the amount of training data available. This disclosure teaches that completely segmenting only 50% of the cell nucleus (while only recording the approximate location of the center of the remaining nuclei) reduces the algorithm performance by only 2%–4%.

[0110] A. Data collection:

[0111] The first step in any machine learning task is collecting training data, which is often a tedious and time-consuming process, and in some cases can be very expensive (labeling often requires experts). To address this, we teach a novel semi-supervised algorithm and user interface called “click-growing,” which enables fast and efficient cell nucleus segmentation.

[0112] 1. Using our in-house whole slide image analysis software, users will be instructed to click on the relevant object (e.g., cell nucleus).

[0113] 2. This software employs a dedicated semi-supervised object segmentation algorithm to attempt to "expand" the click and find the precise boundaries of objects. More specifically, we apply a voting mechanism among multiple automatically generated segments to try to identify stable outlines of cell nuclei. The resulting proposed object segments are displayed on the screen immediately (in real time).

[0114] 3. If the user agrees with the proposed segmentation, he or she can continue clicking on another cell nucleus. In this case, the proposed, automatically generated segmentation is recorded and will be used as the cell nucleus contour during the training process. On the other hand, if the segmentation is inaccurate, the user can try to generate a new segmentation by moving (e.g., dragging, etc.) the clicked location, which will generate a new proposed segmentation. Alternatively, the user can mark the annotation as partial (e.g., by double-clicking it, etc.). In this case, only the clicked location, not the cell nucleus contour, will be recorded, meaning the user confirms the object's existence rather than its contour.

[0115] B. Encode partially segmented data to train a deep neural network:

[0116] Given a set of fully segmented cell nuclei, we teach a novel deep learning segmentation method that encodes the training data into a pair of regression channels. The first is the distance to the center of the cell nucleus (i.e., the detection channel), and the second is the distance to the cell nucleus contour (i.e., the boundary detection channel). This disclosure teaches how the scheme can be extended to support fully annotated data, partially annotated data, and unknown objects. This disclosure utilizes the separation between the channels for detecting cell nuclei and tracking their contours, and masks partially annotated instances in the second channel during training. Specifically, this disclosure teaches using boundary masking around the partially annotated cell nuclei, so that for those particular cell nuclei, the network does not score based on the cell nucleus boundary prediction, but only on the detection accuracy, such as... Figure 3D As shown in the diagram. Furthermore, we masked pixels of unknown objects in both channels (i.e., the nucleus distance transformation from the center and boundary channels).

[0117] C. A biregressive deep neural network for cell nucleus segmentation:

[0118] Our method consists of three main steps (detailed below): First, the ground reality is encoded as a set of two surfaces; second, a fully convolutional neural network (“FCN”) is trained based on the UNet or U-Net architecture proposed by Ronneberger et al. in 2015 to predict these surfaces; and finally, in post-processing, the predicted surfaces are used for constrained watershed segmentation and predicted cell nucleus segmentation.

[0119] Ground-based encoding: For each training image, we segment the pixel-associated ground-based reality into non-overlapping objects (e.g., cell nuclei). We further compute the centroid of each nucleus (see...). Figure 4 and Figure 7 We now calculate two distance metrics for each pixel: (a) the distance to the cell nucleus centroid (in pixels); and (b) the distance to the nearest edge pixel of the cell nucleus. Following the method of Philipp Kainz et al. (Miccai 2015), we transform these distances from the cell nucleus center and edge.

[0120] Network architecture: We replace the final U-Net layer (classification layer for semantic classification) with two regression layers that attempt to predict the surface map. As the loss function, we use a weighted sum of the squared differences between the encoded ground reality and the model output.

[0121] Post-processing: To convert the output network surface into a nucleus segmentation label map, we first apply several morphological operations, such as reconstructed open and region H-minimum transforms, to find foreground and background labels from the centroid surface. Finally, a predicted label map is generated using a label-controlled watershed algorithm with an edge surface regression layer. The parameters of the morphological operations are set after applying Bayesian optimization and aggregating Jaccard index (“AJI”) scores as the objective function.

[0122] D. Adjust some annotations to train a deep neural network:

[0123] We leverage the inherent separation between the channels for detecting cell nuclei and tracking their contours, and mask out partially annotated instances in the second channel during training. Specifically, we use boundary masking around the partially annotated nuclei, so for those specific nuclei, the network does not score on the nucleus boundary prediction, but only on the detection accuracy, such as... Figure 3D As shown.

[0124] Results: To train the model, we used the MoNuSeg H&E-stained multi-organ nucleus segmentation dataset (the nucleus segmentation challenge held at MICCAI 2018). The competition dataset consists of 30 1000x1000 images, each cropped from WSI (captured at 40x magnification). To ensure diversity, the dataset covers 7 organs collected from different patients in 18 hospitals. Nucleus annotations (ground truth) are provided for each image. We used the above method as part of the MoNuSeg competition, achieving an AJI score of 0.62 on the competition test set. For the evaluation described below, since the competition test set has not yet been released, we selected 14 images from the 30 images for the test set (these images were not used during training). To simulate partially annotated data, we conducted a series of experiments at different ratios: 10%–100% (3 cross-validations), where we randomly replaced fully annotated nuclei only with the nucleus center.

[0125] Preliminary results indicate that accessing only 50% of the fully segmented cell nuclei (while using only an approximate location at the center of the cell nucleus for the other 50%) reduces algorithm performance by only 2%–4%.

[0126] III. The example is shown in the figure:

[0127] We now turn to the embodiment shown in the figure. Figures 1-11 Some features of methods, systems, and apparatuses for implementing digital microscopy (e.g., digital pathology or live-cell imaging) are shown, and more specifically, methods, systems, and apparatuses for implementing digital microscopy using deep learning-based segmentation (in some cases, through multiple regression layers or other machine learning or deep learning architectures, etc.), instance segmentation based on partial annotation, and / or implementing user interfaces configured to facilitate instance segmentation within biological samples, as described above. Figures 1-11 The methods, systems, and apparatuses illustrated relate to examples of different embodiments including various components and steps, which may be considered alternatives or may be used in combination with each other in various embodiments. Figures 1-11 The descriptions of the methods, systems, and apparatuses shown are provided for illustrative purposes and should not be construed as limiting the scope of the different embodiments.

[0128] Refer to the attached diagram. Figure 1 This is a schematic diagram of a system 100 according to various embodiments for implementing digital microscopy using deep learning-based segmentation, implementing instance segmentation based on partial annotation, and / or implementing user interfaces configured to facilitate instance segmentation within biological samples.

[0129] exist Figure 1In non-limiting embodiments, system 100 may include computing system 105a, artificial intelligence (“AI”) system 110a, and data storage or database 115a that is local to computing system 105a and / or AI system 110a. In some cases, database 115a may be external to but communicatively coupled to computing system 105a. In other cases, database 115a may be integrated within computing system 105a. In some embodiments, AI system 110a—which may include, but is not limited to, at least one of machine learning systems, deep learning systems, neural networks, convolutional neural networks (“CNN”) or fully convolutional networks (“FCN”) (which may include the U-Net framework, etc.) and / or the like—may be external to but communicatively coupled to computing system 105a, or may be integrated within computing system 105a.

[0130] According to some embodiments, system 100 may also include a display device 120 that allows user 125 to view the field of view (“FOV”) of a biological sample or an image or video of the biological sample. System 100 may also include one or more user devices 130, one or more audio sensors 135 (optional), a camera 140 (optional), and a microscope 145 (optional). In some cases, one or more user devices 130 may include, but are not limited to, smartphones, mobile phones, tablets, laptops, desktop computers, keyboards, keypads, computer mice, or monitors and / or the like. In some cases, one or more audio sensors 135 may include, but are not limited to, one or more microphones, one or more voice recorders, or one or more recorders and / or the like. In some cases, camera 140 may include, but is not limited to, one or more eye-tracking sensors, one or more motion sensors, or one or more tracking sensors and / or the like.

[0131] According to some embodiments, one or more user devices 130 may be used to receive user input from user 125, which indicates annotations or markings of objects of interest observed by user 125 while viewing the field of view of a biological sample, whether viewed on the display screen of display device 120 or observed through the eyepiece of microscope 145. One or more audio sensors 135 may be used to record the user 125's voice or spoken annotations while user 125 views the biological sample at FOV on display device 120 or through the eyepiece of microscope 145. Camera 140 may capture images or videos of user 125 when user 125 is within the FOV 140a of camera 140 (in some cases, capturing images or videos of at least one eye of user 125).

[0132] The computing system 105a may be communicatively coupled (via wireless (as depicted by a lightning bolt symbol) or wired (as depicted by a connecting line) to one or more of the following: the AI ​​system 110a, the database 115a, the display device 120, one or more user devices 130, one or more audio sensors 135, the camera 140, and / or the microscope 145. The computing system 105a, the AI ​​system 110a, the database 115a, the display device 120, the one or more user devices 130, the one or more audio sensors 135, the camera 140, and / or the microscope 145 may be set up or located within a working environment 150, which may include, but is not limited to, a laboratory, clinic, medical facility, research facility, healthcare facility, or room and / or the like.

[0133] System 100 may also include a remote computing system 105b (optional), an AI system 110b (optional), and a database 115b (optional), which can be communicatively coupled to computing system 105a and / or AI system 110a via network 155. In some cases, remote computing system 105b may include, but is not limited to, a web server, a web browser, or a cloud computing system and / or the like. Remote computing system 105b, AI system 110b, and database 115b may also be similar to, or not identical to, computing system 105a, AI system 110a, and database 115a, respectively.

[0134] By way of example only, each of network 155 may include a local area network (“LAN”), including but not limited to fiber optic networks, Ethernet networks, token ring networks and / or the like; a wide area network (“WAN”); a wireless wide area network (“WWAN”); a virtual network, such as a virtual private network (“VPN”); the Internet; an intranet; an extranet; a public switched telephone network (“PSTN”); an infrared network; and a wireless network, including but not limited to any IEEE 802.11 protocol suite, Bluetooth as known in the art. TM Networks operating under protocols and / or any other wireless protocols; and / or any combination of these and / or other networks. In a particular embodiment, each of network 155 may include an access network of an Internet Service Provider (“ISP”). In another embodiment, each of network 155 may include the ISP’s core network and / or the Internet.

[0135] In operation, computing system 105a, remote computing system 105b, and / or AI system 110a or 110b (collectively referred to as the "computing system," etc.) may perform data augmentation on a first image and a second image (optionally), the first image including the field of view ("FOV") of a first biological sample, and the second image including markers of instances of objects of interest in the first biological sample. In some cases, the first biological sample may include, but is not limited to, one of human tissue samples, animal tissue samples, or plant tissue samples and / or the like, wherein the objects of interest may include, but are not limited to, at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures and / or the like. In some embodiments, data augmentation of the first image and the second image may include, but is not limited to, at least one of elasticity enhancement or color enhancement and / or the like (in some cases, configured to facilitate instance segmentation).

[0136] While the focus is on biological samples as described herein, various embodiments are not limited thereto, and user interfaces for instance segmentation, training systems to generate or update AI models to predict instance segmentation, and / or configuring user annotations to facilitate instance segmentation can be adapted to non-biological samples, including but not limited to chemical samples, humans, animals, plants, insects, tools, vehicles, structures, landmarks, planets, stars, specific living objects, or specific inanimate objects, and / or the like. In this document, “instance segmentation” can refer to the separation and / or identification of instances of an object of interest (e.g., cells, tissues, molecular structures, parts of humans, parts of animals, parts of plants, parts of insects, parts of tools, parts of vehicles, parts of physical structures, parts of landmarks, planets, stars, parts of specific living objects, or parts of specific inanimate objects, etc.) from other instances of the object of interest or other objects of interest that are adjacent or neighboring to each other. Elastic enhancements or color enhancements are used to manipulate images to highlight or move the relative position or orientation of adjacent objects of interest or adjacent instances of objects of interest, thereby facilitating instance segmentation of such objects of interest.

[0137] The computing system can receive a (enhanced) first image and a (enhanced) second image. The computing system can train an AI system 110a or 110b to generate or update an AI model to predict instances of an object of interest, at least in part, based on multiple sets of at least two images generated from the second image, each of the at least two images in the multiple sets being distinct from each other. In some embodiments, the at least two images may include, but are not limited to, a centroid layer image that highlights the centroid of each labeled instance of the object of interest in the second image and a boundary layer image that highlights the edges or boundaries of each labeled instance of the object of interest in the second image. Alternatively, the at least two images may include, but are not limited to, at least a centroid layer image, a boundary layer image, and a semantic segmentation layer image, wherein the centroid layer image highlights the centroid of each labeled instance of the object of interest in the second image, the boundary layer image highlights the edges or boundaries of each labeled instance of the object of interest in the second image, and the semantic segmentation layer image includes semantic segmentation data for each labeled instance of the object of interest in the second image. In other alternative embodiments, the at least two images may include any number of images or surfaces that highlight different aspects of instances of the object of interest in the first biological sample.

[0138] In some embodiments, as part of training an AI system to generate or update an AI model to predict instances of an object of interest based at least in part on multiple sets of at least two images generated from a second image, a computing system may encode the (enhanced) second image using an encoder (which may be part of the computing system's software and / or hardware, or may be a separate device communicating with the computing system, such as a dedicated encoder, in some cases) to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image. In some embodiments, encoding the second image to generate the third encoded image may include calculating the centroid of each labeled instance of the object of interest in the second image using the computing system; and generating the third encoded image using the computing system, the third encoded image including a highlight of the centroid of each labeled instance of the object of interest. In some cases, encoding the second image to generate the fourth encoded image may include calculating the edges or boundaries of each labeled instance of the object of interest in the second image using the computing system; and generating the fourth encoded image using the computing system, the fourth encoded image including a highlight of the edges or boundaries of each labeled instance of the object of interest.

[0139] According to some embodiments, encoding a second image to generate a third encoded image may further include a computing system calculating: a first distance metric between each pixel in the third encoded image and each centroid of each marker instance of the object of interest; and a first function for generating a first proximity map, the first function being a function of the first distance metric, the third encoded image including the first proximity map. Similarly, encoding a second image to generate a fourth encoded image may further include a computing system calculating: a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of an edge or boundary of each marker instance of the object of interest; and a second function for generating a second proximity map, the second function being a function of the second distance metric, the fourth encoded image including the second proximity map. In some cases, the computing system may assign a first weighted pixel value to each pixel in the third encoded image based at least in part on at least one of the first distance metric, the first function, or the first proximity map calculated for each pixel; and may assign a second weighted pixel value to each pixel in the fourth encoded image based at least in part on at least one of the second distance metric, the second function, or the second proximity map calculated for each pixel.

[0140] In some embodiments, the computing system can determine a first pixel loss value between each pixel in the third coded image and its corresponding pixel in the fifth image; and can determine a second pixel loss value between each pixel in the fourth coded image and its corresponding pixel in the sixth image. The computing system can calculate the loss value using a loss function based on the product of a first weighted pixel value of each pixel in the third coded image and the first pixel loss value between the corresponding pixel in the third coded image and the fifth image, and the product of a second weighted pixel value of each pixel in the fourth coded image and the second pixel loss value between the corresponding pixel in the fourth coded image and the sixth image. In some cases, the loss function may include, but is not limited to, one of the following: mean squared error loss function, mean square logarithmic error loss function, mean absolute error loss function, Huber loss function, or weighted sum of squared differences loss function and / or the like. For example, calculating the loss value using the mean squared error loss function may include adding the first weighted pixel value of each pixel in the third coded image to the product of the first pixel loss value between each pixel in the third coded image and the corresponding pixel in the fifth image, and the second weighted pixel value of each pixel in the fourth coded image to the product of the second pixel loss value between each pixel in the fourth coded image and the corresponding pixel in the sixth image.

[0141] In some embodiments, the AI ​​system can update the AI ​​model by updating one or more parameters of the AI ​​model based on the calculated loss value. In some cases, one or more parameters may include, but are not limited to, a single parameter, multiple parameters between two and one hundred (inclusive), multiple parameters between one hundred and one thousand (inclusive), multiple parameters between one thousand and one million (inclusive), or more. The computational system can use the updated AI model to generate a fifth and a sixth image based on the first image.

[0142] In some cases, the labeling of instances of objects of interest in the second image may include, but is not limited to, at least one of the following: a complete annotation of the first instance of the object of interest that identifies the centroid and edges of the first instance of the object of interest; a partial annotation of the second instance of the object of interest that only identifies the centroid of the second instance of the object of interest; or an unknown annotation of the third instance of the object of interest that neither identifies the centroid nor the edges (i.e., otherwise represented as unknown); and / or the like. In some embodiments, the computing system may mask the second instance of the object of interest with partial annotations in corresponding pixels in the fourth and sixth images before calculating the loss value, without masking the second instance of the object of interest with partial annotations in the third or fifth images; and may mask the third instance of the object of interest with unknown annotations in corresponding pixels in the third and fifth images and in corresponding pixels in the fourth and sixth images before calculating the loss value. In some cases, for partial annotations or unknown annotations, masking at least a portion of the second or third instance of the object of interest may include masking a circle in the third and / or fourth coded images, the circle representing the distance from the centroid or a point within the partially annotated object represented by user input (e.g., mouse click, etc.). In some cases, the radius of the circle may be predefined or may be calculated "on the fly" based on information from complete annotations of objects in the same region. Although circular masking is described, other polygonal or geometric shapes may be used appropriately or as needed. Alternatively, masking may include changing the weights of specific pixels in a third coded image and corresponding pixels in a fifth image (or specific pixels in a fourth coded image and corresponding pixels in a sixth image) to the same value so that they cancel each other out during pixel-by-pixel comparisons.

[0143] The computing system may use a decoder (which may be part of the computing system's software and / or hardware, or may be a separate device communicating with the computing system, etc. (in some cases, a dedicated decoder, etc.)) to decode the fifth and sixth images to generate a seventh image, the seventh image including predicted labels of instances of objects of interest in the first biological sample. In some cases, this is achieved by applying at least one of one morphological operations that identify foreground and background labels in each of the fifth and sixth images before generating the seventh image, or by directly decoding the fifth and sixth images to generate the seventh image. In some cases, applying at least one of one morphological operations or one of one or more machine learning operations may include applying one or more morphological operations, wherein after decoding the fifth and sixth images by applying one or more morphological operations that identify foreground and background labels in each of the fifth and sixth images, the computing system may apply a watershed algorithm to generate the seventh image. In some cases, one or more morphological operations may include, but are not limited to, at least one of the following: reconstruction open transformation or region H-minimum transformation and / or the like.

[0144] In some embodiments, the first and second images (or enhanced first and second images) can be fed through the system multiple times (i.e., through multiple iterations, including but not limited to fewer than ten times, between ten and one hundred times (inclusive), between one hundred and one thousand times (inclusive), between one thousand and one million times (inclusive), or more). Each time, the third coded image is compared with the fifth image, the fourth coded image is compared with the sixth image, and a loss value is calculated based on a first weighted pixel value of each pixel in the third coded image multiplied by a first pixel loss value between each pixel in the third coded image and its corresponding pixel in the fifth image, and based on a second weighted pixel value of each pixel in the fourth coded image multiplied by a second pixel loss value between each pixel in the fourth coded image and its corresponding pixel in the sixth image. The calculated loss value is used to update one or more parameters of the AI ​​model to generate successive regression layers, each of which generates fifth and sixth images that gradually or successively become closer to the same as the corresponding third and fourth coded images. As a result, with each iteration (and using each result or successive regression layer), the decoded image (i.e., the seventh image) will gradually or successively become closer to the same as the second image (which may be referred to herein as the ground reality image).

[0145] According to some embodiments, the computing system can compare a seventh image with a second image to generate an instance segmentation evaluation result. In some cases, generating the instance segmentation evaluation result may include evaluating instance segmentation performance using one or more metrics, which may include, but are not limited to, at least one of the following: aggregated Jaccard index (“AJI”), F1 score, DICE index, average DICE index, or joint DICE index and / or the like. In some cases, the instance segmentation evaluation result may include, but is not limited to, instance segmentation evaluation metrics, one or more numerical instance segmentation evaluation scores, or instance segmentation classifications (including, but not limited to, true positives (“TP”), true negatives (“TN”), false positives (“FP”), false negatives (“FN”), oversegmentation or undersegmentation, etc.) and / or the like. The computing system may display the generated instance segmentation evaluation result on a display screen. In some cases, the seventh image may be generated by a label-controlled watershed algorithm using a regression layer (which may include an edge surface regression layer, etc.). In some cases, parameters for morphological operations may be set using the instance segmentation evaluation result (e.g., AJI score, etc.) as the objective function after applying Bayesian optimization.

[0146] In some cases, training an AI system to generate or update an AI model to predict instances of an object of interest, at least in part, based on multiple sets of at least two images generated from a second image, may include encoding the second image to generate a third and fourth coded image, training the AI ​​system to generate or update the AI ​​model to predict instances of an object of interest, at least in part, based on the third and fourth coded images, generating a fifth and sixth image, decoding the fifth and sixth images to generate a seventh image, and comparing the seventh image with the second image, etc. Although two images (in this case, the third and fourth coded images) are used to train the AI ​​system, various embodiments are not limited to this, and more than two images (or surfaces) may be used.

[0147] According to some embodiments, the computing system can receive an eighth image, which includes a field of view (FOV) of a second biological sample different from that of the first biological sample; it can generate two or more images based on the eighth image using an AI model generated or updated by a trained AI system, which are different from each other; and it can decode the two or more images using a decoder to generate a ninth image, which includes predicted labels of instances of objects of interest in the second biological sample. Similar to decoding the fifth and sixth images, decoding two or more images to generate the ninth image can include using the computing system and a decoder to decode the two or more images to generate the ninth image by applying at least one of one or more morphological operations that identify foreground and background markers in each of the two or more images before generating the ninth image, or by directly decoding the two or more images to generate the ninth image. In the case of applying one or more morphological operations, after decoding the two or more images by applying one or more morphological operations that identify foreground and background markers in each of the two or more images, the computing system can apply a watershed algorithm to generate the ninth image. In this way, trained AI systems and / or AI models can be used to predict the labeling of instances of objects of interest in new biological samples—in some cases where there is no ground-based imagery (or previously annotated imagery) corresponding to the new biological sample.

[0148] Alternatively or additionally, the computational system may generate a user interface configured to collect training data using at least one of a complete or partial annotation of an object of interest within a biological sample image, and may display a first image including the FOV of the first biological sample within a display portion of the user interface. The computational system may receive first user input from a user (e.g., a pathologist, clinician, physician, nurse, or laboratory technician) via the user interface, indicating the presence or location of each of a first plurality of objects of interest contained in the first image displayed in the display portion of the user interface. The computational system may generate a boundary around each of the first plurality of objects of interest based at least in part on the location of each of the first plurality of objects within the first image identified by the first user input and at least in part on the analysis of pixels in or around the corresponding location using an algorithm (which may include, but is not limited to, object detection algorithms, pixel recognition algorithms, edge detection algorithms, and / or the like).

[0149] In some cases, the computing system may receive second user input from a user via a user interface, the second user input indicating that a point within one of a first plurality of objects of interest (OPIs) moves from a previous position within a new position in the first image. The system may generate a new boundary around the one of the first plurality of OPIs displayed in the display portion of the user interface, based at least in part on the new position of the point within the one of the first plurality of OPIs in the first image, represented by the second user input, and at least in part on the analysis of pixels in or around the new position of the point within the one of the first plurality of OPIs using an algorithm. This new boundary replaces the boundary previously generated around the one of the first plurality of OPIs. In some cases, the computing system may receive third user input from a user via a user interface, the third user input indicating a partial annotation of one of a second plurality of OPIs contained in the first image displayed in the display portion of the user interface. The system may generate a partial annotation symbol in the first image, based at least in part on the position of the third user input within the first image. This partial annotation symbol identifies the location of the unbounded centroid of one of the second plurality of OPIs. In some cases, the computing system may receive a fourth user input from a user via a user interface, which indicates that one of the third or more objects of interest is unknown or that the instance class of one of the third or more objects of interest should be switched to another instance class (e.g., cancer, benign, etc.). The system may also generate an unknown annotation symbol (i.e., a symbol or annotation representing an unknown instance or object, etc.) in the first image based at least in part on the position of the fourth user input within the first image, identifying the location of the unknown object represented by the fourth user input. Alternatively, the system may switch the instance class of one of the third or more objects of interest to another instance class selected by the fourth user input (e.g., switching between cancer and benign, switching between full annotation and partial annotation, switching between partial annotation and unknown annotation, switching between full annotation and unknown annotation, etc.).

[0150] According to some embodiments, the first user input may include, but is not limited to, a click input or a boundary region input. In some cases, the click input may define the location of the centroid of a first object among a first plurality of objects of interest identified by the click input, while the boundary region input may define a region within a first image that marks the outer boundary of a second object among a first plurality of objects of interest identified by the boundary region input. In some cases, the boundary region input may include, but is not limited to, a rectangular boundary region input, a circular boundary region input, a polygon placement input, or a line placement input and / or the like. In some embodiments, the second user input may include, but is not limited to, click and drag input. In some cases, the third user input may include, but is not limited to, a double-click input, wherein the third user input selects or deselects the boundary around one of the second plurality of objects of interest. In some cases, the fourth user input may include, but is not limited to, a shift plus mouse click input or a key plus mouse click input, wherein the fourth user input may include, but is not limited to, switching between full annotations and unknown annotations or switching between instance classes from a list of instance classes. However, various embodiments are not limited to these specific inputs, and these inputs may be any suitable input for indicating full annotations, partial annotations, and / or unknown annotations, etc.

[0151] The computing system can generate at least one of a second image or an annotation dataset based on a first image. The second image includes data about the location of each of the first plurality of objects of interest within the first image, based on the boundaries surrounding each of the first plurality of objects of interest identified by the received first user input and the generated data. The annotation dataset includes at least one of pixel location data or coordinate data for each of the first plurality of objects within the first image, based on the boundaries surrounding each of the first plurality of objects of interest identified by the received first user input and the generated data. In this way, the system provides a fast and efficient UI that allows users (or annotators) to generate annotations in an effective manner. In particular, users do not need to open any menus or follow a complex set of operations to interact with the annotation system's UI. A complete annotation (i.e., generating a boundary around the location marked by the click input or boundary area input, etc.) can be generated through a single operation (i.e., using click input or boundary area input, etc.). To change the automatically generated boundary, the user only needs to move a point within the instance or object using a single operation (i.e., using click-drag input, etc.), causing the system to redraw or regenerate a new boundary around the instance or object. Therefore, users don't need to waste time manually drawing around the edges or boundaries of instances or objects to obtain full annotations. Similarly, a full annotation can be changed to a partial annotation, or the class of an instance or object can be changed, through a single action (i.e., shift plus mouse click, key plus mouse click, or mouse / keyboard combination, etc.). Actions are not bound to specific mouse / keyboard operations; instead, any combination can be used or customized appropriately or as needed.

[0152] In some embodiments, the computing system may train AI system 110a or 110b to generate or update AI models to predict instances of objects of interest in a first biological sample based at least in part on multiple sets of at least two images generated from at least one of a second image or annotation dataset, each of the at least two images in the multiple sets of at least two images being distinct from one another. In some cases, training an AI system to generate or update an AI model to predict instances of an object of interest based at least in part on at least two images may include: encoding at least one of the second images or annotated datasets using a computing system and an encoder (which may be part of the software and / or hardware of the computing system, or a separate device communicating with the computing system, etc. (in some cases, a dedicated encoder, etc.)) to generate a third coded image and a fourth coded image, the fourth coded image being different from the third coded image; training the AI ​​system to generate or update an AI model to predict instances of an object of interest based at least in part on the third coded image and the fourth coded image; using the AI ​​model generated or updated by the AI ​​system to generate a fifth and a sixth image based on the first image and based on training, the sixth image being different from the fifth image; decoding the fifth and sixth images using a computing system and a decoder (which may be part of the software and / or hardware of the computing system, or a separate device communicating with the computing system, etc. (in some cases, a dedicated decoder, etc.)) to generate a seventh image, the seventh image including a predicted label of an instance of an object of interest in the first biological sample; and (optionally) comparing the seventh image with the second image using a computing system to generate an instance segmentation evaluation result. The encoding of the second image and the training of the AI ​​system 110a or 110b can also be done as follows: Figure 9B To achieve as described.

[0153] These and other functions of system 100 (and its components) are described in more detail below with reference to Figures 2-9.

[0154] Figures 2A-2C (Collectively referred to as “Figure 2”) is a system flowchart illustrating various systems 200, 200', and 200" for implementing digital microscopy imaging using deep learning-based segmentation, instance segmentation based on partial annotation, and / or implementing user interfaces configured to facilitate instance segmentation within biological samples, according to various embodiments. In Figure 2, system 200 may include a computational system 205, which includes an encoder 210, a U-Net framework 215 or a regression layer of the U-Net framework 215 (the U-Net framework 215 is an implementation of a fully convolutional network (“FCN”), etc.), a loss function system 220, a decoder 225, and an accuracy evaluation system 230. System 200' may differ from system 200, as the computational system 205' of system 200' may further include a data augmentation system 235. Computational systems 205 or 205' may correspond to Figure 1 The computing system 105a or computing system 105b of the system 100, etc.

[0155] refer to Figure 2A In a non-limiting embodiment, the U-Net framework 215 may receive a first image 240, which includes the field of view (“FOV”) of a first biological sample. According to some embodiments, the first biological sample may include, but is not limited to, one of human tissue samples, animal tissue samples, or plant tissue samples and / or the like, while the object of interest may include, but is not limited to, at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures and / or the like. The encoder 210 may receive a second image 245, which includes labels of instances of the object of interest in the first biological sample. In some cases, the labels of instances of the object of interest in the second image 245 may include, but are not limited to, at least one of the following: a complete annotation (by the user) of the first instance of the object of interest identifying the centroid and edges of the first instance of the object of interest; a partial annotation (by the user) of the second instance of the object of interest identifying only the centroid of the second instance of the object of interest; or an unknown annotation (by the user) of a third instance of the object of interest not identifying the centroid and edges (i.e., otherwise represented as unknown). According to some embodiments, the user may include, but is not limited to, pathologists, clinicians, doctors, nurses, or laboratory technicians. In some cases, the first image 240 may be an image of size H x W (as in the case of a monochrome camera used for bright-field microscopy, phase microscopy, etc.), H x W x RGB, or H x W x N (i.e., height x width x red / green / blue, or height x width x N, etc., where N is an integer value; as in the case of spectral imaging, fluorescence, etc.), while the second image 245 may be an image of size H x W. In some cases, the second image 245 may be referred to as a ground reality image or an instance-segmented ground reality image, etc.

[0156] In some embodiments, the computing system 205 or encoder 210 may mask a second instance of the object of interest with partial annotations in corresponding pixels of the fourth and sixth coded images before calculating the loss value, without masking the second instance of the object of interest with partial annotations in the third or fifth coded images, and may mask a third instance of the object of interest with unknown annotations in corresponding pixels of the third and fifth coded images and object pixels of the fourth and sixth coded images before calculating the loss value. In some cases, masking at least a portion of the second or third instance of the object of interest, for partial annotations or unknown annotations, may include masking a circle in the third and / or fourth coded images that represents the distance from the centroid or a point within the object represented by a partial annotation input by the user (e.g., mouse click, etc.). In some cases, the radius of the circle may be predefined or may be calculated "on the fly" based on information from the complete annotation of the object in the same region. Although circular masking is described, other polygonal or geometric shapes may be used appropriately or as needed. Alternatively, masking may include changing the weights of specific pixels in the third coded image and corresponding pixels in the fifth image (or specific pixels in the fourth coded image and corresponding pixels in the sixth image) to the same value so that they cancel each other out during pixel-by-pixel comparisons.

[0157] Encoder 210 can calculate the centroid of each marker instance of the object of interest in the second image 245; can calculate the edges or boundaries of each marker instance of the object of interest in the second image 245; can generate a third encoded image 250a, which includes a highlight of the centroid of each marker instance of the object of interest; and can generate a fourth encoded image 250b, which includes a highlight of the edges or boundaries of each marker instance of the object of interest. In some embodiments, encoding the second image 245 to generate the third encoded image 250a may further include system 200 or encoder 210 calculating: a first distance metric between each pixel in the third encoded image and each centroid of each marker instance of the object of interest; and a first function for generating a first proximity map, the first function being a function of the first distance metric, the third encoded image including the first proximity map. Similarly, encoding the second image to generate the fourth encoded image may also include system 200 or encoder 210 calculating: a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of the edge or boundary of each labeled instance of the object of interest; and a second function for generating a second proximity map, the second function being a function of the second distance metric, the fourth encoded image including the second proximity map. In some cases, the computing system may assign a first weighted pixel value (collectively referred to as "first weight 250c", "nucleus channel weight 250c", "centroid channel weight 250c", "weight 250c", etc.) to each pixel in the third encoded image based at least in part on a first distance metric, a first function, or a first proximity map calculated for each pixel; and may assign a second weighted pixel value (collectively referred to as "second weight 250d", "edge channel weight 250d", "edge channel weight 250d", "weight 250d", etc.) to each pixel in the fourth encoded image based at least in part on a second distance metric, a second function, or a second proximity map calculated for each pixel; Encoder 210 can output the third encoded image 250a and the fourth encoded image 250b (collectively referred to as "encoded image 250" or "transformed image 250", etc.) to loss function system 220. Encoder 210 can also output the first weighted pixel value of each pixel in the third encoded image and the second weighted pixel value of each pixel in the fourth encoded image to loss function system 220.

[0158] Simultaneously, the U-Net framework 215 can receive the first image 240 and can generate a fifth image 260a and a sixth image 260b based on the first image 240, one or more parameters of the regression layer or AI model, or determined updates of one or more parameters, using the regression layer or AI model of the U-Net framework 215. The generated fifth image 260a can simulate an image including the centroid of each predicted instance of the object of interest (e.g., a third encoded image 250a, etc.), while the generated sixth image 260b can simulate an image including the edges or boundaries of each predicted instance of the object of interest (e.g., a fourth encoded image 250b, etc.). The U-Net framework 215 can send the generated fifth image 260a and the generated sixth image 260b (collectively referred to as "generated image 260" or "predicted image 260," etc.) to the decoder 225 and also to the loss function system 220.

[0159] The loss function system 220 can determine a first pixel loss value between each pixel in the third coded image 250a and a corresponding pixel in the fifth image 260a; and can determine a second pixel loss value between each pixel in the fourth coded image 250b and a corresponding pixel in the sixth image 260b. The loss function system 220 can use a loss function to calculate a loss value based on the product of a first weighted pixel value 250c of each pixel in the third coded image 250a and the first pixel loss value between the corresponding pixel in the third coded image 250a and the fifth image 260a, and the product of a second weighted pixel value 250d of each pixel in the fourth coded image 250b and the second pixel loss value between the corresponding pixel in the fourth coded image 250b and the sixth image 260b. In some embodiments, the loss function may include, but is not limited to, a mean squared error loss function, a mean squared logarithmic error loss function, a mean absolute error loss function, a Huber loss function, or a weighted sum of squared differences loss function and / or the like. The loss function system 220 can update one or more parameters of the regression layer or AI model based on the calculated loss value, and can send the updated one or more parameters or the calculated loss value 255 to the U-Net framework 215. System 200 can train the U-Net framework 215 to generate or update the AI ​​model to generate or regenerate the fifth image 260a and the sixth image 260b using the updated one or more parameters, at least in part based on the third encoded image 250a and the fourth encoded image 250b to predict instances of objects of interest. Although system 200 uses the U-Net framework 215, various embodiments are not limited thereto, and any suitable AI system can be used, including but not limited to at least one of machine learning systems, deep learning systems, neural networks, convolutional neural networks (“CNN”) or fully convolutional networks (“FCN”) and / or the like.

[0160] Decoder 225 can decode fifth image 260a and sixth image 260b to generate a seventh image 265, which includes predicted labels of instances of objects of interest in the first biological sample. In some embodiments, decoding fifth image 260a and sixth image 260b to generate the seventh image 265 may include decoding fifth image 260a and sixth image 260b to generate the seventh image 265 by applying at least one of one or more morphological operations to identify foreground and background labels in each of fifth image 260a and sixth image 260b before generating the seventh image 265, or by directly decoding fifth image 260a and sixth image 260b to generate the seventh image 265. In the case where one or more morphological operations are applied to identify foreground and background labels in each of fifth image 260a and sixth image 260b, after decoding fifth image 260a and sixth image 260b by applying one or more morphological operations, decoder 225 may apply a watershed algorithm to generate the seventh image 265. In some cases, one or more morphological operations may include, but are not limited to, at least one of the following: reconstruction open transformation or region H-minimum transformation and / or the like. During training, decoder 225 may output a seventh image 265 to accuracy evaluation system 230.

[0161] In some cases, the accuracy evaluation system 230 can generate an instance evaluation result by comparing the seventh image 265 with the enhanced second image 245' using one or more metrics to evaluate instance segmentation performance. In some cases, the one or more metrics may include, but are not limited to, at least one of the following: aggregated Jaccard index (“AJI”), F1 index, DICE index, average DICE index, or joint DICE index and / or the like. As described above, in some cases, the seventh image 265 can be generated using a label-controlled watershed algorithm using a regression layer (which may include an edge surface regression layer, etc.). In some cases, the parameters used for morphological operations can be set with the instance segmentation evaluation result (e.g., AJI score, etc.) as the objective function after applying Bayesian optimization. The accuracy evaluation system 230 can output the instance evaluation result or comparison value as a feedback value 270. In some cases, the generated instance segmentation evaluation result 270 can be displayed on a display device (e.g., Figure 1On the display screen of a display device 120, etc. In some cases, the instance segmentation evaluation result 270 may include, but is not limited to, instance segmentation evaluation metrics, one or more numerical instance segmentation evaluation scores, or instance segmentation classifications (including, but not limited to, true positive (“TP”), true negative (“TN”), false positive (“FP”), false negative (“FN”), oversegmentation or undersegmentation, etc.) and / or at least one of the like.

[0162] refer to Figure 2B In a non-limiting embodiment, the first image 240 and the second image 245 can be data-enhanced by the data enhancement system 235 to generate an enhanced first image 240' and an enhanced second image 245', respectively. In some cases, the data enhancement of the first image 240 and the second image 245 may include, but is not limited to, elastic enhancement or color enhancement (configured in some cases to facilitate instance segmentation) and / or the like (e.g., respectively as...). Figure 5 and Figure 6 At least one of the following (shown). The U-Net framework 215 can receive an enhanced first image 240', while the encoder 210 can receive an enhanced second image 245'. Figure 2B The encoder 210, the U-Net framework (or the regression layer or AI model of the U-Net framework) 215, the loss function system 220, the decoder 225, and the accuracy evaluation system 230 can be integrated with... Figure 2A The encoder 210, the U-Net framework (or the regression layer of the U-Net framework) 215, the loss function system 220, the decoder 225 and the accuracy evaluation system 230 work in a similar manner, except that the first image 240 and the second image 245 are replaced with enhanced first images 240' and enhanced second images 245'.

[0163] Turn Figure 2C In a non-limiting embodiment, a visual depiction is provided to illustrate the training process, as described above regarding Figure 2ASpecifically, the U-Net regression layer or frame 215 may receive a first image or input image 240, which includes the field of view (“FOV”) of the first biological sample. The encoder 210 may receive a second image or ground reality image 245, which includes labels of instances of objects of interest in the first biological sample. In some cases, the labels of instances of objects of interest in the second image or ground reality image 245 may include, but are not limited to, at least one of the following: a complete annotation (by the user) of the first instance of the object of interest that identifies the centroid and edges of the first instance of the object of interest; a partial annotation (by the user) of the second instance of the object of interest that identifies only the centroid of the second instance of the object of interest; or an unknown annotation (by the user) of a third instance of the object of interest that neither identifies the centroid nor the edges (i.e., otherwise indicated as unknown); and / or the like. In some cases, the first image or input image 240 can be an image of size H x W (e.g., for monochrome cameras used in bright-field microscopy, phase microscopy, etc.), H x W x RGB, or H x W x N (i.e., height x width x red / green / blue, or height x width x N, etc., where N is an integer value; e.g., in the case of spectral imaging, fluorescence, etc.), while the second image or ground reality image 245 can be an image of size H x W. In some cases, the second image or ground reality image 245 may be referred to as an instance-segmented ground reality image, etc.

[0164] In some embodiments, the computing system 205 or 205' or encoder 210 may mask a second instance of the object of interest with partial annotations in corresponding pixels of the fourth and sixth coded images before calculating the loss value, without masking the second instance of the object of interest with partial annotations in the third or fifth coded images, and may mask a third instance of the object of interest with unknown annotations in corresponding pixels of the third and fifth coded images and corresponding pixels of the fourth and sixth coded images before calculating the loss value. In some cases, for partial annotations or unknown annotations, masking at least a portion of the second or third instance of the object of interest may include masking a circle in the third and / or fourth coded images, the circle representing the distance from the centroid or a point within the object represented by a partial annotation input by the user (e.g., mouse click, etc.). In some cases, the radius of the circle may be predefined, or may be calculated "on the fly" based on information from the complete annotation of the object in the same region. Although circular masking has been described, other polygonal or geometric shapes may be used appropriately or as needed. Alternatively, masking may include changing the weights of a specific pixel in the third coded image and the corresponding pixel in the fifth image (or a specific pixel in the fourth coded image and the corresponding pixel in the sixth image) to the same value so that they cancel each other out when compared pixel by pixel.

[0165] Encoder 210 can calculate the centroid or nucleus of each marked instance of an object of interest in the second image or ground reality image 245; can calculate the edge or boundary of each marked instance of an object of interest in the second image or ground reality image 245; can generate a third encoded image or nucleus distance image 250a, the third encoded image 250a including a highlight of the centroid of each marked instance of an object of interest; and can generate a fourth encoded image or edge distance image 250b, the fourth encoded image 250b including a highlight of the edge or boundary of each marked instance of an object of interest. In some embodiments, encoding the second image or ground reality image 245 to generate the third encoded image 250a may further include computing system 205 or encoder 210 calculating: a first distance metric between each pixel in the third encoded image and each centroid of each marked instance of an object of interest; and a first function generating a first proximity map, the first function being a function of the first distance metric, the third encoded image 250a including the first proximity map (or a first proximity score image). Similarly, encoding the second image or ground reality image to generate the fourth encoded image may also include the computation system 205 or encoder 210 calculating: a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of the edge or boundary of each labeled instance of the object of interest; and a second function for generating a second proximity map, the second function being a function of the second distance metric, the fourth encoded image 250b including the second proximity map (or a second proximity score image). In some cases, the computation system 205 or encoder 210 may assign a first weighted pixel value 250c to each pixel in the third encoded image based at least in part on a first distance metric, a first function, or a first proximity map calculated for each pixel; and may assign a second weighted pixel value 250d to each pixel in the fourth encoded image based at least in part on a second distance metric, a second function, or a second proximity map calculated for each pixel. As stated above regarding Figure 2A As described, but as Figure 2CThe example images depicted illustrate that the computational system 205 or encoder 210 can generate a first weighted image or nucleus channel weighted image 250c, which contains a first weighted pixel value for each pixel in the third encoded image 250a, and can generate a second weighted image or edge channel weighted image 250d, which contains a second weighted pixel value for each pixel in the fourth encoded image 250b. Encoder 210 can output the third encoded image 250a and the fourth encoded image 250b (collectively referred to as "encoded image 250" or "transformed image 250", etc.). Encoder 210 can also output the first weighted pixel value assigned to each pixel in the third encoded image and the second weighted pixel value assigned to each pixel in the fourth encoded image to the loss function system 220, in some cases, as the first weighted image or nucleus channel weighted image 250c and the second weighted image or edge channel weighted image 250d (collectively referred to as "weighted image", "weight", or "weight matrix", etc.).

[0166] Simultaneously, the U-Net regression layer or framework 215 can receive the first image or input image 240, and can generate a fifth image 260a and a sixth image 260b based on the first image or input image 240, based on one or more parameters of the regression layer or AI model, or determined updates of one or more parameters, etc. (In some cases, they can be combined into a single output image, such as the U-Net output image 260 (which is an H x W x 2 image), etc.). The generated fifth image 260a can simulate an image including the centroid or nucleus of each predicted instance of the object of interest (e.g., the third encoded image 250a, etc.), while the generated sixth image 260b can simulate an image including the edge or boundary of each predicted instance of the object of interest (e.g., the fourth encoded image 250b, etc.). The U-Net regression layer or frame 215 can send the generated fifth image 260a and the generated sixth image 260b (collectively referred to as "generated image 260" or "predicted image 260", etc.) to the decoder 225 and also to the loss function system 220.

[0167] The loss function system 220 can determine a first pixel loss value between each pixel in the third encoded image 250a and a corresponding pixel in the fifth image 260a; and can determine a second pixel loss value between each pixel in the fourth encoded image 250b and a corresponding pixel in the sixth image 260b. The loss function system 220 can use a loss function to calculate a loss value based on the product of a first weighted pixel value of each pixel in the third encoded image 250a (or the first weighted image or the nucleus channel weighted image 250c) multiplied by the first pixel loss value between each pixel in the third encoded image 250a and a corresponding pixel in the fifth image 260a, and the product of a second weighted pixel value of each pixel in the fourth encoded image 250b (or the second weighted image or the edge channel weighted image 250d) multiplied by the second pixel loss value between each pixel in the fourth encoded image 250b and a corresponding pixel in the sixth image 260b. In some embodiments, the loss function may include, but is not limited to, one of the following: mean squared error loss function, mean square logarithmic error loss function, mean absolute error loss function, Huber loss function, or weighted sum of squared differences loss function and / or the like. Loss function system 220 can update one or more parameters of the regression layer or AI model based on the calculated loss value, and can send the updated one or more parameters or the calculated loss value 255 (collectively referred to as "net weight update 255" or "parameter update 255", etc.) to the U-Net regression layer or frame 215. Computation system 205 can train the U-Net regression layer or frame 215 to generate or update the AI ​​model to predict instances of objects of interest, at least in part, based on the third encoded image 250a and the fourth encoded image 250b, by generating or regenerating the fifth image 260a and the sixth image 260b using the updated one or more parameters. Although computation system 205 uses the U-Net regression layer or frame 215, various embodiments are not limited thereto, and any suitable AI system can be used, including but not limited to machine learning systems, deep learning systems, neural networks, convolutional neural networks ("CNN") or fully convolutional networks ("FCN") and / or the like.

[0168] Decoder 225 can decode fifth image 260a and sixth image 260b to generate a seventh image or decoded image 265, the seventh image or decoded image 265 including predicted labels of instances of objects of interest in the first biological sample. In some embodiments, decoding fifth image 260a and sixth image 260b to generate the seventh image or decoded image 265 may include at least one of one morphological operations by applying one or more morphological operations to identify foreground and background labels in each of fifth image 260a and sixth image 260b before generating the seventh image or decoded image 265, or one or more machine learning operations by directly decoding fifth image 260a and sixth image 260b to generate the seventh image or decoded image 265. When applying one or more morphological operations to identify foreground and background markers in each of the fifth image 260a and the sixth image 260b, after decoding the fifth image 260a and the sixth image 260b by applying one or more morphological operations, the decoder 225 may apply a watershed algorithm to generate a seventh image or decode the image 265. In some cases, the one or more morphological operations may include, but are not limited to, at least one of the following: reconstruction open transform or region H-minimum transform.

[0169] In some embodiments, the first image 240 and the second image 245 (or enhanced first and second images 240' and 245') can be fed by the system multiple times (i.e., through multiple iterations, including but not limited to less than ten times, between ten and one hundred times (inclusive), between one hundred and one thousand times (inclusive), between one thousand and one million times (inclusive), or more). Each time, the third encoded image 250a is compared with the fifth image 260a, the fourth encoded image 250b is compared with the sixth image 260b, and a loss value is calculated based on the first weighted pixel value of each pixel in the third encoded image (or the first weighted image or the cell nucleus channel weighted image 250c) multiplied by the first pixel loss value between each pixel in the third encoded image and the corresponding pixel in the fifth image, and based on the second weighted pixel value of each pixel in the fourth encoded image (or the second weighted image or the edge channel weighted image 250d) multiplied by the second pixel loss value between each pixel in the fourth encoded image and the corresponding pixel in the sixth image. The calculated loss value is used to update one or more parameters 255 of the AI ​​model to generate successive regression layers. Each regression layer generates fifth and sixth images that gradually or successively become closer to the same as the corresponding third and fourth encoded images. As a result, with each iteration (and using each result or successive regression layer), the decoded image 260 (i.e., the seventh image) will gradually or successively become closer to the same as the ground reality image 245 (i.e., the second image). Although two images (in this case, the third encoded image 250a and the fourth encoded image 250b) are used to train the AI ​​system, various embodiments are not limited to this, and more than two images (or surfaces) can be used.

[0170] Figures 3A-3E (Collectively referred to as "Figure 3") is a schematic diagram illustrating various embodiments 300, 300', 300", 300'" and 300"" of a user interface for facilitating instance segmentation within biological samples, according to various embodiments. User interface 305 can be configured to collect training data for predicting instance segmentation within biological samples.

[0171] refer to Figure 3A In a non-limiting embodiment 300, the user interface 305 can display a first image 310 of the first biological sample (e.g., an image or video, etc.), and in some cases, can also display the field of view (“FOV”) 315 of the first image 310 of the first biological sample. The computing system (similar to...) Figure 1The computing system 105a or 105b, or the computing system 205 or 205' of Figure 2, etc., can receive first user input from a user (e.g., a pathologist, clinician, doctor, nurse, or laboratory technician, etc.) through user interface 305. The first user input indicates the presence or location of each of a first plurality of objects of interest contained within a first image displayed in the display portion of the user interface. The computing system can generate a boundary around each of the first plurality of objects of interest based at least in part on the location of each of the first plurality of objects within the first image identified by the first user input and at least in part on the analysis of pixels in or around the corresponding location using algorithms (which may include, but are not limited to, object detection algorithms, pixel recognition algorithms, edge detection algorithms, and / or the like).

[0172] In some cases, the computing system may receive second user input from a user via a user interface, the second user input indicating that a point within one of a first plurality of objects of interest (OPIs) moves from a previous position to a new position within the first image. The system may generate a new boundary around the one of the first plurality of OPIs displayed in the first image, based at least in part on the new position of the point within the one of the first plurality of OPIs indicated by the second user input and at least in part on the analysis of pixels in or around the new position of the point within the one of the first plurality of OPIs using an algorithm. This new boundary replaces a previously generated boundary around the one of the first plurality of OPIs. In some cases, the computing system may receive third user input from a user via a user interface, the third user input indicating a partial annotation of one of a second plurality of OPIs contained in the first image displayed in the user interface. The system may generate a partial annotation symbol in the first image, based at least in part on the position of the third user input within the first image. This partial annotation symbol identifies the location of the unbounded centroid of one of the second plurality of OPIs. In some cases, the computing system may receive a fourth user input from a user via a user interface, the fourth user input indicating that one of the third plurality of objects of interest is unknown or that the instance class of one of the third plurality of objects of interest should be switched to another instance class (e.g., cancer, benign, etc.), and may generate an unknown annotation symbol (i.e., a symbol or annotation representing an unknown instance or object, etc.) in the first image based at least in part on the position of the fourth user input within the first image, identifying the location of the unknown object represented by the fourth user input, or may switch the instance class of the selected object among the third plurality of objects of interest to another instance class selected by the fourth user input (e.g., switching between cancer and benign, switching between full annotation and partial annotation, switching between partial annotation and unknown annotation, switching between full annotation and unknown annotation, etc.).

[0173] In some embodiments, the first user input may include, but is not limited to, a click input or a boundary region input. In some cases, the click input may define the location of the centroid of a first object among a first plurality of objects of interest identified by the click input, while the boundary region input may define a region within a first image that marks the outer boundary of a second object among the first plurality of objects of interest identified by the boundary region input. In some cases, the boundary region input may include, but is not limited to, a rectangular boundary region input, a circular boundary region input, a polygon placement input, or a line placement input and / or the like. In some embodiments, the second user input may include, but is not limited to, click and drag input. In some cases, the third user input may include, but is not limited to, a double-click input, wherein the third user input selects or deselects the boundary surrounding one of the second plurality of objects of interest. In some cases, the fourth user input may include, but is not limited to, a shift plus mouse click input or a key plus mouse click input, wherein the fourth user input may include, but is not limited to, switching between full annotations and unknown annotations or switching between instance classes from a list of instance classes. However, various embodiments are not limited to these specific inputs, and these inputs may be any suitable input for indicating full annotations, partial annotations, and / or unknown annotations, etc.

[0174] like Figure 3A As shown, object 320 corresponds to the object of interest with a complete annotation, object 325 corresponds to the object of interest with a partial annotation, and object 330 corresponds to an unknown object of interest.

[0175] Turning Figure 3B In a non-limiting embodiment 300', the user interface 305 may display a second image 310a of the second biological sample (which may correspond to an image of the same type as image 240 in FIG2, etc.), a third image 310b depicting an instance segmentation 335 of objects of interest annotated or marked by the user (which may correspond to an image of the same type as image 245 in FIG2, etc.), a fourth image 310c depicting the centroid 340 of each of the first plurality of objects of interest within the third image 310b displayed in the display portion of the user interface (which may correspond to an image of the same type as image 250a in FIG2, etc.), and a fifth image 310d depicting the boundary or boundary region 345 around each of the first plurality of objects of interest contained within the third image 310b displayed in the display portion of the user interface (which may correspond to an image of the same type as image 250b in FIG2, etc.).

[0176] Figure 3CA user interface 305 is depicted, in which the second to fifth images 310a-310d are scaled down to result in second to fifth images 310a′-310d′ (here, images 310c′ and 310d′ may correspond to images of the same type as images 250c and 250d in Figure 2, respectively, etc.), and weight values ​​(e.g., weight matrices, etc.) may be introduced to focus the model's attention on challenging regions of the biological sample (e.g., crowded regions, etc.). In some cases, the weight values ​​or matrices may be defined by equations, such as, but not limited to:

[0177]

[0178] Where w(x) is a weighting function applied to each pixel x, w0 is a weighting constant, d1(x) is a first distance metric between each pixel x in image 310c and each centroid or nearest center pixel of each of the first plurality of objects of interest in the central image (e.g., image 250a, image 310c, etc.), d2(x) is a second distance metric between each pixel x in image 310d and the nearest (or second nearest) edge pixel of the edge or boundary of each of the first plurality of objects of interest, and σ is the sigma value representing the cluster edge of each object of interest, and d1(x) and d2(x) always refer to pixels within the same source image.

[0179] exist Figure 3C In the non-limiting example 300″, the centroid weight transformation can utilize, for example, a w0 value of 5 and a σ or sigma value of 10 to produce a weighted image as shown in the fourth image 310c′ (similar to the weighted image 250c in FIG. 2, etc.), while the edge weight transformation can utilize, for example, a w0 value of 10 and a σ or sigma value of 3 to produce a weighted image as shown in the fifth image 310d′ (similar to the weighted image 250d in FIG. 2, etc.). Alternatively, the weight transformation can include changing the weights of a specific pixel in the third coded image and a corresponding pixel in the fifth image (or a specific pixel in the fourth coded image and a corresponding pixel in the sixth image) to the same value so that they cancel each other out when compared pixel by pixel. Although two images (in this case, the fourth image 310c and the fifth image 310d) are used to train the AI ​​system, different embodiments are not limited to this and more than two images (or surfaces) can be used.

[0180] refer to Figure 3D In a non-limiting embodiment 300", the user interface 305 can display a second biological sample (different from...) Figure 3B and Figure 3CThe sixth image 310e shows the first biological sample shown, and the seventh image 310f shows an instance segmentation 335' depicting objects of interest (e.g., fully annotated object 320, partially annotated object 325, unknown object 330, etc.) annotated or tagged by the user, a centroid 340 depicting each of the first plurality of objects of interest (e.g., fully annotated object 320, etc.) and a first mask 350 depicting each of the second plurality of objects of interest (e.g., unknown object 330, but not partially annotated object 325, etc.) contained in the seventh image 310f displayed in the display portion of the user interface. Eight images 310g (which is a centroid proximity map, which may correspond to an image of the same type as image 250a in FIG2, etc.), and a ninth image 310h (which is an edge proximity map, which may correspond to an image of the same type as image 250b in FIG2, etc.) depicting the boundary region 345 around each of the first plurality of objects of interest (e.g., fully annotated object 320, etc.) and the second mask 355 depicting each of the second plurality of objects of interest (e.g., unknown object 330 and partially annotated object 325, etc.) contained in the seventh image 310f displayed in the display portion of the user interface.

[0181] Turn Figure 3E In a non-limiting embodiment 300", the user interface 305 can display a sixth image 310e of the second biological sample, a seventh image 310f depicting instance segmentation 335 of objects of interest (e.g., fully annotated objects 320, partially annotated objects 325, unknown objects 330, etc.) annotated or labeled by the user, a tenth image 310i depicting predicted edge proximity scores with foreground and background markers overlaid on top of the image, and an eleventh image 310j depicting predicted instance segmentation of the first plurality of objects of interest contained within the seventh image 310f displayed in the display portion of the user interface. This system can be used to train an AI system (e.g., Figure 1 (And the AI ​​systems 110a, 110b, or 215 in Figure 2, etc.) generate or update AI models to predict instances of objects of interest, wherein the regression layer of the AI ​​system generates the tenth image 310i, such as Figure 3EAs shown. The tenth image 310i can be used as the input image for a label-based watershed algorithm, which can be used to generate the eleventh image 310j. In some cases, this is achieved by applying one or more morphological operations to identify foreground and background labels in the tenth image 310i before generating the eleventh image 310j, or by directly decoding the tenth image 310i to generate the eleventh image 310j. In the case of applying one or more morphological operations, after decoding the tenth image 310i by applying one or more morphological operations to identify foreground and background labels in the tenth image 310i, the computing system can apply the watershed algorithm to generate the eleventh image 310j. In some cases, the one or more morphological operations may include, but are not limited to, at least one of the following: reconstruction open transformation or region H-minimum transformation and / or the like. The eleventh image 310j may include predicted labels of instances of objects of interest in a second biological sample. As mentioned above, in some cases, the eleventh image 310j can be generated by using a label-controlled watershed algorithm with a regression layer (which may include an edge surface regression layer, etc.). In some cases, the parameters used for morphological operations can be set as the objective function after applying Bayesian optimization, using the instance segmentation evaluation result (e.g., AJI score, etc.). Although two images (in this case, images 310g and 310h (not shown) used to create proximity maps) are used to train the AI ​​system, various embodiments are not limited to this, and more than two images (or surfaces) can be used.

[0182] Figure 4 Examples 400 depict annotations of objects of interest in original images of first biological samples according to various embodiments and illustrate predictions of objects of interest by an artificial intelligence (“AI”) system.

[0183] refer to Figure 4A non-limiting example 400 depicts a ground-based image 405 and a predicted image 410. A first image 405a—also referred to herein as an “RGB image”, etc.—may include the field of view (“FOV”) of a first biological sample, while a second image 405b may include markers (by the user) of instances of objects of interest within the first biological sample. According to some embodiments, the first biological sample may include, but is not limited to, one of human tissue samples, animal tissue samples, or plant tissue samples and / or the like, while the objects of interest may include, but is not limited to, at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures and / or the like. The second image 405b—also referred to herein as a “ground-based segmented image”, etc.—may include fully annotated (depicted in second image 405b by colored shapes without white dots in the center) and partially annotated objects of interest (depicted in second image 405b by colored shapes with white dots in the center). The third image 405c—also referred to herein as the “centroid distance transformation image”, etc.—may include a highlighting of the centroid of each labeled instance of the object of interest (fully annotated and partially annotated), while the fourth image 405d—also referred to herein as the “edge distance transformation image”, etc.—may include a highlighting of the edges or boundaries of each labeled instance of the object of interest (for fully annotated objects), wherein objects for each partially annotated object are masked.

[0184] The fifth image 410a—also referred to herein as the “predicted centroid distance transformation image”, etc.—may include a highlighting of the centroid of each predicted instance of the object of interest, while the sixth image 410b—also referred to herein as the “predicted edge distance transformation image”, etc.—may include a highlighting of the edges or boundaries of each predicted instance of the object of interest. The seventh image 410c may include foreground markers (depicted by red dots, etc., in the seventh image 410c) and background markers (depicted by a green background, etc., in the seventh image 410c). The eighth image 410d—also referred to herein as the “instance segmentation prediction image” or “decoded image”, etc.—may include predicted markers of instances of the object of interest in the first biological sample. In some embodiments, in addition to highlighting cell nuclei omitted in the original annotation, the system may also highlight cell nuclei that are weakly or partially annotated or correctly segmented objects.

[0185] Figure 5 Examples 500 depict various images, according to various embodiments, showing elastic enhancement of the original image of a first biological sample and elastic enhancement of an annotated image of the original image.

[0186] refer to Figure 5In a non-limiting example 500, the first image 505a—also referred to herein as an “RGB image”, etc.—may include the field of view (“FOV”) of the first biological sample, while the second image 505b—also referred to herein as an “instance segmentation image”, etc.—may include (by the user) markers of instances of objects of interest in the first biological sample. According to some embodiments, the first biological sample may include, but is not limited to, one of human tissue samples, animal tissue samples, or plant tissue samples and / or the like, while the object of interest may include, but is not limited to, at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures and / or the like. The first image 505a and the second image 505b may each be provided with grid lines overlaid on the FOV of the first biological sample or an annotated image of the first biological sample to illustrate variations, but are not required for the instance segmentation process.

[0187] The third image 505c—also referred to herein as a “deformed RGB image”, etc.—may include an elastic enhancement of the first image 505a, while the fourth image 505d—also referred to herein as a “deformed instance segmentation image”, etc.—may include an elastic enhancement of the second image 505b. Figure 5 As shown, the grid lines in the third image 505c and the fourth image 505d are used to highlight the elastic enhancement of the first image 505a and the second image 505b.

[0188] Figure 6 Examples 600 depict various color-enhanced images of the original image showing a first biological sample according to various embodiments.

[0189] refer to Figure 6 In a non-limiting example 600, the first image 605a—also referred to herein as an “RGB image”, etc.—may include the field of view (“FOV”) of the first biological sample, while each of the second image 605b, the third image 605c, and the fourth image 605d—also referred herein as a “color-changing RGB image”, etc.—may include color changes of the FOV of the first biological sample to highlight objects of interest of different colors contained within the FOV of the first biological sample. According to some embodiments, the first biological sample may include, but is not limited to, one of human tissue samples, animal tissue samples, or plant tissue samples and / or the like, while the object of interest may include, but is not limited to, at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures and / or the like.

[0190] Figure 7 Example 700 depicts various images illustrating the predictive power of objects of interest based on complete and partial segmentation according to various embodiments.

[0191] refer to Figure 7 In a non-limiting example 700, the first image 705a—also referred to herein as an “RGB image”, etc.—may include the field of view (“FOV”) of the first biological sample. According to some embodiments, the first biological sample may include, but is not limited to, one of human tissue samples, animal tissue samples, or plant tissue samples and / or the like, while the object of interest may include, but is not limited to, at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures and / or the like. The second image 705b, the third image 705c, and the fourth image 705d (collectively referred to herein as “ground reality images”, etc.) may include markers (by the user) of instances of the object of interest in the first biological sample. The second image 705b—also referred herein as a “ground reality centroid distance transformation image”, etc.—may include a highlighting of the centroid of each marked instance of the object of interest, while the third image 705c—also referred herein as a “ground reality edge distance transformation image”, etc.—may include a highlighting of the edges or boundaries of each marked instance of the object of interest. The fourth image 705d—also referred to herein as the “ground real-world instance segmentation image”, etc.—may include (by the user) a label of an instance of an object of interest in a first biological sample based at least in part on a combination of the second image 705b and the third image 705c.

[0192] Images 710b, 710c, and 710d (collectively referred to as “completely segmented image 710”, etc.) depict the AI ​​model predictions when the training model is trained with complete segmentation annotations (i.e., 100% labels (by the user) of instances of the object of interest in the first biological sample). Image 710b—also referred to herein as “completely segmented centroid distance transformation image”, etc.—may include a highlighting of the predicted centroid for each labeled instance of the object of interest, while image 710c—also referred to herein as “completely segmented edge distance transformation image”, etc.—may include a highlighting of the predicted edges or boundaries for each labeled instance of the object of interest. Image 710d—also referred to herein as “completely segmented instance segmentation image”, etc.—may include predicted labels of instances of the object of interest in the first biological sample, at least in part based on a combination of images 710b and 710c.

[0193] Images 715b, 715c, and 715d (collectively referred to as “50% Partial Segmentation Image 715”, etc.) depict AI model predictions when the training model is trained with 50% segmentation annotations (i.e., the labels (by the user) of instances of the object of interest in the first biological sample include ground reality with centroids and edges for 50% of the instances, and ground reality with only centroids for 50% of the instances). Image 715b—also referred to herein as “50% Partial Segmentation Centroid Distance Transformation Image”, etc.—may include a highlighting of the predicted centroids for each labeled instance of the object of interest, while Image 715c—also referred to herein as “50% Partial Segmentation Edge Distance Transformation Image”, etc.—may include a highlighting of the predicted edges or boundaries for each labeled instance of the object of interest. Image 715d—also referred to herein as “50% Partial Segmentation Instance Segmentation Image”, etc.—may include predicted labels of instances of the object of interest in the first biological sample, at least in part based on a combination of Images 715b and 715c.

[0194] Figures 8A-8D (collectively referred to as "Figure 8") is a flowchart illustrating a method 800 for achieving digital microscopy using deep learning-based segmentation and / or instance segmentation based on partial annotation, according to various embodiments. Figure 8A Method 800 continues after the circular mark represented as "A" to Figure 8C And after the circular mark indicated by "B" from Figure 8C Return to Figure 8A . Figure 8A Method 800 continues after the circular mark represented by "C" to Figure 8D .

[0195] While techniques and processes are depicted and / or described in a particular order for illustrative purposes, it should be understood that specific processes may be reordered and / or omitted within the scope of various embodiments. Furthermore, while the method 800 shown in Figure 8 can be derived from or used by (and in some cases, described below with respect to the following items)... Figure 1 , Figure 2A , Figure 2B , Figure 2C , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 4 , Figure 5 , Figure 6 and Figure 7The corresponding systems, examples, or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300"", 400, 500, 600, and 700 (or components thereof) are implemented, but such a method can also be implemented using any suitable hardware (or software). Similarly, although Figure 1 , Figure 2A , Figure 2B , Figure 2C , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 4 , Figure 5 , Figure 6 and Figure 7 Each of the corresponding systems, examples, or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300", 400, 500, 600, and 700 (or components thereof) can operate according to the method 800 shown in FIG8 (e.g., by executing instructions embodied on a computer-readable medium), but Figure 1 , Figure 2A , Figure 2B , Figure 2C , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 4 , Figure 5 , Figure 6 and Figure 7 The systems, examples, or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300"", 400, 500, 600, and 700 may also operate and / or execute other suitable procedures according to other operating modes.

[0196] exist Figure 8A In a non-limiting embodiment, method 800 may include performing data augmentation on a first image using a computing system at optional block 802, the first image including the field of view (“FOV”) of the first biological sample. At optional block 804, method 800 may perform the same data augmentation on a second image using a computing system, the second image including markers of instances of objects of interest in the first biological sample.

[0197] In some embodiments, the computing system may include, but is not limited to, a computing system located in the work environment, a remote computing system located outside the work environment and accessible via a network, a web server, a web browser, or a cloud computing system and / or the like. In some cases, the work environment may include, but is not limited to, at least one of a laboratory, clinic, medical facility, research facility, healthcare facility, or room and / or the like. In some cases, the first biological sample may include, but is not limited to, a human tissue sample, an animal tissue sample, or a plant tissue sample and / or the like. In some cases, the object of interest may include, but is not limited to, at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures and / or the like. In some cases, the labeling of instances of the object of interest in the second image may include, but is not limited to, at least one of a complete annotation of the first instance of the object of interest identifying the centroid and edges of the first instance of the object of interest, or a partial annotation of the second instance of the object of interest identifying only the centroid of the second instance of the object of interest, and / or the like. In some cases, data enhancement of the first and second images may include, but is not limited to, at least one of elastic enhancement or color enhancement (configured in some cases to facilitate instance segmentation) and / or the like.

[0198] Method 800 may include receiving a first image or an enhanced first image (box 806) and receiving a second image or an enhanced second image (box 808). At box 810, method 800 may include encoding the second image using an encoder to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image. Method 800 may continue to the process at box 812, or may... Figure 8C The process continues from the circular mark labeled "A" to box 838.

[0199] At box 812, method 800 may include training an artificial intelligence (“AI”) system to generate or update an AI model to predict instances of an object of interest at least in part based on a third coded image and a fourth coded image. In some embodiments, the AI ​​system may include, but is not limited to, at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network (“CNN”), or a fully convolutional network (“FCN”) (which may include the U-Net framework, etc.) and / or the like. Method 800 may also include, at box 814, generating a fifth image and a sixth image based on the first image using a regression layer of the AI ​​system or the (updated) AI model, the sixth image being different from the fifth image. Method 800 may also include decoding the fifth image and the sixth image using a computational system and a decoder to generate a seventh image, the seventh image including predicted labels of instances of an object of interest in the first biological sample (box 816); comparing the seventh image with the second image using a computational system to generate an instance segmentation evaluation result (optional box 818); and displaying the generated instance segmentation evaluation result on a display screen using a computational system (optional box 820). In some embodiments, decoding the fifth and sixth images to generate a seventh image (in box 816) may include using a computing system and a decoder to decode the fifth and sixth images by applying one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or by directly decoding the fifth and sixth images to generate the seventh image using one or more machine learning operations. When applying one or more morphological operations, after decoding the fifth and sixth images by applying one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images, method 800 may include applying a watershed algorithm using a computing system to generate the seventh image. In some cases, the one or more morphological operations may include, but are not limited to, at least one of the following: reconstruction open transform or region H-minimum transform and / or the like. According to some embodiments, generating an instance segmentation evaluation result (in box 818) may include using one or more metrics to evaluate instance segmentation performance, which may include, but are not limited to, at least one of the following: aggregated Jaccard index (“AJI”), F1 index, DICE index, average DICE index, or joint DICE index and / or the like. In some cases, instance segmentation assessment results may include, but are not limited to, instance segmentation assessment metrics, one or more numerical instance segmentation assessment scores, or instance segmentation classifications (including, but not limited to, true positive (“TP”), true negative (“TN”), false positive (“FP”), false negative (“FN”), oversegmentation or undersegmentation, etc.) and / or at least one of the like.

[0200] Method 800 can continue after the circular mark represented by "C" to... Figure 8DThe process at box 852 in the middle.

[0201] refer to Figure 8B Encoding the second image to generate a third and fourth encoded image (in box 810) may include calculating the centroid of each marker instance of the object of interest in the second image using a computing system (box 822); and generating the third encoded image using a computing system, the third encoded image including a highlight of the centroid of each marker instance of the object of interest (box 824). In some embodiments, encoding the second image to generate the third and fourth encoded images (in box 810) may further include calculating a first distance metric between each pixel in the third encoded image and each centroid of each marker instance of the object of interest using a computing system (box 826); and calculating a first function using a computing system to generate a first proximity map, the first function being a function of the first distance metric, the third encoded image including the first proximity map (box 828). Alternatively or additionally, encoding the second image to generate a third and fourth encoded image (in box 810) may include calculating the edges or boundaries of each marked instance of the object of interest in the second image using a computing system (box 830); and generating the fourth encoded image using a computing system, the fourth encoded image including highlighting the edges or boundaries of each marked instance of the object of interest (box 832). In some embodiments, encoding the second image to generate the third and fourth encoded images (in box 810) may further include calculating a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of the edge or boundary of each marked instance of the object of interest using a computing system (box 834); and calculating a second function using a computing system to generate a second proximity map, the second function being a function of the second distance metric, the fourth encoded image including the second proximity map (box 836).

[0202] According to some embodiments, the labeling of instances of objects of interest in the second image may include, but is not limited to, at least one of the following: a complete annotation of the first instance of the object of interest that identifies the centroid and edges of the first instance of the object of interest; a partial annotation of the second instance of the object of interest that only identifies the centroid of the second instance of the object of interest; or an unknown annotation of the third instance of the object of interest that neither identifies the centroid nor the edges. Figure 8CIn box 838 (after the circular marker denoted as "A"), method 800 may include using a computing system to mask a second instance of the object of interest with partial annotations in corresponding pixels of the fourth and sixth images, without masking the second instance of the object of interest with partial annotations in the third or fifth image, and using a computing system to mask a third instance of the object of interest with unknown annotations in corresponding pixels of the third and fifth images and corresponding pixels of the fourth and sixth images. Method 800 may also include using a computing system to assign a first weighted pixel value to each pixel in the third coded image based at least in part on at least one of a first distance metric, a first function, or a first proximity map for each pixel (box 840), and to assign a second weighted pixel value to each pixel in the fourth coded image based at least in part on at least one of a second distance metric, a second function, or a second proximity map for each pixel (box 842). In box 844, method 800 may include using a computing system to determine a first pixel loss value between each pixel in the third coded image and a corresponding pixel in the fifth image. At box 846, method 800 may include using a computational system to determine a second pixel loss value between each pixel in the fourth coded image and a corresponding pixel in the sixth image. Method 800 may also include, at box 848, using a computational system to calculate a loss value based on a loss function, using a first weighted pixel value of each pixel in the third coded image multiplied by the product of the first pixel loss value between each pixel in the third coded image and a corresponding pixel in the fifth image, and a second weighted pixel value of each pixel in the fourth coded image multiplied by the product of the second pixel loss value between each pixel in the fourth coded image and a corresponding pixel in the sixth image. The loss function may include, but is not limited to, one of the following: mean squared error loss function, mean squared logarithmic error loss function, mean absolute error loss function, Huber loss function, or weighted sum of squared differences loss function and / or the like. At box 850, method 800 may include using an AI system to update one or more parameters of the AI ​​model based on the calculated loss value. Method 800 may return after a circular marker denoted as "B". Figure 8A The process at box 812. In some cases, generating the fifth and sixth images (in box 814) may include generating the fifth and sixth images based on the first image using a regression layer of the AI ​​system or using an updated AI model.

[0203] exist Figure 8DAt box 852 (after the circular marker indicated by "C"), method 800 may include receiving an eighth image using a computing system, the eighth image including a FOV of a second biological sample different from that of the first biological sample. Method 800 may further include, at box 854, generating two or more images based on the eighth image using an AI model generated or updated by a trained AI system, the two or more images being different from each other. At box 856, method 800 may include decoding the two or more images using a computing system and a decoder to generate a ninth image, the ninth image including predicted labels of instances of objects of interest in the second biological sample.

[0204] In some embodiments, decoding the fifth and sixth images to generate the seventh image may include using a computing system and a decoder to decode the fifth and sixth images to generate the seventh image by applying at least one of one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images before generating the seventh image, or by directly decoding the fifth and sixth images to generate the seventh image. When applying one or more morphological operations, the method may include applying a watershed algorithm to generate the seventh image (not shown in Figure 8) after decoding the fifth and sixth images by applying one or more morphological operations that identify foreground and background markers in each of the fifth and sixth images. In some cases, the one or more morphological operations may include, but are not limited to, at least one of the following: reconstruction open transform or region H-minimum transform and / or the like.

[0205] Similarly, decoding two or more images to generate a ninth image may include using a computational system and a decoder to decode two or more images by applying one or more morphological operations that identify foreground and background markers in each of the two or more images before generating the ninth image, or by directly decoding two or more images to generate the ninth image. In the case of applying one or more morphological operations, the method may include applying a watershed algorithm to generate the ninth image (not shown in Figure 8) after decoding two or more images by applying one or more morphological operations that identify foreground and background markers in each of the two or more images.

[0206] Figures 9A-9D (collectively referred to as “Figure 9”) is a flowchart illustrating a method 900 for implementing digital microscopy using deep learning-based segmentation, implementing instance segmentation based on partial annotation, and / or implementing a user interface configured to facilitate instance segmentation within biological samples, according to various embodiments. Figure 9A Method 900 continues after the circular mark represented as "A" to Figure 9BAfter the circular mark indicated by "B" Figure 9B Continue to Figure 9C And after the circular mark indicated by "C" from Figure 9C Continue to Figure 9D .

[0207] While the techniques and processes are depicted and / or described in a particular order for illustrative purposes, it should be understood that specific processes may be reordered and / or omitted within the scope of various embodiments. Furthermore, although the method 900 shown in Figure 9 can be derived from or used by (and in some cases, as described below with respect to the following items)... Figure 1 , Figure 2A , Figure 2B , Figure 2C , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 4 , Figure 5 , Figure 6 and Figure 7 The corresponding systems, examples, or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300"", 400, 500, 600, and 700 (or components thereof) are implemented, but such a method can also be implemented using any suitable hardware (or software). Similarly, although Figure 1 , Figure 2A , Figure 2B , Figure 2C , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 4 , Figure 5 , Figure 6 and Figure 7 Each of the corresponding systems, examples, or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300", 400, 500, 600, and 700 (or components thereof) can operate according to the method 900 shown in FIG9 (e.g., by executing instructions embodied on a computer-readable medium), but Figure 1 , Figure 2A , Figure 2B , Figure 2C , Figure 3A , Figure 3B , Figure 3C , Figure 3D , Figure 3E , Figure 4 , Figure 5 , Figure 6 and Figure 7 The systems, examples, or embodiments 100, 200, 200', 200", 300, 300', 300", 300'", 300"", 400, 500, 600, and 700 may also operate and / or execute other suitable procedures according to other operating modes.

[0208] exist Figure 9A In a non-limiting embodiment, method 900 at block 902 may include generating a user interface using a computing system, the user interface being configured to collect training data using at least one of full or partial annotations of objects of interest within an image of the biological sample. At block 904, method 900 may include using a computing system and displaying a first image including the field of view (“FOV”) of the first biological sample within a display portion of the user interface.

[0209] In some embodiments, the computing system may include, but is not limited to, a computing system located in the work environment, a remote computing system located outside the work environment and accessible via a network, a web server, a web browser, or a cloud computing system and / or the like. In some cases, the work environment may include, but is not limited to, at least one of a laboratory, clinic, medical facility, research facility, healthcare facility, or room and / or the like. In some cases, the first biological sample may include, but is not limited to, a human tissue sample, an animal tissue sample, or a plant tissue sample and / or the like.

[0210] Method 900 may further include receiving first user input from a user (e.g., a pathologist, clinician, physician, nurse, or laboratory technician) via a user interface using a computing system, the first user input indicating the presence or location of each of a first plurality of objects of interest contained within a first image displayed in a display portion of the user interface (box 906); generating a boundary around each of the first plurality of objects of interest using a computing system based at least in part on the location of each of the first plurality of objects within the first image identified by the first user input and at least in part on the analysis of pixels in or around the corresponding location using an algorithm (which may include, but is not limited to, object detection algorithms, pixel recognition algorithms, edge detection algorithms, and / or the like) (box 908); receiving second user input from a user via a user interface using a computing system, the second user input indicating the movement of a point within one of the first plurality of objects of interest from a previous location to a new location within the first image (optional box 910); and generating a boundary around each of the first plurality of objects of interest using a computing system based at least in part on the location of each of the first plurality of objects ... second user input and at least in part on the analysis of pixels in or around the corresponding location using an algorithm (which may include, but is not limited to, object detection algorithms, pixel recognition algorithms, edge detection algorithms, and / or the like) (box 908); receiving second user input from a user via a user interface using a computing system, the second user input indicating the movement of User input represents a new position of a point within one of a first plurality of objects of interest (OPIs) in a first image, and at least in part, based on the analysis of pixels in or around the new position of the point within the one OPIs using an algorithm, a new boundary is generated around the one OPIs contained in the first image displayed in the user interface, replacing the boundary previously generated around the one OPIs (optional box 912); a third user input is received from the user via the user interface using a computing system, the third user input indicating a partial annotation of one of a second plurality of OPIs contained in the first image displayed in the user interface (optional box 914); and a partial annotation symbol is generated in the first image using a computing system at least in part based on the position of the third user input within the first image, the partial annotation symbol identifying the location of the unbounded centroid of one of the second plurality of OPIs (optional box 916). Method 900 can continue after the circular marker represented as "A" to... Figure 9B The process at checkbox 918 in the text.

[0211] exist Figure 9BIn option 918 (following the circular marker denoted as "A"), method 900 may include receiving fourth user input from a user via a user interface using a computing system. The fourth user input indicates that one of a third plurality of objects of interest is unknown, or that an instance class of one of the third plurality of objects of interest should be switched to another instance class (e.g., cancer, benign, etc.). Method 900 may further include, in option 920, generating an unknown annotation symbol (i.e., a symbol or annotation representing an unknown instance or object, etc.) in the first image based at least in part on the location of the fourth user input within the first image. This unknown annotation symbol identifies the location of the unknown object represented by the fourth user input, or may switch the instance class of a selected third plurality of objects of interest to another instance class selected by the fourth user input (e.g., switching between cancer and benign, switching between full annotation and partial annotation, switching between partial annotation and unknown annotation, switching between full annotation and unknown annotation, etc.).

[0212] According to some embodiments, the first user input may include, but is not limited to, a click input or a boundary region input. In some cases, the click input may define the location of the centroid of a first object among a first plurality of objects of interest identified by the click input, while the boundary region input may define a region within a first image that marks the outer boundary of a second object among a first plurality of objects of interest identified by the boundary region input. In some cases, the boundary region input may include, but is not limited to, a rectangular boundary region input, a circular boundary region input, a polygon placement input, or a line placement input and / or the like. In some embodiments, the second user input may include, but is not limited to, click and drag input. In some cases, the third user input may include, but is not limited to, a double-click input, wherein the third user input selects or deselects the boundary around one of the second plurality of objects of interest. In some cases, the fourth user input may include, but is not limited to, a shift plus mouse click input or a key plus mouse click input, wherein the fourth user input may include, but is not limited to, switching between full annotations and unknown annotations or switching between instance classes from a list of instance classes. However, various embodiments are not limited to these specific inputs, and these inputs may be any suitable input for indicating full annotations, partial annotations, and / or unknown annotations, etc.

[0213] In box 922, method 900 may include generating at least one of a second image or an annotation dataset based on a first image using a computing system. The second image includes data about the location of each of the first plurality of objects of interest within the first image, relating to the boundaries around each of the first plurality of objects of interest identified by the received first user input and the generated data. The annotation dataset includes at least one of pixel location data or coordinate data for each of the first plurality of objects of interest within the first image, relating to the boundaries around each of the first plurality of objects of interest identified by the received first user input and the generated data.

[0214] At optional box 924, method 900 may include performing data augmentation on a first image and a second image using a computing system. In some cases, the data augmentation of the first image and the second image may include, but is not limited to, at least one of elastic enhancement or color enhancement (configured in some cases to facilitate instance segmentation) and / or the like. Method 900 may also include, at optional box 926, encoding the second image using a computing system and an encoder to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image. In some cases, the third encoded image may contain a first user input for each of a first plurality of objects of interest, while the fourth encoded image may contain a second user input for each of a second plurality of objects of interest.

[0215] Method 900 may further include: calculating a first distance metric between each pixel in the third coded image and each centroid of each marker instance of the object of interest using a computing system (optional box 928); calculating a first function using a computing system to generate a first proximity map, the first function being a function of the first distance metric, the third coded image including the first proximity map (optional box 930); calculating a second distance metric between each pixel in the fourth coded image and the nearest edge pixel of the edge or boundary of each marker instance of the object of interest using a computing system (optional box 932); and calculating a second function using a computing system to generate a second proximity map, the second function being a function of the second distance metric, the fourth coded image including the second proximity map (optional box 934). Method 900 may continue after the circular marker represented as "B" to... Figure 9C The process at checkbox 939 in the text.

[0216] According to some embodiments, the labeling of instances of objects of interest in the second image may include, but is not limited to, at least one of the following: a complete annotation of the first instance of the object of interest that identifies the centroid and edges of the first instance of the object of interest; a partial annotation of the second instance of the object of interest that only identifies the centroid of the second instance of the object of interest; or an unknown annotation of the third instance of the object of interest that neither identifies the centroid nor the edges. Figure 9C In optional box 936 (after the circular marker denoted as "B"), method 900 may include using a computing system to mask a second instance of the object of interest with partial annotations in the fourth coded image and object pixels in the sixth image, without masking the second instance of the object of interest with partial annotations in the third or fifth coded image, and using a computing system to mask a third instance of the object of interest with unknown annotations in corresponding pixels in the third and fifth coded images and corresponding pixels in the fourth and sixth coded images. Method 900 may also include using a computing system to assign a first weighted pixel value to each pixel in the third coded image based at least in part on a first distance metric, a first function, or a first proximity map calculated for each pixel (optional box 938); and using a computing system to assign a second weighted pixel value to each pixel in the fourth coded image based at least in part on at least one of a second distance metric, a second function, or a second proximity map calculated for each pixel (optional box 940). Method 900 may further include using a computational system to determine a first pixel loss value between each pixel in the third coded image and a corresponding pixel in the fifth image (optional box 942); using a computational system to determine a second pixel loss value between each pixel in the fourth coded image and a corresponding pixel in the sixth image (optional box 944); and using a computational system to calculate a loss value based on a loss function, using a first weighted pixel value of each pixel in the third coded image multiplied by the product of the first pixel loss value between each pixel in the third coded image and the corresponding pixel in the fifth image, and a second weighted pixel value of each pixel in the fourth coded image multiplied by the product of the second pixel loss value between each pixel in the fourth coded image and the corresponding pixel in the sixth image (optional box 946). In some cases, the loss function may include, but is not limited to, a mean squared error loss function, a mean squared logarithmic error loss function, a mean absolute error loss function, a Huber loss function, or a weighted sum of squared differences loss function and / or the like. In option 948, method 900 may include using an AI system to update one or more parameters of the AI ​​model by updating the AI ​​model based on the calculated loss value. Method 900 can return to the circled marker represented by "C". Figure 9D The process at position 950 in the box.

[0217] In optional box 950 (following the circular marker indicated by "C"), method 900 may include training an artificial intelligence ("AI") system to generate or update an AI model to predict instances of an object of interest at least in part based on a third coded image and a fourth coded image. Method 900 may also include generating a fifth and a sixth image based on the first image using a regression layer of the AI ​​system or the (updated) AI model, the sixth image being different from the fifth image (optional box 952); decoding the fifth and sixth images using a computational system and a decoder to generate a seventh image including predicted labels of instances of the object of interest in the first biological sample (optional box 954); comparing the seventh image and the second image using a computational system to generate an instance segmentation evaluation result (optional box 956); and displaying the generated instance segmentation evaluation result on a display screen using a computational system (optional box 958). According to some embodiments, generating instance segmentation evaluation results (in box 956) may include using one or more metrics to evaluate instance segmentation performance, which may include, but are not limited to, at least one of the following: aggregated Jaccard index (“AJI”), F1 score, DICE metric, average DICE metric, or combined DICE metric and / or the like. In some cases, instance segmentation evaluation results may include, but are not limited to, instance segmentation evaluation metrics, one or more numerical instance segmentation evaluation scores, or instance segmentation classifications (including, but not limited to, at least one of the following: true positive (“TP”), true negative (“TN”), false positive (“FP”), false negative (“FN”), oversegmentation or undersegmentation, etc.) and / or the like.

[0218] Exemplary System and Hardware Implementation

[0219] Figure 10 This is a block diagram illustrating an exemplary computer or system hardware architecture according to various embodiments. Figure 10 A schematic diagram of one embodiment of a computer system 1000 providing service provider system hardware is provided. The computer system 1000 can perform methods provided by various other embodiments, as described herein, and / or can perform the functions of a computer or hardware system (i.e., computing systems 105a, 105b, 205 and 205', artificial intelligence (“AI”) systems 110a and 110b, display device 120, user equipment 130, encoder 210, U-Net system or architecture 215, loss function system 220, decoder 225, accuracy evaluation system 230, and data augmentation system 235, etc.), as described above. It should be noted that... Figure 10 This is intended only to provide a general overview of the various components, of which one or more (or none) may be used as appropriate. Therefore, Figure 10 It extensively demonstrates how individual system elements can be implemented in a relatively separate or relatively more integrated manner.

[0220] Computer or hardware system 1000—which may represent embodiments of computer or hardware systems (i.e., computing systems 105a, 105b, 205 and 205', AI systems 110a and 110b, display device 120, user device 130, encoder 210, U-Net system or architecture 215, loss function system 220, decoder 225, accuracy evaluation system 230, and data augmentation system 235, etc.), as described above regarding Figure 1 Figure 9 illustrates hardware elements that can be electrically coupled (or otherwise communicated, as applicable) via bus 1005. These hardware elements may include one or more processors 1010, including but not limited to one or more general-purpose processors and / or one or more special-purpose processors (e.g., microprocessors, digital signal processing chips, graphics accelerators, and / or the like); one or more input devices 1015, which may include, but are not limited to, a mouse, keyboard, and / or the like; and one or more output devices 1020, which may include, but are not limited to, display devices, printers, and / or the like.

[0221] The computer or hardware system 1000 may also include (and / or communicate with) one or more storage devices 1025, which may include, but are not limited to, local and / or network-accessible storage devices, and / or may include, but are not limited to, disk drives, drive arrays, optical storage devices, solid-state storage devices such as random access memory (“RAM”) and / or read-only memory (“ROM”), which may be programmable, flash-updatable, and / or the like. Such storage devices can be configured to implement any suitable data storage, including but not limited to various file systems, database structures, and / or the like.

[0222] Computer or hardware system 1000 may also include communication subsystem 1030, which may include, but is not limited to, modems, network interface cards (wireless or wired), infrared communication devices, wireless communication devices and / or chipsets (e.g., Bluetooth). TM Devices, 802.11 devices, WiFi devices, WiMax devices, WWAN devices, cellular communication facilities, etc.) and / or the like. The communication subsystem 1030 may allow the exchange of data with networks (such as those described below, to name just one example), other computer or hardware systems, and / or any other devices described herein. In many embodiments, the computer or hardware system 1000 will further include working memory 1035, which may include RAM or ROM devices as described above.

[0223] The computer or hardware system 1000 may also include software elements, shown as currently residing in working memory 1035, including an operating system 1040, device drivers, executable libraries, and / or other code, such as one or more application programs 1045, which may include computer programs provided by various embodiments (including but not limited to hypervisors, VMs, etc.), and / or may be designed to implement methods provided by other embodiments and / or configure systems provided by other embodiments, as described herein. By way of example only, one or more processes described with respect to the above methods may be implemented as code and / or instructions executable by a computer (and / or a processor within a computer); in one aspect, such code and / or instructions may be used to configure and / or adapt a general-purpose computer (or other device) to perform one or more operations according to the described methods.

[0224] A set of these instructions and / or code may be encoded and / or stored on a non-transitory computer-readable storage medium, such as the storage device 1025 described above. In some cases, the storage medium may be incorporated into a computer system (e.g., system 1000). In other embodiments, the storage medium may be separable from the computer system (i.e., a removable medium, such as an optical disc, etc.) and / or provided in an installation package, such that the storage medium can be used to program, configure, and / or adapt a general-purpose computer having instructions / code stored thereon. These instructions may be in the form of executable code, which can be executed by the computer or hardware system 1000, and / or may be in the form of source code and / or installable code, which, when compiled and / or installed on the computer or hardware system 1000 (e.g., using any of a variety of generally available compilers, installers, compression / decompression utilities, etc.), then take the form of executable code.

[0225] It will be apparent to those skilled in the art that substantial variations can be made to suit specific requirements. For example, custom hardware (e.g., programmable logic controllers, field-programmable gate arrays, application-specific integrated circuits, and / or the like) may be used, and / or specific elements may be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connectivity to other computing devices, such as network input / output devices, may be employed.

[0226] As described above, in one aspect, some embodiments may employ a computer or hardware system (e.g., computer or hardware system 1000) to perform methods according to various embodiments of the invention. According to one set of embodiments, some or all of these methods are executed by computer or hardware system 1000 in response to processor 1010 executing one or more sequences of one or more instructions contained in working memory 1035 (which may be incorporated into operating system 1040 and / or other code, such as application 1045). These instructions may be read into working memory 1035 from another computer-readable medium (e.g., one or more storage devices 1025). By way of example only, execution of the sequence of instructions contained in working memory 1035 may cause processor 1010 to perform one or more processes of the methods described herein.

[0227] As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any medium that participates in providing data that enables a machine to operate in a particular manner. In embodiments implemented using a computer or hardware system 1000, various computer-readable media may relate to providing instructions / code to a processor 1010 for execution and / or being used to store and / or carry such instructions / code (e.g., as signals). In many implementations, the computer-readable medium is a non-transitory, physical, and / or tangible storage medium. In some embodiments, the computer-readable medium may take many forms, including but not limited to non-volatile media, volatile media, etc. Non-volatile media include, for example, optical discs and / or magnetic disks, such as storage device 1025. Volatile media include, but are not limited to, dynamic memory, such as working memory 1035. In some alternative embodiments, the computer-readable medium may take the form of a transmission medium, including but not limited to coaxial cables, copper wires, and optical fibers, including wires containing bus 1005, and various components of the communication subsystem 1030 (and / or the medium through which the communication subsystem 1030 provides communication with other devices). In another set of embodiments, the transmission medium may also take the form of waves (including, but not limited to, radio waves, sound waves and / or light waves, such as those generated during radio wave and infrared data communication).

[0228] Common forms of physical and / or tangible computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape or any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tapes, any other physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, any other memory chips or cassette tapes, carrier waves as described below, or any other media in which a computer can read instructions and / or code.

[0229] Various forms of computer-readable media may involve carrying one or more sequences of one or more instructions to processor 1010 for execution. By way of example only, the instructions may initially be carried on a disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit the instructions as signals via a transmission medium for reception and / or execution by computer or hardware system 1000. According to various embodiments of the invention, these signals (which may be in the form of electromagnetic signals, acoustic signals, optical signals, and / or similar signals) are examples of carrier waves on which instructions can be encoded.

[0230] The communication subsystem 1030 (and / or its components) typically receives signals, and then the bus 1005 can transmit the signals (and / or the data, instructions, etc. carried by the signals) to the working memory 1035, from which the processor 1005 retrieves and executes the instructions. The instructions received by the working memory 1035 may optionally be stored on the storage device 1025 before or after execution by the processor 1010.

[0231] As described above, a set of embodiments includes methods and systems for implementing digital microscopy imaging, and more specifically, methods, systems, and apparatus for implementing digital microscopy imaging using deep learning-based segmentation, implementing instance segmentation based on partial annotation, and / or implementing user interfaces configured to facilitate instance segmentation within biological samples. Figure 11 A schematic diagram of a system 1100, which may be used according to a set of embodiments, is shown. System 1100 may include one or more user computers, user devices, or client devices 1105. The user computers, user devices, or client devices 1105 may be running various commercial UNIX systems. TMThe user computer, user device, or client device 1105 may be any general-purpose personal computer (including, by way of example only, desktop computers, tablet computers, laptop computers, handheld computers, etc., running any suitable operating system, some of which are available from vendors such as Apple, Microsoft Corp.), cloud computing device, server, and / or workstation computer, or any of the following UNIX-like operating systems. The user computer, user device, or client device 1105 may also have any of a variety of applications, including one or more applications configured to perform the methods provided by the various embodiments (e.g., as described above), and one or more office applications, database client and / or server applications, and / or web browser applications. Alternatively, the user computer, user device, or client device 1105 may be any other electronic device capable of communicating and / or displaying and navigating web pages or other types of electronic documents via a network (e.g., network 1110 described below), such as a thin client computer, an internet-enabled mobile phone, and / or a personal digital assistant. Although the exemplary system 1100 is shown as having two user computers, user devices, or client devices 1105, any number of user computers, user devices, or client devices may be supported.

[0232] Some embodiments operate in a networked environment (which may include network 1110). Network 1110 can be any type of network familiar to those skilled in the art, and can support the use of various commercial (and / or free or proprietary) protocols (including but not limited to TCP / IP, SNA, etc.). TM IPX TM AppleTalk TM Data communication (etc.). For example only, network 1110 (similar to...) Figure 1 Networks 155, etc., may each include local area networks (“LANs”), including but not limited to fiber optic networks, Ethernet networks, token ring networks, etc. TM Networks, etc.; wide area networks (“WAN”); wireless wide area networks (“WWAN”); virtual networks, such as virtual private networks (“VPN”); the Internet; intranets; extranets; public switched telephone networks (“PSTN”); infrared networks; wireless networks, including but not limited to any IEEE 802.11 protocol suite, Bluetooth as known in the art, etc. TM A network operating under a protocol and / or any other wireless protocol; and / or any combination of these and / or other networks. In a particular embodiment, the network may include the access network of a service provider (e.g., an Internet service provider (“ISP”)). In another embodiment, the network may include the service provider’s core network and / or the Internet.

[0233] The embodiment may also include one or more server computers 1115. Each server computer 1115 may be configured with an operating system, including but not limited to any operating system discussed above, and any commercially (or free) available server operating system. Each server 1115 may also run one or more applications that may be configured to provide services to one or more clients 1105 and / or other servers 1115.

[0234] As an example only, as described above, one of the servers 1115 can be a data server, a web server, a cloud computing device, etc. The data server may include (or communicate with) a web server, which, as an example only, can be used to process requests for web pages or other electronic documents from user computer 1105. The web server can also run various server applications, including HTTP servers, FTP servers, CGI servers, database servers, Java servers, etc. In some embodiments of the invention, the web server may be configured to provide web pages that can be operated within a web browser on one or more user computers 1105 to perform the methods of the invention.

[0235] In some embodiments, server computer 1115 may include one or more application servers configured with one or more applications accessible to clients running on client computer 1105 and / or one or more of other servers 1115. By way of example only, server 1115 may be one or more general-purpose computers capable of executing programs or scripts (including, but not limited to, network applications, which in some cases may be configured to perform methods provided by various embodiments) in response to user computer 1105 and / or other servers 1115. By way of example only, network applications may be implemented in any suitable programming language (e.g., Java). TM C, C# TM One or more scripts or programs written in C++ and / or any scripting language (such as Perl, Python, or TCL), and any combination of programming and / or scripting languages. The application server may also include a database server, including but not limited to those available from Oracle. TM Microsoft TM Sybase TM IBM TMCommercially acquired database servers, such as those for processing requests from clients running on a user computer, user device, or client device 1105 and / or another server 1115 (including, depending on configuration, dedicated database clients, API clients, web browsers, etc.). In some embodiments, the application server can execute one or more processes for implementing digital microscopy imaging, more specifically, methods, systems, and apparatus for implementing digital microscopy imaging using deep learning-based segmentation, instance segmentation based on partial annotation, and / or implementing user-defined annotations configured to facilitate instance segmentation within biological samples, as described in detail above. Data provided by the application server can be formatted as one or more web pages (e.g., including HTML, JavaScript, etc.) and / or can be forwarded to the user computer 1105 via a web server (e.g., as described above). Similarly, the web server can receive web page requests and / or input data from the user computer 1105 and / or forward web page requests and / or input data to the application server. In some cases, the web server can be integrated with the application server.

[0236] According to a further embodiment, one or more servers 1115 may serve as a file server and / or may include one or more files (e.g., application code, data files, etc.) necessary for implementing various disclosed methods, incorporating an application running on user computer 1105 and / or another server 1115. Alternatively, as those skilled in the art will understand, the file server may include all necessary files that allow such an application to be remotely invoked by user computer, user device, or client device 1105 and / or server 1115.

[0237] It should be noted that the functions described in this article for the various servers (e.g., application servers, database servers, network servers, file servers, etc.) can be performed by a single server and / or multiple dedicated servers, depending on the specific requirements and parameters for implementation.

[0238] In some embodiments, the system may include one or more databases 1120a-1120n (collectively, “database 1120”). The location of each database 1120 is arbitrary: by way of example only, database 1120a may reside on storage media local to (and / or residing in) server 1115a (and / or user computer, user device, or client device 1105). Alternatively, database 1120n may reside remotely to any or all computers 1105, 1115, as long as it can communicate with one or more of these (e.g., via network 1110). In a particular set of embodiments, database 1120 may reside in a storage area network (“SAN”) familiar to those skilled in the art. (Similarly, any necessary files for performing functions belonging to computers 1105, 1115 may be stored locally on the respective computers and / or remotely, as appropriate.) In one set of embodiments, database 1120 may be a relational database, such as an Oracle database, suitable for storing, updating, and retrieving data in response to commands in SQL format. For example, as described above, the database may be controlled and / or maintained by a database server.

[0239] According to some embodiments, system 1100 may also include computing system 1125 (similar to...). Figure 1 The computing system 105a, etc.) and the corresponding database 1130 (similar to Figure 1 Database 110a, etc.). System 1100 may also include display device 1135 (similar to...). Figure 1 Display device 120, etc.), which allows user 1140 to view an optical view of a first biological sample displayed on display device 1135 (e.g., such as...). Figures 3A-3E (As shown in the user interface, etc.). User 1140 can use one or more user devices 1145 (similar to...). Figure 1 User equipment 130, etc.; including but not limited to smartphones, mobile phones, tablet computers, laptop computers, desktop computers, keyboards, keypads, computer mice, or monitors and / or the like). In some embodiments, system 1100 may also include one or more audio sensors 1150 (optional; similar to...). Figure 1 Audio sensor 135, etc.; including but not limited to one or more microphones, one or more voice recorders, or one or more recorders and / or similar devices), camera 1155 (optional; similar to Figure 1 Camera 140, etc.; including but not limited to one or more eye-tracking sensors, one or more motion sensors, or one or more tracking sensors and / or the like) and microscope 1160 (optional; similar to Figure 1(e.g., microscope 1145). In some cases, when a user views the FOV of the first biological sample on display device 1135 or through the eyepiece of microscope 1160, audio sensor 1150 can be used to record the voice or verbal annotations of user 1140. When a user views the FOV of the first biological sample on display device 1135 or through the eyepiece of microscope 1160, and when user 1140 is within the field of view (“FOV”) 1155a of camera 1155, camera 1155 can capture an image of user 1140 (in some cases, an image of at least one eye of user 1140). In some cases, two or more of computing system 1125, database 1130, display device 1135, user device 1145, audio sensor 1150 (optional), camera 1155 (optional), and / or microscope 1160 (optional) can be placed in working environment 1165, which may include, but is not limited to, at least one of laboratory, clinic, medical facility, research facility, healthcare facility, or room and / or the like.

[0240] As an alternative to or addition to computing system 1125 and corresponding database 1130, system 1100 may also include remote computing system 1170 (similar to...). Figure 1 Remote computing system 105b, etc.) and corresponding database 1175 (similar to Figure 1 (e.g., database 110b). In some embodiments, system 1100 may also include an artificial intelligence (“AI”) system 1180. In some embodiments, computing systems 1125 and / or 1170 may include, but are not limited to, a computing system located in the work environment, a remote computing system located outside the work environment and accessible via a network, a web server, a web browser, or a cloud computing system and / or the like. According to some embodiments, AI system 1180 may include, but is not limited to, at least one of machine learning systems, deep learning systems, neural networks, convolutional neural networks (“CNN”) or fully convolutional neural networks (“FCN”) and / or the like.

[0241] In operation, computing system 1125, remote computing system 1170, and / or AI system 1180 (collectively referred to as the "computing system," etc.) may perform data augmentation on a first image and a second image (optionally), the first image including the field of view ("FOV") of a first biological sample, and the second image including markers of instances of objects of interest in the first biological sample. In some cases, the first biological sample may include, but is not limited to, human tissue samples, animal tissue samples, or plant tissue samples and / or the like, wherein the objects of interest may include, but are not limited to, at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures, or organ structures and / or the like. In some embodiments, data augmentation of the first image and the second image may include, but is not limited to, at least one of elasticity enhancement or color enhancement and / or the like (configured in some cases to facilitate instance segmentation).

[0242] The computing system can receive a (enhanced) first image and a (enhanced) second image. The computing system can train an AI system 1180 to generate or update an AI model to predict instances of an object of interest, at least in part, based on multiple sets of at least two images generated from the second image, each of the at least two images in the multiple sets being distinct from each other. In some embodiments, the at least two images may include, but are not limited to, a centroid layer image that highlights the centroid of each labeled instance of the object of interest in the second image and a boundary layer image that highlights the edges or boundaries of each labeled instance of the object of interest in the second image. Alternatively, the at least two images may include, but are not limited to, at least a centroid layer image, a boundary layer image, and a semantic segmentation layer image, wherein the centroid layer image highlights the centroid of each labeled instance of the object of interest in the second image, the boundary layer image highlights the edges or boundaries of each labeled instance of the object of interest in the second image, and the semantic segmentation layer image includes semantic segmentation data for each labeled instance of the object of interest in the second image. In other alternative embodiments, the at least two images may include any number of images or surfaces that highlight different aspects of instances of the object of interest in the first biological sample.

[0243] In some embodiments, as part of training an AI system to generate or update an AI model to predict instances of an object of interest based at least in part on multiple sets of at least two images generated from a second image, the computing system may use an encoder (which may be part of the software and / or hardware of the computing system, or may be a separate device communicating with the computing system (in some cases, a dedicated encoder, etc.)) to encode the (enhanced) second image to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image. In some embodiments, encoding the second image to generate the third encoded image may include calculating the centroid of each marked instance of the object of interest in the second image using the computing system; and generating the third encoded image using the computing system, the third encoded image including a highlight of the centroid of each marked instance of the object of interest. In some cases, encoding the second image to generate the fourth encoded image may include calculating the edges or boundaries of each marked instance of the object of interest in the second image using the computing system; and generating the fourth encoded image using the computing system, the fourth encoded image including a highlight of the edges or boundaries of each marked instance of the object of interest.

[0244] According to some embodiments, encoding a second image to generate a third encoded image may further include a computing system calculating: a first distance metric between each pixel in the third encoded image and each centroid of each marker instance of the object of interest; and a first function for generating a first proximity map, the first function being a function of the first distance metric, the third encoded image including the first proximity map. Similarly, encoding a second image to generate a fourth encoded image may further include a computing system calculating: a second distance metric between each pixel in the fourth encoded image and the nearest edge pixel of an edge or boundary of each marker instance of the object of interest; and a second function for generating a second proximity map, the second function being a function of the second distance metric, the fourth encoded image including the second proximity map. In some cases, the computing system may assign a first weighted pixel value to each pixel in the third encoded image based at least in part on at least one of the first distance metric, the first function, or the first proximity map calculated for each pixel; and may assign a second weighted pixel value to each pixel in the fourth encoded image based at least in part on at least one of the second distance metric, the second function, or the second proximity map calculated for each pixel.

[0245] In some embodiments, the computing system can determine a first pixel loss value between each pixel in the third coded image and its corresponding pixel in the fifth image; and can determine a second pixel loss value between each pixel in the fourth coded image and its corresponding pixel in the sixth image. The computing system can calculate the loss value using a loss function based on the product of a first weighted pixel value of each pixel in the third coded image and the first pixel loss value between the corresponding pixel in the third coded image and the fifth image, and the product of a second weighted pixel value of each pixel in the fourth coded image and the second pixel loss value between the corresponding pixel in the fourth coded image and the sixth image. In some cases, the loss function may include, but is not limited to, one of the following: mean squared error loss function, mean square logarithmic error loss function, mean absolute error loss function, Huber loss function, or weighted sum of squared differences loss function and / or the like.

[0246] In some embodiments, the AI ​​system can update the AI ​​model by updating one or more parameters of the AI ​​model based on the calculated loss value. In some cases, one or more parameters may include, but are not limited to, a single parameter, multiple parameters between two and one hundred (inclusive), multiple parameters between one hundred and one thousand (inclusive), multiple parameters between one thousand and one million (inclusive), or more. The computational system can use the updated AI model to generate a fifth and a sixth image based on the first image.

[0247] In some cases, the labeling of instances of objects of interest in the second image may include, but is not limited to, at least one of the following: a complete annotation of the first instance of the object of interest that identifies the centroid and edges of the first instance of the object of interest; a partial annotation of the second instance of the object of interest that only identifies the centroid of the second instance of the object of interest; or an unknown annotation of the third instance of the object of interest that neither identifies the centroid nor the edges (i.e., otherwise represented as unknown); and / or the like. In some embodiments, the computing system may mask the second instance of the object of interest with partial annotations in corresponding pixels in the fourth and sixth images before calculating the loss value, without masking the second instance of the object of interest with partial annotations in the third or fifth images; and may mask the third instance of the object of interest with unknown annotations in corresponding pixels in the third and fifth images and corresponding pixels in the fourth and sixth images before calculating the loss value. In some cases, for partial annotations or unknown annotations, masking at least a portion of the second or third instance of the object of interest may include masking a circle in the third and / or fourth coded images, the circle representing the distance from the centroid or a point within the partially annotated object represented by user input (e.g., mouse click, etc.). In some cases, the radius of the circle may be predefined or may be calculated "on the fly" based on information from complete annotations of objects in the same region. Although circular masking is described, other polygonal or geometric shapes may be used appropriately or as needed. Alternatively, masking may include changing the weights of specific pixels in a third coded image and corresponding pixels in a fifth image (or specific pixels in a fourth coded image and corresponding pixels in a sixth image) to the same value so that they cancel each other out during pixel-by-pixel comparisons.

[0248] The computing system may use a decoder (which may be part of the computing system's software and / or hardware, or may be a separate device communicating with the computing system, etc. (in some cases, a dedicated decoder, etc.)) to decode the fifth and sixth images to generate a seventh image, the seventh image including predicted labels of instances of objects of interest in the first biological sample. In some cases, this is achieved by applying at least one of one morphological operations that identify foreground and background labels in each of the fifth and sixth images before generating the seventh image, or by directly decoding the fifth and sixth images to generate the seventh image. In some cases, applying at least one of one morphological operations or one of one or more machine learning operations may include applying one or more morphological operations, wherein after decoding the fifth and sixth images by applying one or more morphological operations that identify foreground and background labels in each of the fifth and sixth images, the computing system may apply a watershed algorithm to generate the seventh image. In some cases, one or more morphological operations may include, but are not limited to, at least one of the following: reconstruction open transformation or region H-minimum transformation and / or the like.

[0249] According to some embodiments, the computing system can compare a seventh image with a second image to generate an instance segmentation evaluation result. In some cases, generating the instance segmentation evaluation result may include evaluating instance segmentation performance using one or more metrics, which may include, but are not limited to, at least one of the following: Aggregate Jaccard Index (“AJI”), F1 score, DICE index, average DICE index, or joint DICE index and / or the like. In some cases, the instance segmentation evaluation result may include, but is not limited to, instance segmentation evaluation metrics, one or more numerical instance segmentation evaluation scores, or instance segmentation classifications (including, but not limited to, at least one of true positive (“TP”), true negative (“TN”), false positive (“FP”), false negative (“FN”), oversegmentation or undersegmentation, etc.) and / or the like. The computing system may display the generated instance segmentation evaluation result on a display screen.

[0250] In some cases, training an AI system to generate or update an AI model to predict instances of an object of interest, at least in part, based on multiple sets of at least two images generated from a second image, may include encoding the second image to generate a third and fourth coded image, training the AI ​​system to generate or update the AI ​​model to predict instances of an object of interest, at least in part, based on the third and fourth coded images, generating a fifth and sixth image, decoding the fifth and sixth images to generate a seventh image, and comparing the seventh image with the second image, etc. Although two images (in this case, the third and fourth coded images) are used to train the AI ​​system, various embodiments are not limited to this, and more than two images (or surfaces) may be used.

[0251] According to some embodiments, the computing system can receive an eighth image, which includes the field of view (FOV) of a second biological sample different from that of the first biological sample; two or more images, distinct from each other, can be generated based on the eighth image using an AI model generated or updated by a trained AI system; and the two or more images can be decoded using a decoder to generate a ninth image, which includes predicted labels of instances of objects of interest in the second biological sample. In this way, a trained AI system and / or AI model can be used to predict labels of instances of objects of interest in new biological samples—in some cases where there is no ground-based image (or previously user-annotated image) corresponding to the new biological sample.

[0252] Alternatively or additionally, the computational system may generate a user interface configured to collect training data using at least one of complete or partial annotations of objects of interest within an image of a biological sample, and may display a first image including the FOV of the first biological sample within a display portion of the user interface. The computational system may receive first user input via the user interface from a user (e.g., a pathologist, clinician, physician, nurse, or laboratory technician), indicating the presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface. The computational system may generate a boundary around each of the first plurality of objects of interest based at least in part on the location of each of the first plurality of objects within the first image identified by the first user input and at least in part on analyzing pixels in or around the corresponding location using algorithms (which may include, but are not limited to, object detection algorithms, pixel recognition algorithms, edge detection algorithms, and / or the like).

[0253] In some cases, the computing system may receive second user input from a user via a user interface, indicating a movement of a point within one of a first plurality of objects of interest (OPIs) from a previous position to a new position within the first image. The system may generate a new boundary around the one of the first PIIs contained in the first image, displayed in the user interface, based at least in part on the new position of the point within the one of the first PIIs in the first image, as indicated by the second user input, and at least in part on the analysis of pixels in or around the new position of the point within the one of the first PIIs using an algorithm. This new boundary replaces the boundary previously generated around the one of the first PIIs. In some cases, the computing system may receive third user input from a user via a user interface, indicating a partial annotation of one of a second plurality of PIIs contained in the first image, displayed in the user interface. The system may generate a partial annotation symbol in the first image, based at least in part on the position of the third user input within the first image. This partial annotation symbol identifies the location of the unbounded centroid of one of the second PIIs. In some cases, the computing system may receive a fourth user input from a user via a user interface, which indicates that one of the third plurality of objects of interest is unknown or that the instance class of one of the third plurality of objects of interest should be switched to another instance class (e.g., cancer, benign, etc.). The system may generate an unknown annotation symbol (i.e., a symbol or annotation representing an unknown instance or object, etc.) in the first image based at least in part on the position of the fourth user input within the first image, identifying the location of the unknown object represented by the fourth user input. Alternatively, the system may switch the instance class of the selected object among the third plurality of objects of interest to another instance class selected by the fourth user input (e.g., switching between cancer and benign, switching between full annotation and partial annotation, switching between partial annotation and unknown annotation, switching between full annotation and unknown annotation, etc.).

[0254] According to some embodiments, the first user input may include, but is not limited to, a click input or a boundary region input. In some cases, the click input may define the location of the centroid of a first object among a first plurality of objects of interest identified by the click input, while the boundary region input may define a region within a first image that marks the outer boundary of a second object among a first plurality of objects of interest identified by the boundary region input. In some cases, the boundary region input may include, but is not limited to, a rectangular boundary region input, a circular boundary region input, a polygon placement input, or a line placement input and / or the like. In some embodiments, the second user input may include, but is not limited to, click and drag input. In some cases, the third user input may include, but is not limited to, a double-click input, wherein the third user input selects or deselects the boundary around one of the second plurality of objects of interest. In some cases, the fourth user input may include, but is not limited to, a shift plus mouse click input or a key plus mouse click input, wherein the fourth user input may include, but is not limited to, switching between full annotations and unknown annotations or switching between instance classes from a list of instance classes. However, various embodiments are not limited to these specific inputs, and these inputs may be any suitable input for indicating full annotations, partial annotations, and / or unknown annotations, etc.

[0255] The computing system can generate at least one of a second image or an annotation dataset based on a first image. The second image includes data about the location of each of the first plurality of objects of interest within the first image, based on the boundaries surrounding each of the first plurality of objects of interest identified by the received first user input and the generated data. The annotation dataset includes at least one of pixel location data or coordinate data for each of the first plurality of objects within the first image, based on the boundaries surrounding each of the first plurality of objects of interest identified by the received first user input and the generated data. In this way, the system provides a fast and efficient UI that allows users (or annotators) to generate annotations effectively. In particular, users do not need to open any menus or follow a complex set of operations to interact with the annotation system's UI. A complete annotation (i.e., generating boundaries around the location marked by the click input or boundary area input, etc.) can be generated through a single operation (i.e., using click input or boundary area input, etc.). To change the automatically generated boundaries, the user only needs to move a point within the instance or object using a single operation (i.e., using click-drag input, etc.), causing the system to redraw or regenerate a new boundary around the instance or object. Therefore, users don't need to waste time manually drawing around the edges or boundaries of instances or objects to obtain full annotations. Similarly, a full annotation can be changed to a partial annotation, or the class of an instance or object can be changed, through a single action (i.e., shift plus mouse click, key plus mouse click, or mouse / keyboard combination, etc.). Actions are not bound to specific mouse / keyboard operations; instead, any combination can be used or customized appropriately or as needed.

[0256] In some embodiments, the computing system may train the AI ​​system 1180 to generate or update an AI model to predict instances of objects of interest in a first biological sample based at least in part on multiple sets of at least two images generated from at least one of a second image or annotation dataset, each of the at least two images in the multiple sets of at least two images being distinct from one another. In some cases, training an AI system to generate or update an AI model to predict instances of an object of interest based at least in part on at least two images may include: encoding at least one of the second images or annotated datasets using a computing system and an encoder (which may be part of the software and / or hardware of the computing system, or may be a separate device communicating with the computing system, etc. (in some cases, a dedicated encoder, etc.)) to generate a third coded image and a fourth coded image, the fourth coded image being different from the third coded image; training the AI ​​system to generate or update an AI model to predict instances of an object of interest based at least in part on the third coded image and the fourth coded image; using the AI ​​model generated or updated by the AI ​​system to generate a fifth and a sixth image based on the first image and based on training, the sixth image being different from the fifth image; decoding the fifth and sixth images using a computing system and a decoder (which may be part of the software and / or hardware of the computing system, or may be a separate device communicating with the computing system, etc. (in some cases, a dedicated decoder, etc.)) to generate a seventh image, the seventh image including a predicted label of an instance of an object of interest in the first biological sample; and (optionally) comparing the seventh image with the second image using a computing system to generate an instance segmentation evaluation result. The encoding of the second image and the training of the AI ​​system 1180 can also be done as described above. Figure 9B To achieve as described.

[0257] These and other functions of System 1100 (and its components) are described above regarding Figure 1 Figure 9 describes the scene in more detail.

[0258] While certain features and aspects have been described with respect to exemplary embodiments, those skilled in the art will recognize that many modifications are possible. For example, the methods and processes described herein can be implemented using hardware components, software components, and / or any combination thereof. Furthermore, while the various methods and processes described herein may be described with reference to specific structural and / or functional components for ease of description, the methods provided in the various embodiments are not limited to any particular structural and / or functional architecture, but can be implemented on any suitable hardware, firmware, and / or software configuration. Similarly, although some functionality is attributed to certain system components, unless the context otherwise indicates, that functionality may be distributed across a variety of other system components according to several embodiments.

[0259] Furthermore, although the methods and processes described herein are presented in a specific order for ease of description, various processes may be reordered, added, and / or omitted according to various embodiments unless the context otherwise specifies. Moreover, processes described with respect to a method or process may be incorporated into other described methods or processes; similarly, system components described according to a particular architectural architecture and / or with respect to a system may be organized in alternative architectural architectures and / or incorporated into other described systems. Therefore, while various embodiments with or without certain features have been described for ease of description and illustration of exemplary aspects of those embodiments, various components and / or features described herein with respect to particular embodiments may be replaced, added, and / or subtracted from other described embodiments unless the context otherwise specifies. Therefore, although several exemplary embodiments have been described above, it should be understood that the invention is intended to cover all modifications and equivalents within the scope of the appended claims.

Claims

1. A method comprising: A computing system generates a user interface configured to collect training data using a mixture of full or partial annotations of objects of interest within images of biological samples, for use in generating or updating an artificial intelligence model to predict instances of objects of interest. The computing system displays a first image, including the field of view ("FOV") of the first biological sample, within the display portion of the user interface; The computing system receives first user input from the user via the user interface, the first user input indicating the presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface; The computing system generates, at least in part, a boundary around each of the first plurality of objects of interest (OPIs) based on the location of each of the first plurality of OPIs in the first image identified by the first user input and at least in part based on the analysis of pixels in or around the corresponding location using an algorithm, and thus generates a complete annotation for each of the first plurality of OPIs, the complete annotation identifying the centroid and edge of each of the first plurality of OPIs; Using the computing system and receiving second user input from the user via the user interface, the second user input indicates a partial annotation of one of a second plurality of objects of interest contained within the first image displayed in the display portion of the user interface; The computing system generates partial annotations in the first image based at least in part on the location of the second user input within the first image, the partial annotations identifying the location of the unbounded centroid of one of the second plurality of objects of interest; and The computing system generates at least one of a second image or an annotation dataset based on the first image, wherein... The second image includes: Data regarding the following: the position of each of the first plurality of objects of interest (OPIs) within the first image, based on the received first user input, and the generated boundaries surrounding each of the first plurality of OPIs identified by the received fully annotated first user input; and Data regarding the following: the position of each of the second plurality of objects of interest within the first image, based on the second user input annotated with the received instruction portion; The annotation dataset includes: At least one of the following: pixel location data or coordinate data of each of the first plurality of objects of interest within the first image, based on the first user input, and the generated boundary around each of the first plurality of objects of interest identified by the received first user input indicating full annotation; and At least one of the following: pixel location data or coordinate data of each of the second plurality of objects of interest within the first image, based on the second user input of the indicated portion annotation.

2. The method of claim 1, wherein the computing system comprises one of a computing system disposed in the work environment, a remote computing system disposed outside the work environment and accessible via a network, a web server, a web browser, or a cloud computing system, wherein the work environment comprises at least one of a laboratory, clinic, medical facility, research facility, healthcare facility, or room.

3. The method according to claim 1, wherein the first biological sample includes one of human tissue sample, animal tissue sample or plant tissue sample, and wherein the object of interest includes at least one of normal cells, abnormal cells, damaged cells, cancer cells, tumors, subcellular structures or organ structures.

4. The method according to claim 1, further comprising: The computing system receives third user input from the user via the user interface, the third user input indicating the movement of a point within one of the first plurality of objects of interest from a previous position to a new position within the first image; and Using the computing system, at least in part based on the new position of a point within one of the first plurality of objects of interest in the first image represented by the third user input, and at least in part based on analyzing pixels in or around the new position of the point within the one of the first plurality of objects of interest using the algorithm, a new boundary is generated around the one of the first plurality of objects of interest contained in the first image displayed in the display portion of the user interface, the new boundary replacing the boundary previously generated around the one of the first plurality of objects of interest.

5. The method according to claim 4, further comprising: Using the computing system and via the user interface, a fourth user input is received from the user, the fourth user input indicating that one of the third plurality of objects of interest is unknown or that the instance class of one of the third plurality of objects of interest should be switched to another instance class; and The computing system generates, at least in part, an unknown annotation symbol in the first image that identifies the location of an unknown object represented by the fourth user input, based on the location of the fourth user input within the first image, or the computing system switches the instance class of a selected object of interest from the third plurality of objects of interest to another instance class selected by the fourth user input.

6. The method of claim 5, wherein the first user input includes one of a click input or a boundary region input, wherein the click input defines the position of the centroid of a first object among the first plurality of objects of interest identified by the click input, wherein the boundary region input defines a region within the first image that marks the outer boundary of a second object among the first plurality of objects of interest identified by the boundary region input, wherein the boundary region input includes one of a rectangular boundary region input, a circular boundary region input, a polygon placement input, or a line placement input, wherein the third user input includes a click and drag input, wherein the second user input includes a double-click input, wherein the second user input includes selecting or deselecting the boundary around one of the second plurality of objects of interest, wherein the fourth user input includes one of a shift plus mouse click input or a key plus mouse click input, wherein the fourth user input includes one of a conversion between a complete annotation and an unknown annotation or a switch between instance classes from a list of instance classes.

7. The method according to claim 1, further comprising: Training an artificial intelligence (AI) system to generate or update an AI model to predict instances of objects of interest in a first biological sample, at least in part, based on at least two images generated from multiple sets of at least two images from the second image or the annotation dataset, wherein each of at least two images in the multiple sets of at least two images is distinct from the others, wherein training the AI ​​system to generate or update the AI ​​model to predict instances of objects of interest, at least in part, based on the multiple sets of at least two images includes: The computing system and an encoder are used to encode at least one of the second image or the annotation dataset to generate a third encoded image and a fourth encoded image, the fourth encoded image being different from the third encoded image; The AI ​​system is trained to generate or update the AI ​​model to predict instances of objects of interest, at least in part, based on the third coded image and the fourth coded image. A fifth image and a sixth image are generated based on the first image using the AI ​​model generated or updated by the AI ​​system, wherein the sixth image is different from the fifth image; and The computing system and the decoder are used to decode the fifth and sixth images to generate a seventh image, which includes predicted labels of instances of objects of interest in the first biological sample.

8. The method of claim 7, wherein the AI ​​system comprises at least one of a machine learning system, a deep learning system, a neural network, a convolutional neural network ("CNN"), or a fully convolutional network ("FCN").

9. The method of claim 7, wherein training the AI ​​system to generate or update the AI ​​model to predict instances of an object of interest based at least in part on the plurality of sets of at least two images further comprises: The computing system is used to compare the seventh image with the second image to generate an instance segmentation evaluation result.

10. The method of claim 7, wherein the third encoded image comprises the centroid of each of the first plurality of objects of interest based on the first user input, and wherein the fourth encoded image comprises the boundary of each of the generated first plurality of objects of interest.

11. The method of claim 10, wherein: Encoding the second image to generate the third encoded image includes: The computing system is used to calculate a first distance metric between each pixel in the third encoded image and each centroid of each of the first plurality of objects of interest; and The computing system computes a first function to generate a first proximity map, the first function being a function of the first distance metric, and the third coded image includes the first proximity map; and Encoding the second image to generate the fourth encoded image includes: The computing system calculates a second distance metric between each pixel in the fourth coded image and the nearest edge pixel of the edge or boundary of each of the first plurality of objects of interest; and The computing system calculates a second function to generate a second proximity map, the second function being a function of the second distance metric, and the fourth encoded image includes the second proximity map.

12. The method of claim 11, further comprising: The computing system assigns a first weighted pixel value to each pixel in the third coded image based at least in part on at least one of the first distance metric, the first function, or the first proximity map for each pixel. and The computing system assigns a second weighted pixel value to each pixel in the fourth coded image based at least in part on at least one of the second distance metric, the second function, or the second proximity map for each pixel.

13. The method of claim 11, further comprising: The computing system is used to determine a first pixel loss value between each pixel in the third coded image and the corresponding pixel in the fifth image; The computing system is used to determine a second pixel loss value between each pixel in the fourth coded image and the corresponding pixel in the sixth image; The computational system uses a loss function to calculate a loss value based on the product of a first weighted pixel value of each pixel in the third encoded image and a first pixel loss value between each pixel in the third encoded image and a corresponding pixel in the fifth image, and the product of a second weighted pixel value of each pixel in the fourth encoded image and a corresponding pixel loss value between each pixel in the sixth image. The loss function includes one of the following: a mean squared error loss function, a mean logarithmic error loss function, a mean absolute error loss function, a Huber loss function, or a weighted sum of squared differences loss function. The AI ​​system updates the AI ​​model by updating one or more parameters of the AI ​​model based on a calculated loss value; Generating the fifth and sixth images includes generating the fifth and sixth images based on the first image using the updated AI model.

14. The method of claim 7, wherein decoding the fifth image and the sixth image to generate the seventh image comprises using the computing system and the decoder, by applying at least one of one or more morphological operations that identify foreground and background markers of each of the fifth image and the sixth image before generating the seventh image, or by directly decoding the fifth image and the sixth image to generate the seventh image.

15. The method of claim 14, wherein applying at least one of the one or more morphological operations or the one or more machine learning operations includes applying the one or more morphological operations, wherein the method further comprises: After decoding the fifth and sixth images by applying one or more morphological operations to identify foreground and background markers in each of the fifth and sixth images, the watershed algorithm is applied to generate the seventh image.

16. A system comprising: Computing systems, including: At least one first processor; and A first non-transitory computer-readable medium communicatively coupled to the at least one first processor, the first non-transitory computer-readable medium having computer software stored thereon, the computer software including a first set of instructions, which, when executed by the at least one first processor, cause the computing system to: Generate a user interface configured to collect training data using a mixture of full or partial annotations of objects of interest within images of biological samples, for use in generating or updating an artificial intelligence model to predict instances of objects of interest; A first image, including the field of view ("FOV") of the first biological sample, is displayed within the display portion of the user interface. The user receives first user input from the user via the user interface, the first user input indicating the presence or location of each of a first plurality of objects of interest contained within the first image displayed in the display portion of the user interface; Based at least in part on the position of each of the first plurality of objects of interest in the first image identified by the first user input and at least in part on the analysis of pixels in or around the corresponding position using an algorithm, a boundary around each of the first plurality of objects of interest is generated, and thus a complete annotation of each of the first plurality of objects of interest is generated, the complete annotation identifying the centroid and edge of each of the first plurality of objects of interest; The second user input is received from the user via the user interface, the second user input indicating a partial annotation of one of a second plurality of objects of interest contained within the first image displayed in the display portion of the user interface; At least partially based on the location of the second user input within the first image, a partial annotation is generated in the first image, the partial annotation identifying the location of the unbounded centroid of one of the second plurality of objects of interest; and Generate at least one of a second image or an annotation dataset based on the first image, wherein... The second image includes: Data regarding the following: the position of each of the first plurality of objects of interest (OPIs) within the first image, based on the received first user input, and the generated boundaries surrounding each of the first plurality of OPIs identified by the received fully annotated first user input; and Data regarding the following: the position of each of the second plurality of objects of interest within the first image, based on the second user input annotated with the received instruction portion; The annotation dataset includes: At least one of the following: pixel location data or coordinate data of each of the first plurality of objects of interest within the first image, based on the first user input, and the generated boundary around each of the first plurality of objects of interest identified by the received first user input indicating full annotation; and At least one of the following: pixel location data or coordinate data of each of the second plurality of objects of interest within the first image, based on the second user input of the indicated portion annotation.

17. The system of claim 16, wherein the computing system comprises one of a computing system disposed in the work environment, a remote computing system disposed outside the work environment and accessible via a network, a web server, a web browser, or a cloud computing system, wherein the work environment comprises at least one of a laboratory, clinic, medical facility, research facility, healthcare facility, or room.

18. The system of claim 16, wherein the first biological sample comprises one of a human tissue sample, an animal tissue sample, or a plant tissue sample, and wherein the object of interest comprises at least one of a normal cell, an abnormal cell, a damaged cell, a cancer cell, a tumor, a subcellular structure, or an organ structure.

19. The system of claim 16, wherein the first user input includes one of a click input or a boundary region input, wherein the click input defines the position of the centroid of a first object among the first plurality of objects of interest identified by the click input, wherein the boundary region input defines a region within the first image that marks the outer boundary of the boundary of at least one second object among the first plurality of objects of interest identified by the boundary region input, wherein the boundary region input includes one of a rectangular boundary region input, a circular boundary region input, a polygon placement input, or a line placement input.

Images