Image analysis system and method using the same

Abstract

A system and method for analyzing an image using a non-transitory programmable device is disclosed. A user interface generator operating on the non-transitory programmable device receives a first image of a biological sample and a specification of a first image analysis step. A sequencer operating on the non-transitory programmable device applies the first image analysis step to the first image to develop annotated training data, and a machine learning system trainer operating on the non-transitory programmable device trains an untrained machine learning system to develop a trained machine learning system. When the first image is presented as input to the trained machine learning system, the trained machine learning system develops predictions of the annotated training data. In addition, the sequencer receives a specification of a workflow including a second image analysis step, the second image analysis step specifies operating the trained machine learning system, and the sequencer executes the workflow on a second image to analyze the second image.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Application No. 16 / 254,823, filed on January 23, 2019, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0003] The presently disclosed subject matter relates to image analysis systems for analyzing biological sample images, and more particularly, to incorporating machine learning systems in such image analysis systems. Background Art

[0004] The imaging system can be used to obtain microscopic images of biological samples. Such images can be processed using an image analysis system to identify pixels of the image associated with objects of interest (cells, organelles, etc.) therein, to classify different types of objects of interest represented in the image, to obtain indicators about the objects of interest or parts thereof, etc. The indicators can include, for example, counts of objects of interest, counts of each type of object of interest represented in the biological sample, the size of cells (area, perimeter, volume), statistics of the size of objects of interest, etc. (mean, mode, median, standard deviation, etc.).

[0005] Some image analysis systems allow an operator to interactively specify a workflow for analyzing one or more images of a biological sample to create an output image. Each pixel of the output image is associated with a corresponding pixel of the image of the biological sample, and the pixel value of the output image indicates whether the corresponding pixel of the image of the biological sample is associated with an object of interest, such as a cell, a component of a particular type of cell, a cell of a particular type, or a cell having certain characteristics, etc.

[0006] The workflow specified using the image analysis system includes a sequence of image analysis steps applied to an image of a biological sample or an image generated from a previous image analysis step, and parameters for the image analysis steps. Typically, one or more images of an exemplary biological sample are used to define a sequence of image analysis steps including the workflow and its parameters. For example, the operator selects one or more images acquired from the imaging system or obtained from performing a previous image analysis step. The operator also selects an image analysis step from an available image analysis step library to perform using the selected image, and iteratively adjusts the parameters associated with the selected image analysis step while viewing the effect of such adjustment on the output of the image analysis step. Thereafter, if such adjustment of the parameters meets the operator's desired output, the operator adds the selected image analysis step and the parameters associated therewith to the workflow. Otherwise, the operator can select a different image analysis step or terminate the creation of the workflow. The operator continues to develop the workflow in this manner until the output of the workflow is satisfactory. After the operator has selected all image analysis steps and the sequence in which these steps are performed, the image analysis system can apply the workflow to one or more additional images to analyze these images.

[0007] Some of the image analysis steps comprising the workflow are associated with the acquisition of one or more images of a biological sample under specific conditions, for example, using specific lighting conditions during fluorescence, using one or more filters between a light emitter and a biological sample, using one or more filters between a biological sample and an image capture device, using a specific objective lens, etc. Some of the image analysis steps may include using image processing operations to process the acquired (multiple) images and / or (multiple) images generated as a result of performing another image analysis step. Such image processing operations may include applying one or more image processing filters to one or more images, such as a blur masking filter, a threshold filter, an edge detection filter, a smoothing filter, etc. The image processing operations specified in the workflow may also include generating images by combining one or more captured images with each other, combining one or more captured images with one or more images generated by a previous image processing operation, etc. The image processing operations may also include measurement operations for counting different types of objects (e.g., cells or parts thereof) in an image, and measuring indicators associated with the objects as described above.

[0008] Such an image analysis system is disclosed in U.S. Pat. No. 8,577,079 to Cohen et al., entitled "Image Processing System Providing Selective Arrangement and Configuration for Image Analysis Sequences". In addition, U.S. Pat. No. 9,928,403 to Cohen et al., entitled "System and Method for Image Analysis of Multidimensional Data", discloses another such image analysis system for specifying and applying workflows that can be used for multidimensional images. The entire contents of these two patents are incorporated herein by reference.

[0009] It is useful to incorporate a trained machine learning system that has been trained to analyze one or more images into a workflow. However, such a trained machine learning system must analyze images of a specific type of biological sample for specific features, such as, for example, pixels associated with the cells or their organelles represented therein, cell types, etc. A machine learning system trained to analyze one type of biological sample may not be useful for analyzing another type of biological sample. In addition, the training of a machine learning system typically requires expertise in artificial intelligence and / or machine learning, and therefore, it is not feasible to develop a variety of trained machine learning systems, each of which is trained to analyze a specific type of biological sample. It would be useful if the image analysis system included a feature that enabled an operator to easily train the machine learning system to analyze a specific type of biological sample for specific features and incorporate the trained machine learning system into a workflow developed using the image analysis system. Summary of the invention

[0010] According to one aspect, a system for analyzing an image includes a non-transitory programmable device and includes a user interface generator, a sequencer, and a machine learning system trainer operating on the non-transitory programmable device. The user interface generator receives a specification of a first image of a biological sample and a specification of a first image analysis step, and the sequencer applies the first image analysis step to the first image to develop annotated training data. The machine learning system trainer trains an untrained machine learning system to develop a trained machine learning system, wherein when the trained machine learning system takes the first image as input, the trained machine learning system develops predictions of the annotated training data. In addition, the sequencer receives a specification of a workflow including a second image analysis step, the second image analysis step specifies operating the trained machine learning system, and the sequencer executes the workflow on a second image to analyze the second image.

[0011] According to another aspect, a method for analyzing an image includes operating a non-transitory programmable device to perform the steps of receiving a designation of a first image of a biological sample and a designation of a first image analysis step. The non-transitory programmable device performs the additional steps of generating annotated training data by applying the first image analysis step to the first image, and training an untrained machine learning system to develop a trained machine learning system. When the trained machine learning system takes the first image as input, the trained machine learning system develops predictions of the annotated training data. The non-transitory programmable device performs the additional step of receiving a designation of a workflow including a second image analysis step, wherein the second analysis step includes operating the trained machine learning system and executing the workflow on a second image to analyze the second image.

[0012] Other aspects and advantages will become apparent upon consideration of the following detailed description and accompanying drawings, wherein like reference numerals refer to like structures throughout the specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A block diagram showing an image analysis system;

[0014] Figure 2 Shows how operators can use Figure 1 an image analysis system to create and use a flowchart of an image analysis workflow including a machine learning system;

[0015] Figure 3 Show Figure 1 Graphical user interface for image analysis systems;

[0016] Figure 3A Show Figure 3 The panel of the graphical user interface shown;

[0017] Figure 4 Show Figure 1 A flowchart of the operation of an image analysis system for creating a workflow;

[0018] Figure 5 Show Figure 1 another flow chart of the operation of an image analysis system for creating a workflow;

[0019] Figure 6 Show Figure 1 A flowchart of the operation of an image editor of an image analysis system;

[0020] Figure 7 Shows the Figure 3 Graphical user interface;

[0021] Fig. 7A Shown in Figure 7 The graphical user interface depicts the image before being edited;

[0022] Figure 8 Show Figure 1 A flowchart of the operation of a machine learning system trainer of an image analysis system;

[0023] Fig. 8A Show Figure 3 The graphical user interface panel shown is used to operate Figure 1 A machine learning system trainer for an image analysis system;

[0024] Fig. 9 Show Figure 1 A flowchart of the operation details of the machine learning system trainer of the image analysis system; and

[0025] Fig.10 Show use Figure 1 Flowchart of the operational details of the machine learning system trainer of the image analysis system to retrain the machine learning system. DETAILED DESCRIPTION

[0026] For clarity, in this disclosure, image analysis steps, including the training and use of machine learning systems, are described as operating on one image and producing another image. However, it will be apparent to one of ordinary skill in the art that some image analysis steps operate on multiple input images and produce one output image, other image analysis steps operate on one input to produce multiple output images, and still other image analysis steps operate on multiple input images and produce multiple output images. Furthermore, in some embodiments, measurements may be made from an input image or from an image produced by applying one or more image analysis steps, and the measurements may be provided as input to additional image analysis steps.

[0027] As described in detail below, an operator (e.g., a user of the image analysis system) can interactively use the image analysis system to train a machine learning system to analyze images of biological samples, and then specify use of the trained machine learning system in a workflow in conjunction with one or more image analysis steps to analyze additional images of other biological samples, such as images of biological samples acquired by an imaging system.

[0028] The operator can also use the image analysis system to train the machine learning system, and then incorporate the operation of the trained machine learning system into one of the image analysis steps in the workflow. Specifically, as described in more detail below, the operator selects an example input image, applies one or more image analysis steps to the input image to develop annotated training data, and trains the machine learning system to generate predictions of the annotated training data when the example input image is used as input. Typically, the annotated data is a segmented image, a classified image, or an image derived from one or both of the segmented and classified images. The annotated data includes an output value corresponding to each pixel of the input image.

[0029] If the annotated data is a segmented image, the output value indicates whether the corresponding pixel of the input image is associated with an object of interest, such as a cell, a feature of a cell, a specific type of cell, etc. The value of the annotated data corresponding to the pixel of the input image associated with the object of interest is set to a predetermined value associated with the foreground pixel, and the value of the annotated data corresponding to the pixel of the input image not associated with the object of interest is set to a predetermined value associated with the background pixel.

[0030] If the annotated data is a classified image, the output value associated with the pixel of the input image indicates the class of the part of the biological sample represented by the pixel. The class can be, for example, a cell type, an organelle type, a cell condition, etc. The cell condition can include, for example, whether the cell is alive or dead, whether the cell is undergoing cell division, or a specific state of the cell division. Each class can be associated with a specific pixel value. The value of the annotated data corresponding to the pixel of the input image associated with a specific class is set to a specific value. The value of the annotated data corresponding to the pixel of the input image not associated with any of the classes is set to a predetermined value associated with the background pixel.

[0031] As described above, annotated data can be derived from a segmented and / or classified image. For example, in one embodiment, a segmented image can be processed to identify the coordinates of the centroid of one or more groups of consecutive pixels having values ​​associated with foreground pixels (e.g., the centroid of each blob in a segmented image, biological cell, organelle, etc.). The value of the annotated data associated with the coordinates is set to a first predetermined value, while the value of the annotated data not associated with such coordinates is set to a second predetermined value. In some embodiments, the first predetermined value is equal to the value associated with the foreground pixel in the segmented image, and the second predetermined value is equal to the value associated with the background pixel.

[0032] In some embodiments, annotated data may be developed by processing the segmented image, the annotated data including, for example, coordinates of one or more centers, one or more bounding boxes, regions, etc., of one or more objects of interest in the training image. In some embodiments, the annotated data may include, for example, an object score or confidence level associated with each object represented in the segmented image. The object score may indicate, for example, the confidence level that the object associated therewith matches a particular criterion, such as, for example, a particular shape, a particular intensity of a pixel in a source image associated with a particular class of objects, etc. In some embodiments, the annotated data training data may be a table having one entry for each object identified in the segmented image. Each such entry may include, for example, an object identifier and one or more characteristics or values ​​associated with the entry (e.g., the coordinates described above, an object score, etc.). For example, it may be advantageous to develop the annotated data as a table rather than an image because a table may require less memory than an image or may be easier to analyze.

[0033] In some embodiments, the annotated training data may be viewed as images, and the values ​​of pixels of the images indicate whether the corresponding pixels of the input image are associated with an object of interest.

[0034] After the machine learning system is trained as described above, an additional image of a biological sample can be used as an input to the machine learning system, and in response, the trained machine learning system generates output values. In some embodiments, each output value corresponds to a pixel of the image of the additional biological sample and indicates whether the pixel of the image is associated with an object of interest.

[0035] In addition, a first type and a second type of image of the biological sample can be acquired separately, for example, a fluorescence image and a transmitted light image. Thereafter, annotated training data can be developed from the first type of image, and a machine learning system can be trained to accept the second type of image of the biological sample as input and generate annotated training data in response. The machine learning system trained in this manner can then be used to analyze additional biological samples using only the second type of image of the additional biological sample. One of ordinary skill in the art will appreciate that a machine learning system trained in this manner would be beneficial, for example, if only the second type of image of the biological sample is available, acquiring the first type of image is harmful or otherwise harmful to the biological sample, or acquiring the first type of image requires more complexity or cost than acquiring the second type of image.

[0036] refer to Figure 1The image analysis system 100 includes a user interface generator 102, an image acquirer 104, a sequencer 106, an image editor 108, an image processor 110, a machine learning system 112, a machine learning system trainer 114, and a measurement calculator 115. The image analysis system 100 also includes an image data store 116, a workflow data store 118, and a machine learning system parameter data store 120.

[0037] The user interface generator 102 is coupled to a user computer 122, and the image acquirer 104 is coupled to an imaging system 124. In some embodiments, the imaging system 124 may be, for example, a high-volume imaging system, a lensless or lens-based electron microscope, a manually operated microscope, or the like.

[0038] As described in more detail below, to develop a workflow, the user interface generator 102 instructs the user computer 122 to generate a graphical user interface (GUI) on its display that allows the user to specify image analysis steps and, if desired, parameters associated therewith, and view the results of the image analysis steps. It should be understood by those of ordinary skill in the art that when the user interface generator 102 is described herein as receiving information from the user computer 122 or an operator of the computer, the user interface generator 102 generates instructions and sends them to the user computer 122 that cause the GUI to obtain information from the user and send the information to the user interface generator 102. Similarly, when the user interface generator 102 is described as displaying information on the user computer 122, the user interface generator 102 generates instructions and sends them to the user computer 122 that cause the GUI to display the information.

[0039] Figure 2 A flowchart 150 is shown that illustrates an exemplary use of features of the image analysis system 100 in a workflow to create and use a trained machine learning system 112. Figure 2 As described in more detail below, at step 152 , the operator directs the sequencer 106 to start a new workflow via a GUI generated by the user interface generator.

[0040] At step 154, the operator specifies one or more image analysis operations, such as (multiple) image processing operations performed by the image processor 110 to develop first annotated data from one or more images of the biological sample. The biological sample can be a labeled (e.g., stained) or unlabeled biological sample, and the image can be captured using various wavelengths of light, when the biological sample is fluorescent, using transmitted light to capture the image, etc. The above-mentioned image processing operations can be applied to one or more of these images, and the results of applying these steps can be combined to create the first annotated data. In some embodiments, the first annotated data represents a binary segmented image, in which each pixel has a first predetermined value associated with an object of interest in the biological sample (e.g., a foreground pixel) or a second predetermined value that is not associated with the object of interest (e.g., a background pixel).

[0041] At step 156, optionally, the operator specifies one or more image analysis operations, such as (one or more) image processing operations performed by the image processor 110, to develop second annotated data of one or more images of the biological sample or intermediate images developed from one or more images of the biological sample, including, for example, the first annotated image. The one or more images of the biological sample may include those images used to create the first annotated data and / or additional images of the biological sample. In some embodiments, pixels of the second annotated data are associated with a classification, which is determined using, for example, numerical filtering and / or based on the number of neighboring pixels of the first annotated data having similar or identical values, the size of a bounding box of the neighboring pixels, the roundness or other shape indicator of the neighboring pixels, etc. A predetermined unique value (e.g., a color) can be associated with each classification, and the value of the pixels of the second annotation set is assigned according to its classification.

[0042] At step 158, the operator may edit the first annotated data using the image editor 108, for example, to change foreground pixels to background pixels or background pixels to foreground pixels. The operator may also edit the second annotated data using the image editor 108 to adjust classification values ​​associated with certain pixels, change foreground pixels to background pixels, and / or change background pixels to foreground pixels.

[0043] At step 160 , an operator identifies images of biological samples to use as training images.

[0044] At step 162, the operator uses the machine learning system trainer 114 to train an untrained machine learning system 112 using the training image, the first annotated data, and the second annotated data (if any). The machine learning system 112 is trained so that when the machine learning system 112 uses the training image as input, the machine learning system generates first prediction data of the first annotated data as output. If both the first annotated data and the second annotated data are used to train the machine learning system 112, when the training data is presented as input to the machine learning system, the machine learning system generates first and second prediction data of the first annotated data and the second annotated data, respectively. Each datum of the prediction data provides an indication or probability that the corresponding datum of the annotated data has a specific value.

[0045] At step 164, the operator uses the machine learning system trainer 114 to store the parameters associated with the trained machine learning system 112 in the machine learning system parameter data store 120 and associates the identifier with the parameters. Thereafter, the identifier can be used to incorporate the trained machine learning system 112 associated with the identifier into additional workflows.

[0046] At step 166 , the operator ends or closes the first workflow.

[0047] In some embodiments, the first workflow is associated with the identifier and saved in the workflow data store 118. The first workflow can then be called at a later time, if necessary, to update (e.g., expand or refine) the training of the machine learning system 112 or to train another instance of the untrained machine learning system 112.

[0048] In some embodiments, the training images selected by the operator at step 160 may include a sufficient number of objects (e.g., cells, organelles, etc.) to provide sufficient data to train the machine learning system 112. However, it will be apparent to one of ordinary skill in the art that the operator may select multiple training images at step 160; develop a first annotated dataset corresponding to each of the multiple training images at steps 154 to 158, and optionally develop a second annotated dataset; and train the machine learning system 112 at step 162 using the multiple images and the first annotated dataset and the optional second annotated dataset.

[0049] To use the trained machine learning system developed at step 162 , at step 168 , the operator uses sequencer 106 to create a second workflow.

[0050] At step 170 , the operator uses the sequencer 106 to add the operation of the trained machine learning system 112 as an image processing operation by selecting an identifier associated with the trained machine learning system from a menu of image processing operations provided by the image analysis system 100 .

[0051] At step 172, the operator adds any additional post-processing image analysis steps to the workflow using the sequencer 106. The post-processing image analysis steps may clean the output generated by the trained machine learning system and may include, for example, automatic border filling, edge detection, removal of spurious values, removal of all neighboring elements with values ​​that are related to the foreground but are too large or too small to be associated with the object of interest, removal of spurious content, etc.

[0052] At step 174, the operator uses the sequencer 106 to add any desired measurement steps to automatically develop one or more indicators from the feature image, such as a count of the number of cells or parts thereof represented in the image analyzed by the second workflow that have at least a predetermined probability of being associated with the feature. Automatically developing such indicator(s) may involve applying boundary or edge detection, boundary filling, and similar algorithms to the output generated by the trained machine learning system 112.

[0053] At step 176 , the operator uses the sequencer 106 to associate a name with the second workflow and save the second workflow in the workflow data store 118 .

[0054] If the first and / or second annotated dataset developed at steps 154 and / or 156 is a table of entries or a table of numerical data (e.g., coordinates of an object, a bounding box, etc.), the machine learning system 112 generates the table or numerical data, respectively, in response to an image presented as input.

[0055] Furthermore, it should be apparent to one of ordinary skill in the art that, selecting training data as a first image of a first biological sample acquired using a first imaging condition and annotated training data as a second image of the first biological sample acquired using a second imaging condition, the machine learning system 112 can be trained to generate a fourth image from a third image of a second biological sample acquired using the first imaging condition, wherein the fourth image is an estimate of an image of the second biological sample that would be acquired using the second imaging condition. Such first and second imaging conditions can include, for example, acquiring a transmitted light image using different illuminants, filters, and / or lenses; acquiring a fluorescence image using different illuminants, filters, and / or lenses; acquiring transmitted light or fluorescence images at different focal lengths, etc. For example, the machine learning system 112 can be trained to generate an estimate of a fluorescence image when presented with a transmitted light image as input, to generate an estimate of a transmitted light image when presented with a fluorescence image, to generate an estimate of a first fluorescence image acquired using a first illuminant when presented with a second fluorescence image acquired using a second illuminant, etc.

[0056] After the second workflow has been created and saved, the second workflow can be applied to one or more additional images to automatically perform the steps specified by the workflow, at step 178. For example, in some embodiments, an operator can use a graphical user interface to select one or more images stored in the image data store 116 and apply the second workflow to the images. In other embodiments, an operator can use a graphical user interface to direct the image analysis system 100 to automatically capture one or more images of a biological sample loaded in the imaging system 124 and apply the second workflow to the captured images. The data resulting from the application of the second workflow can be displayed to the user, stored in a data storage associated with the image analysis system 100, or transmitted to another system (not shown) for additional analysis.

[0057] Figure 3 and Figure 3A An embodiment of a GUI 200 generated by the user interface generator 102 on the user computer 122 is shown. Figure 3 and Figure 3A, GUI200 includes an area 202 in which a panel for specifying an image analysis step can be displayed and an area 204 in which one or more images can be displayed. In some embodiments, area 204 includes one or more sub-areas 205a, 205b, etc., so that multiple images can be displayed simultaneously. GUI200 also includes area 206, which includes one or more selectable icons 208, each of which icons 208a, 208b, ... 208, 208j is associated with an available image analysis step. In some embodiments, one or more icons 208 can be associated with a drop-down menu (not shown) that allows selection of a specific image analysis step. It is obvious to a person of ordinary skill in the art that GUI200 can be combined with a manner of selecting an image analysis step.

[0058] refer to Figure 3A , selecting icon 208 displays a corresponding panel 210a, 210b, 210c, etc. in area 202. Each panel is associated with a particular image analysis step. Each panel 210 includes one or more pop-up menus 212, which can be used to select the source(s) of any input data used by the image analysis step associated with the panel 210. Each pop-up menu 212 presents information related to the imaging system 124 ( Figure 1 ) for capturing an image of a biological sample loaded in the imaging system 124 or outputting data generated by a previously defined image analysis step.

[0059] Predetermined imaging configurations of the imaging system 124 include acquiring a transmitted light image of a biological sample, acquiring a biological sample under fluorescence, and using a specific combination of illumination conditions (e.g., wavelength of light), one or more lenses, one or more filters disposed in the optical path between an illumination source and the biological sample, one or more filters disposed in the optical path between the biological sample and an image capture device, and the like.

[0060] In some embodiments, the imaging system 100 is configured to associate a specific identifier, such as a DAPI, with a corresponding data source. Figure 3A In the example shown, the data sources DAPI and FITC are associated with images of biological samples acquired using illumination that shows DAPI and FITC staining, respectively, as will be apparent to one of ordinary skill in the art. Similarly, the identifier "transmitted light" is associated with a data source that is associated with images of biological samples captured using transmitted light illumination.

[0061] Each panel 210 may also include one or more input areas 214, where parameters used by the image analysis step associated with the panel may be entered, and input areas 216, where identifiers associated with outputs of the image analysis steps may be entered. The identifier entered into the input area 216 of a particular panel 210 is a name or label by which the output generated by the image analysis step associated with the particular panel 210 may be referenced for use in a subsequent image analysis step. For example, an operator may specify that the output of a particular panel 210 is to be used as an input or as a parameter of a panel 210 associated with a subsequent image analysis step by selecting an identifier associated with the output in a pop-up menu 212 or entering the identifier in the input area 214.

[0062] Some panels 210 include an "Apply" button 218 that, when selected, performs the image analysis steps associated with the panel and generates a result (e.g., an image) associated with the identifier 216 specified in the panel 210. Some panels 210 include an area 220 that, when selected, closes the panel 210 (e.g., removes the panel from area 202) and removes the image analysis steps associated with the panel from the workflow.

[0063] For example, panel 210a is associated with an image processing operation named "Count Kernel Objects" and allows a data source to be specified using pop-up menu 212a. Figure 3A In the example shown, a data source called "DAPI" is selected using pop-up menu 212a. Parameters are entered in input area 214a, which specify that the image analysis step associated with panel 210a should identify those consecutive pixels spanning between 10 and 30 pixels in the input image (e.g., from source DAPI), with pixel values ​​that are at least 500 intensity levels higher than adjacent pixels, and that the "FAST" algorithm should be used for such processing. The result (e.g., a binary image) will be associated with the identifier "core" (specified in input area 216A), wherein the value of the result associated with the pixels of the input image that meet the parameters specified in input area 214A is set to the value associated with the foreground pixel. In addition, the value of the result associated with the pixels of the input image that do not meet these parameters is set to the value associated with the background pixel. If the user selects the "Apply" button 218a and / or as part of a workflow that includes such an image processing step, the image analysis step specified by panel 210a can be interactively performed.

[0064] continue Figure 3 and Figure 3AIn the example shown in , panels 210a to 210e apply image analysis steps to DAPI and FITC images to generate results associated with identifiers "training candidates". As will be appreciated by those skilled in the art, the image analysis steps include applying the "counting nuclear objects" image analysis described above in conjunction with panel 210a, an adaptive threshold image analysis step (panel 210b), a watershed image analysis step (210c), a logical AND operation (210d), and an image analysis step (210e) that selects pixels based on the results of the "counting nuclear objects" and logical AND image analysis steps.

[0065] Panel 210f specifies that a user is allowed to manually edit an image associated with the identifier “training candidate,” as described in more detail below, to generate a result image associated with the identifier “training target.” Panel 210g specifies an image analysis step that uses such result image with a transmitted light image to train a machine learning system, as described in more detail below.

[0066] Reference again Figure 1 To develop a workflow, the user interface generator 102 receives a designation of an image analysis step from a user computer 122, the sequencer 106 coordinates the steps of the remaining components of the image analysis system 100 to perform the designated image analysis step, thereby developing the output of the image analysis step (if any), and directing the user interface generator 102 to display any such output on the user computer 122.

[0067] As described in more detail below, the user interface generator 102 and sequencer 106 allow the operator to specify image processing operations in this manner until the operator of the user computer 122 indicates that a workflow has been specified or that the workflow should be discarded. If the operator indicates that a workflow has been specified, the sequencer 106 records the image analysis steps and the parameters associated therewith in the workflow data store. In some embodiments, the operator provides a name associated with the workflow. In other embodiments, the sequencer 106 can generate a name to be associated with the workflow. Other ways of associating a name with a workflow will be apparent to one of ordinary skill in the art.

[0068] After a workflow has been developed and stored as described above, an operator can load a biological sample to be analyzed into the imaging system 124 and select a name associated with the workflow. In response, the sequencer 106 operates the other components of the image analysis system 100 to perform the image analysis steps specified by the workflow to generate results that are displayed by the user interface generator 102 on the user computer 122, recorded in a data store (not shown) associated with the image analysis system 100, or provided to another system (not shown) for further analysis.

[0069] Figure 4 A flowchart 250 is shown showing the steps performed by the image analysis system 100 to create a new workflow. Figures 1 to 4 At step 252, the user interface generator 102 generates an instruction and sends it to the user computer 122, so that the user computer 122 displays the above GUI 200. At step 254, the user interface generator 102 receives an instruction from the user computer 122 to start specifying a new workflow. For example, the operator of the user computer 122 can select GUI 200 ( Figure 3 ) button 222 to provide such instructions.

[0070] At step 256 , the user interface generator 102 receives a specification of an image analysis step as described above, including the image analysis step to be taken and any parameters associated with the step.

[0071] At step 258 , sequencer 106 performs the image analysis step received at step 256 and, at step 260 , directs user interface generator 102 to display any results of performing the step in area 204 of GUI 200 .

[0072] At step 262, the user interface generator 102 receives an indication from the user computer 122 whether the operator has changed one or more parameters specified at step 256. If the operator has changed one or more parameters, the user interface generator 102 proceeds to step 256 to receive the changed specifications for the image analysis steps.

[0073] Otherwise, at step 264, the user interface generator 102 receives an indication of whether the image analysis step and any parameter(s) associated therewith received at step 256 are acceptable. If such step and parameter(s) are acceptable, then at step 266, the sequencer 106 stores information about the image analysis step (e.g., an identifier associated with the step) and the parameter(s) in the workflow data store 118. Otherwise, at step 268, the sequencer 106 clears any information about such image analysis step from its memory, and the user interface generator 102 removes the panel 210 associated with the image analysis step from the GUI 200.

[0074] Thereafter, at step 270, the user interface generator 102 receives an indication from the user computer 122 from the user computer 122 indicating whether the operator wishes to specify another image analysis step to be added to the workflow. If the operator does want to add another image analysis step, the image analysis system 100 proceeds to step 256. Otherwise, at step 272, the sequencer 106 stores the workflow in the workflow data store 118 for use with images of other biological samples.

[0075] Figure 5 Shown in Figure 4 Flowchart 300 of steps performed by one embodiment of sequencer 106 during step 258 of flowchart 250 is shown. Figures 1 to 5 In step 302, the sequencer 106 determines that in step 256 ( Figure 4 ) whether the specified image analysis step specifies the acquisition of an image from the imaging system 124. If so, then at step 304, the sequencer 106 directs the image acquirer 104 to instruct the imaging system 124 to acquire an image of the sample loaded in the imaging system 124 according to the parameters associated with the specified image analysis step. The sequencer 106 receives the acquired image at step 306, stores the acquired image in the image data storage 116 at step 308, and proceeds to Figure 4 Step 260 is used to direct user interface generator 102 to display the acquired image in area 204 of GUI 200 .

[0076] If, at step 302, the sequencer 106 determines that the image analysis step does not specify the acquisition of an image, then at step 312, the sequencer 106 determines whether the image analysis step is specified to perform image processing operations, including using the trained machine learning system 112. If so, then at step 316, the sequencer 106 loads the image specified by any parameters associated with the image analysis step at step 316 and calls the image processor 110 to process the image at step 318. Thereafter, the sequencer 106 proceeds to steps 306 and 308 to receive and store the results generated by the image processor 110, respectively.

[0077] If, at step 312, the sequencer 106 determines that the image analysis step does not specify an image processing operation, then the sequencer 106 determines at step 320 whether the image analysis step specifies an indicator for exploiting characteristics of one or more objects represented in the one or more images. If so, then at step 322, the sequencer 106 loads the one or more images specified as parameters into step 256 ( Figure 4) and calls the measurement calculator 115 in step 324 to develop metrics for these objects. For example, the measurement calculator 115 generates a count of the number of objects (cells or portions thereof) represented in the loaded image or statistics associated with the objects. The statistics may include the mean, mode and / or median of the area of ​​the objects, a histogram or distribution of the areas, the perimeter and / or size of the objects, a bounding box of the objects, etc. Obviously, other types of metrics that are obvious to those of ordinary skill in the art can be developed by the measurement calculator 115. Also in step 324, the measurement calculator 115 can display the metrics thus developed via a graphical user interface, or store the metrics in step 256 ( Figure 4 ) in a data location (not shown) identified in the parameters specified in the parameters. In some cases, the measurement calculator 115 may develop an image representing the metric thus developed. The image may include, for example, a bounding box superimposed around an object represented in the image(s) from which the metric was developed, color coding objects in the image(s) according to the metric associated with the object (e.g., color coded objects within one standard deviation from the mean using a first color and within two standard deviations from the mean using a second color). It will be apparent to one of ordinary skill in the art that the measurement calculator 115 may apply boundary or edge detection, boundary filling, and similar algorithms to the image being measured to develop the metric. Thereafter, the sequencer 106 returns to Figure 4 Step 262.

[0078] If, at step 320, the sequencer 106 determines that Figure 4 ) The specified image analysis step does not specify an exploitation indicator, then the sequencer 106 determines at step 326 whether the image analysis step specifies manual editing of the image identified by the parameters associated with the specified image analysis step. If so, then at step 328, the sequencer 106 loads the image to be edited, and at step 330, calls the image editor 108 to allow the operator to manually edit the image, as described in more detail below. Thereafter, the sequencer 106 proceeds to steps 306 and 308 to receive and store the results generated by the image editor 108, respectively.

[0079] An operator may wish to edit an image, such as that produced by a previous image processing operation and displayed on the user computer 122. For example, if a segmented image generated by performing one or more image processing operations incorrectly associates some of its pixels with objects, the operator may set the value of each such incorrectly associated pixel to a value associated with a background (e.g., non-object) pixel. Such editing may be performed, for example, to edit annotated training data prior to using such annotated training data to train the machine learning system 112.

[0080] If, at step 326, the sequencer 106 determines that the image analysis step does not specify manual editing of the image, then the sequencer 106 determines at step 332 whether the image analysis step specifies training the machine learning system 112. If so, then at step 334, the sequencer 106 calls the machine learning system trainer 114 to train the machine learning system 112, as described below. Thereafter, processing proceeds to Figure 4 Step 262.

[0081] If, at step 332, the sequencer 106 determines that the image processing operation specified by the operator does not specify training the machine learning system 112, the image analysis system proceeds to step 256 ( Figure 4 ) to wait for the specification of another image analysis step. Those skilled in the art will appreciate that the image analysis system 100 may receive additional inputs from the user computer 122 that are not relevant to the embodiments described herein. For the sake of clarity, such additional inputs are not described herein.

[0082] Figure 6 When the operator selects the image editing step, Figure 5 At step 330 of flowchart 300, flowchart 350 of the process performed by image editor 108 is shown. Figure 1 and Figures 4 to 7A , in step 352( Figure 6 ), the image editor 108 guides the user interface generator 102 to generate Figure 7 An operator can edit the segmented or characterized image, for example, by adjusting pixels associated with the edges of objects to make such edges more distinct, connecting pixels of adjacent objects so that such pixels are associated into a single larger object, removing misidentified objects, etc. In addition to these adjustments, an operator can edit the classified image to modify the classification of the objects.

[0083] The image editing panel 210f includes a pop-up menu 212f (as described above) for selecting a source image to edit, a text box 354 in which the operator can enter an identifier associated with the edited image, and selectable buttons 356, 358, 360, 362, and 364 for selecting a drawing mode as described below. In some embodiments, the image editing panel 210f also includes a button 366 labeled "Apply".

[0084] Also refer to Figure 5 and Figure 6 , in step 372( Figure 6 ), the image editor 108 creates an edited image, which is edited in step 328 ( Figure 5) a copy of the image to be edited (e.g., the source image) loaded by the sequencer 106. At step 374, the image editor 108 directs the user interface generator 102 to display the source image in the sub-region 368 of the area 204 of the GUI 200 and to display the edited image in the sub-region 370.

[0085] At step 376 , image editor 108 waits to receive input (via user interface generator 102 ) indicating operator selection of one of buttons 356 - 364 , pixels of sub-region 368 of the displayed source image, or “Apply” button 366 .

[0086] If, in step 378, the image editor 108 determines whether the input indicates selection of one of the drawing mode buttons 356 to 364, and if so, proceeds to step 380. Otherwise, the image editor 108 proceeds to step 382. In step 380, the image editor 108 selects the drawing mode button 356 to 364 ( Figure 7 ) to set an effective drawing mode. In particular, if button 358 is selected by the operator, a drawing mode is selected that sets the pixel value of the edited image corresponding to each subsequently selected pixel of the source image to a first predetermined value associated with the foreground (e.g., to indicate that such pixel is associated with an object in the source image).

[0087] Similarly, if button 360 is selected, the drawing mode sets the value of a pixel of the edited image 370 corresponding to each subsequently selected pixel of the source image displayed in sub-area 368 to a second predetermined value associated with background pixels (e.g., to indicate that the pixel is not associated with an object in the source image).

[0088] Button 362 selects a drawing mode that sets values ​​of the edited image 370 corresponding to subsequently selected values ​​to their unedited values ​​(e.g., erasing any edits performed on such selected pixels). Button 364 adjusts the number of pixels of the edited image 370 that are affected when a pixel of the source image displayed in sub-region 368 is selected (e.g., adjusts the width of the edit cursor). Selecting button 356 ends the drawing operation so that a subsequent selection of pixels of the source image displayed in sub-region 368 does not affect any pixels of the edited image 370. Image editor 108 may provide additional drawing operations, such as undoing all edits, redoing previously undone edits, clearing all edits, etc.

[0089] The drawing mode set in step 380 remains in effect until the operator selects a different one of the buttons 356 to 364, or selects the "Apply" button 366. After setting the drawing mode, the image editor 108 returns to step 376 to wait for receiving another input.

[0090] At step 382, ​​if image editor 108 determines that the input received at step 376 indicates sub-region 368 ( Figure 7 ), the image editor 108 proceeds to step 384. Otherwise, the image editor 108 proceeds to step 386.

[0091] At step 384, image editor 108 determines the coordinates of the pixels of the source image corresponding to the selected pixels of sub-region 368, and records these coordinates and the effective drawing mode in its memory (not shown). Thereafter, at step 388, image editor 108 instructs user interface generator 102 to overlay the selected pixels of sub-region 368 with the color associated with the drawing mode. In addition, also at step 388, image editor 108 directs user interface generator 102 to overlay the pixels of sub-region 370 with the color corresponding to the foreground pixels or the color corresponding to the background pixels, depending on the effective drawing mode.

[0092] For example, Fig. 7A An unedited version of the source image displayed in sub-region 368 is shown. As shown in the edited image displayed in sub-region 370, the portion of the unedited image indicated by circles 390 and 392 has been edited to change the foreground pixels, represented as white pixels in this example, to background pixels, represented as gray pixels in this example. The operator can perform such editing to set pixels of the edited image that are incorrectly identified as being associated with an object of interest in the source image. In addition, a portion of the unedited image represented by circle 394 has been edited to change the foreground pixels within the object to background pixels, for example, to separate a single object in the source image into two objects in the edited image. In addition, a portion of the source image indicated by circle 396 has been edited to replace background pixels with foreground pixels, thereby connecting two objects in the source image into a single object in the edited image.

[0093] Reference again Figure 6 After the displayed source and edited images have been updated at step 388, the image editor 108 returns to step 376 to await further input from the operator.

[0094] If, at step 382, ​​the image editor 108 determines that the operator has not selected pixels in the sub-region 368, the image editor 108 determines at step 386 whether the "Apply" button 366 is selected to indicate that the image editing is complete. If so, at step 398, the image editor 108 generates a result image that is a copy of the source image. Thereafter, the image editor 108 modifies the pixels of the result image according to the coordinates and drawing mode recorded at step 384. After the result image is generated, the process proceeds to Figure 5 Step 306.

[0095] If, at step 386, the image editor 108 determines that the operator has not selected the "Apply" button 366, the image editor 108 returns to step 376 to await another input.

[0096] In some embodiments, the operator can select an image of the biological sample for editing as described above, and the selected image is displayed in area 368. Thereafter, the operator can specify certain pixels of the image displayed in area 368 as foreground or associated with a specific classification value as described above. In response, the image editor 108 modifies the pixels of area 368 according to this designation from the operator. When the operator selects the "Apply" button 366, the image editor 108 generates a result image having a result pixel for each pixel of the image displayed in area 368. The value of each such result pixel associated with the unmodified pixel displayed in area 368 is set to the background value. The value of each result pixel associated with the modified pixel displayed in area 368 is set to the value of the modification specified by the operator for the pixel.

[0097] Figure 8 When the sequencer 106 ( Figure 1 ) calls the machine learning system trainer 114 ( Figure 1 ) in step 334 ( Figure 4 )Training machine learning system 112( Figure 1 ) is a flowchart 400 of the steps taken when Figure 1 , Figure 4 , Figure 8 and Fig. 9 At step 402, the machine learning system trainer 114 instructs the user interface generator 102 to generate Fig. 9 The machine learning system training panel 210g is shown. Fig. 9 The machine learning system training panel 210g is shown with Figure 2 The machine learning system training panel 210g shown in A is basically the same.

[0098] The machine learning system training panel 210g includes a pop-up menu 212g that the operator can use to select a training image and a pop-up menu 212h for selecting first annotated training data, which can be, for example, a segmented image corresponding to the training image. As described above, the training image and the segmented image can be generated from the same or different image sources. For example, the training image can be a transmitted light image of a biological sample, and the segmented image can be developed by applying one or more image analysis steps to the transmitted light image of the biological sample. Alternatively, the segmented image can be developed by applying one or more image analysis steps to another transmitted light image that is different from the training image of the biological sample, wherein the other transmitted light image is captured using a different combination of light sources, (one or more) lenses, (one or more) filters, etc. that were used to capture the training image. In some cases, the segmented image can be developed by applying one or more image analysis steps to an image of the biological sample captured when the biological sample undergoes fluorescence (e.g., a fluorescent image of the biological sample).

[0099] If the first annotated training data set is a segmented image, then as described above, the segmented image includes a pixel corresponding to each pixel of the training image, and the value of each pixel of the segmented image indicates whether the corresponding pixel of the training image is associated with an object (e.g., is a foreground pixel) or is not associated with an object (e.g., is a background pixel).

[0100] In addition, the machine learning system training panel 210g includes a button 430 to allow the operator to specify an optional second annotated data set. The second annotated data set can be, for example, a classified image. In some embodiments, the second annotated data set can identify features of an object identified in the first annotated data set. For example, if the first annotated data set identifies a cell, the second annotated data set can identify a specific organelle within the cell. Similarly, the first annotated data set can identify one or more organelles, and the second annotated data set can identify one or more cells of the organelle.

[0101] The machine learning system training panel 210g also includes a “Train” button 432 for initiating training of the machine learning system 112, an “Update Training” button 434 for updating the training of the machine learning system 112 with additional data, and a “Save Training” button 436 for saving parameters associated with the trained machine learning system 112.

[0102] Reference again Figure 8After generating the machine learning system training panel 210g, at step 404, the machine learning system trainer 114 waits for input from the operator via the user interface generator 102. At step 408, the machine learning system trainer 114 determines whether the input indicates the selection of a training image. If so, at step 410, the machine learning system trainer 114 loads the training image and returns to step 404 to wait for additional input.

[0103] Otherwise, at step 412 , the machine learning system trainer 114 determines whether the input indicates selection of a source of the first annotated data, and if so, loads the first annotated data at step 414 and returns to step 404 to await additional input.

[0104] In some embodiments, the “Train” button 432 is not selectable when the machine learning system training panel 210g is initially generated by the user interface generator 102. In the described embodiment, also at step 410, the machine learning system trainer 114 directs the user interface generator 102 to allow the “Train” button 432 to be selected after the source of the first annotated training data has been selected.

[0105] Otherwise, at step 416 , the machine learning system trainer 114 determines whether the input indicates selection of a source of second annotated data, and if so, loads the second annotated data set at step 418 , and returns to step 404 to await additional input.

[0106] If, at step 416, the input does not indicate selection of a second data set, the machine learning system trainer 114 determines at step 419 whether the input indicates that the "Train" button 432 is selected. If so, the machine learning system trainer 114 proceeds to step 420, otherwise the machine learning system trainer 114 proceeds to step 422. At step 420, the machine learning system trainer 114 trains the machine learning system 112 using the training images loaded at step 410, the first annotated data loaded at step 414, and, if specified, the second annotated data selected and loaded at step 414, as described in more detail below. After training the machine learning system 112 is complete, the machine learning system trainer returns to step 404 to wait for additional input.

[0107] At step 422, the machine learning system trainer 114 determines whether the "Update Training" button 434 is selected, and if so, proceeds to step 424 to update the training of the previously trained machine learning system 112, as described in more detail below. Otherwise, the machine learning system trainer 114 proceeds to step 426.

[0108] In some embodiments, the “Save Training” button 436 is not selectable when the machine learning system training panel 210 is initially generated by the user interface generator 102. In the depicted embodiment, the machine learning system trainer 114 directs the user interface generator 102 to allow the “Save Training” button 436 to be selected after the machine learning system 112 is trained or the training of the machine learning system 112 is updated, at steps 420 and 424, respectively.

[0109] At step 426, the machine learning system trainer 114 determines whether the input indicates that the "Save Training" button 436 is selected. If so, the machine learning system trainer 114 proceeds to step 428, otherwise the machine learning system trainer proceeds to step 404 to wait for additional input.

[0110] At step 428, the machine learning system trainer 114 directs the user interface generator 102 to request an identifier to associate with the trained machine learning system 112. If the previously trained machine learning system 112 has been retrained prior to selecting the “Save Training” button 436, the machine learning system trainer may allow the operator to specify the identifier associated with the previously trained machine learning system 112 or a different identifier.

[0111] Thereafter, the machine learning system trainer 114 stores parameters associated with each node of the machine learning system 112 in the machine learning system parameter data store 120. Such parameters include, for example, activation thresholds, input and output scaling factors or weights, convolution kernels, and the architecture of the machine learning system 112. For example, if the machine learning system 112 is a neural network, the architecture of the nodes or neurons in each layer, the interconnections between the node layers, etc. are stored in the machine learning system parameter data store 120. As understood by one of ordinary skill in the art, such neural network parameters may include the interconnections of one or more convolutional layers and one or more neuron layers, with one or more pooling layers between them. The stored parameters are sufficient to configure the untrained machine learning system 112 to a training state.

[0112] The stored parameters are associated with an identifier of the trained machine learning system provided by the operator, so that such parameters can be retrieved from the machine learning parameter data store by using the identifier. In some embodiments, the identifier of the trained machine learning system is used as the name of the file or directory in the machine learning system parameter data store 120 in which the parameters are stored. In other embodiments, the parameters are stored in a relational or other type of database, and the identifier of the trained machine learning system is an index value that can be used to query the database for such parameters. It will be apparent to those of ordinary skill in the art that other ways of associating the identifier with the parameters associated with the trained machine learning system can be used.

[0113] Additionally, at step 420, the machine learning system trainer 114 adds an icon 208 with an identifier of the trained machine learning to area 206 of the GUI 200 (or a menu associated with an already existing icon 208) as an image processing operation, such that the trained machine learning system can be specified in an image analysis step, such as at step 318, in future workflows developed using the image analysis system 100.

[0114] Fig. 9 is Figure 8 450 of the process performed by step 420 of flowchart 400 is shown. Fig. 9 At step 452, the machine learning system trainer 114 configures the architecture of the untrained machine learning system 112. This configuration specifies the number of inputs and outputs of this machine learning system 112. For example, the number of inputs may be specified according to the number of pixels in the training image loaded at step 410, and the number of outputs may be specified according to the sum of the number of values ​​in the first annotated dataset and the second annotated dataset.

[0115] In addition, if the machine learning system 112 is a neural network, the configuration may specify the number of layers in the neural network, such as the number of convolutional layers, pooling layers, and / or neuron layers. In addition, the configuration may specify the interconnections between the layers that make up the machine learning system 112.

[0116] In some embodiments, if the machine learning system 112 includes a neural network, such a neural network is configured using AutoML and NASNet technologies developed by Google, Inc. of Mountain View, Calif. It should be appreciated that other neural network technologies known to those skilled in the art may be used, including, for example, fully convolutional DenseNet, neural networks optimized for machine vision applications, etc. It should be appreciated that the machine learning system 112 may be another type of machine learning system including random forest trees, etc.

[0117] In some embodiments, the architecture of the untrained machine learning system 112 is selected from one or more predetermined configurations. Factors that may be considered to select such a configuration may include, for example, the size of the training data (e.g., number of pixels, image size, etc.), the size of the annotated data set, the speed of training the machine learning system 112, the acceptable level of accuracy of the results generated by the machine learning system 112, the memory and processing requirements of the machine learning system 112, the type of output to be produced (e.g., if the machine learning system is to produce a segmented image, data identifying one or more centroids of objects in the input image, an image that classifies the objects in the input image, etc.), the complexity of the data provided as input to the machine learning system 112, the size and / or complexity of the objects represented in the image analyzed by the machine learning system 112, etc. Some configurations may be optimized for specific types of samples represented in the image to be analyzed by the machine learning system 112 (e.g., neurites, TL cell monolayers, etc.). Different configurations may be used to analyze a two-dimensional image of a biological sample versus a series of two-dimensional images that capture a three-dimensional view of the biological sample.

[0118] At step 454, the machine learning system trainer 114 presents the training image loaded at step 410 as an input to the machine learning system 112 configured at step 452, and in response, the machine learning system 112 develops first predicted output values ​​corresponding to the first annotated training data. Each of the first predicted output values ​​corresponds to one of the first annotated training data. If the training image is an image of a biological sample, and the first annotated training data represents a segmented image developed from a first source image, where the first source image is the training image or another image of the biological sample captured under different imaging conditions, each first predicted output value indicates a probability that a pixel corresponding to the first source image is associated with an object of interest in the training image (e.g., is a foreground pixel), as specified by the segmented image.

[0119] Additionally, if second annotated training data is loaded at step 414, then in some embodiments, the machine learning system 112 simultaneously develops second prediction output values, each such value corresponding to a pixel of the second annotated training data. For example, if the second annotated training data is developed from a second source image, which is the training image or another image of the biological sample, and such annotated training data identifies pixels in the second source image that are associated with particular characteristics (e.g., using intensity encoding to identify different characteristics), then each second prediction output value indicates a probability that the corresponding pixel of the training image is associated with one such particular characteristic.

[0120] At step 456, the machine learning system trainer 114 develops a first difference between the first set of output values ​​and the first annotated training data. If a second set of annotated training data is loaded, the machine learning system trainer 114 also derives a second difference between the second set of output values ​​and the second annotated training data. In some embodiments, this difference is calculated using a loss function (e.g., a weighted categorical cross entropy function).

[0121] At step 458, the machine learning system trainer 114 combines the first difference(s) and any second difference(s) to form an aggregate error metric. Such an error metric may be, for example, the number of pixels in the training image that are correctly associated with the segmentation and the characteristics represented in the first annotated training data and the second annotated training data, respectively. At step 460, the machine learning system trainer 114 determines whether the magnitude of the aggregate error metric is less than a predetermined acceptable error, and if so, the process proceeds to Figure 8 Step 404 of flowchart 400 is shown.

[0122] Otherwise, at step 462, the machine learning system trainer 114 determines whether additional training of the machine learning system 112 should be performed based on, for example, the number of iterations of training that have been performed, the rate of improvement of the total error metric between successful training passes, the amount of time that training has been performed, and other such conditions that will be apparent to one of ordinary skill in the art. If additional training is warranted, the machine learning system trainer 114 proceeds to step 464, otherwise processing returns to step 465. Figure 8 Step 404 of flowchart 400 is shown.

[0123] In step 464, the machine learning system trainer 114 uses the difference values ​​developed in step 456 to adjust the parameters of the machine learning system 112, for example, using back propagation or another technique, as understood by one of ordinary skill in the art.

[0124] After adjusting the parameters, the machine learning system trainer 114 returns to step 454 to operate the machine learning system 112 using the adjusted parameters.

[0125] In some cases, at step 462, the machine learning system trainer 114 may determine that the summary error metric is greater than a predetermined acceptable error, but additional training is not warranted (e.g., if the summary error metric has not improved). In this case, the machine learning system trainer 114 may direct the user interface generator 102 to display the summary error metric and a message that such summary error metric is greater than a predetermined acceptable error on the user computer 122, and not perform additional training.

[0126] In some embodiments, the machine learning system 112 can be trained to operate on sub-portions (or tiles) of the training image, rather than using all pixels of the training image at the same time. In such an embodiment, the number of inputs configured in step 452 can be based on the number of pixels in each tile. In addition, the machine learning system 112 can include functionality to divide the training image into such tiles. For example, the training image can be divided into tiles having a size of, for example, 32 pixels by 32 pixels, where each tile replicates some pixels of an adjacent tile (e.g., 8 pixels in each direction). Those skilled in the art will appreciate that tiles of other sizes and with different numbers of repeated pixels can be used. In other embodiments, the size of the tile can be selected based on the expected number of pixels expected to be occupied by cells or another portion of interest of the biological specimen.

[0127] When the machine learning system 112 is configured to operate on tiles developed from training images, in some embodiments, the machine learning system 112 can be configured to generate an output value per tile that indicates the probability that the tile includes one or more pixels representing a characteristic of a particular object of interest or biological sample. In other embodiments, the machine learning system 112 can be configured to generate output data including a value for each pixel of the tile. Furthermore, in some embodiments, the machine learning system 112 can be configured with additional functionality that combines output data associated with tiles into summary data associated with the training images from which such tiles were developed.

[0128] Fig.10 is Figure 8 Flowchart 480 of a process for updating the training of a previously trained machine learning system 112, performed at step 420 of flowchart 400, is shown. For example, if an operator encounters an image that a previously trained machine learning system 112 does not adequately analyze (e.g., produces an erroneous output value), the operator may wish to update the training of the machine learning system 112. As one of ordinary skill in the art will appreciate, the machine learning system trainer 114 may use transfer learning or similar techniques to update the parameters of an already trained machine learning system 112, rather than starting with an untrained machine learning system 112. Reference Fig. 8A To initiate such retraining, the operator selects new training images for updating the training of the machine learning system 112 using parameter box 212g, selects an annotated dataset using parameter box 212h, and optionally selects further annotated datasets using button “Add classification mask” 430, as described above. The operator then selects “Update training” button 434.

[0129] refer to Fig.10In one embodiment, at step 482, the machine learning system trainer 114 directs the user interface generator 102 to obtain from the operator an identifier associated with the machine learning system to be retrained. In some embodiments, the user interface generator 102 presents a list of identifiers associated with machine learning parameters stored in the machine learning system parameter data store 120, receives a selection of one of the presented identifiers, and provides such selection to the machine learning system trainer 114.

[0130] Thereafter, at step 484, the machine learning system trainer 114 loads the parameters associated with the selected identifier from the machine learning system parameter data store 120. At step 486, the machine learning system trainer 114 uses the loaded parameters to configure the untrained machine learning system 112. Thereafter, the machine learning system trainer 114 proceeds to step 454 ( Fig. 9 ) so that you can use Figure 8 The configured machine learning system 112 is trained using the training images loaded in steps 410 , 414 , and 418 , the first annotated dataset, and the second annotated dataset (if any).

[0131] Although the image analysis system 100 described above is described as being used to train the machine learning system 112 to analyze a single two-dimensional image, such an image analysis system 100 may be adapted to train the machine learning system 112 to analyze a series of two-dimensional transmitted light images of a biological sample taken at different focal points, the two-dimensional transmitted light images representing a three-dimensional representation of such a biological sample. In such an embodiment, the training images used to train the machine learning system 112 include a series of training images, wherein all images comprising such a series are associated with substantially the same XY position of the biological sample and are associated with different focal points (e.g., different Z positions). All images comprising such a series of images are provided to the machine learning system 112 simultaneously, and the machine learning system 112 generates prediction values, wherein each value is associated with a pixel of the image series and indicates the probability that such pixel is associated with an object of interest (in three dimensions) or is associated with a particular characteristic.

[0132] In some embodiments, the annotated data generated from a series of training images acquired at different Z positions represents a confocal image generated from the series. Thereafter, the series of training images and the annotated data set can be used to train the machine learning system 112 to generate a confocal image from a series of images acquired at different focal points.

[0133] Reference again Figure 2As will be apparent to one of ordinary skill in the art, the image of the biological sample selected as training data at step 160 may be a binary image, a grayscale image, or an image with multiple color planes. Similarly, the first annotated data and / or the second annotated data developed at steps 154 and 156, respectively, may include binary values ​​(e.g., a single bit), grayscale or single byte values, multi-color or multi-byte values, and / or combinations thereof. Furthermore, the data generated by the trained machine learning system may include binary values ​​(e.g., a single bit), grayscale or single byte values, multi-color or multi-byte values, and / or combinations thereof.

[0134] It will be apparent to those skilled in the art that any combination of hardware and / or software may be used to implement the image analysis system 100 described herein. Figures 1 to 10 One or more of the processes, sub-processes and process steps described may be performed by hardware, software or a combination of hardware and software on one or more electronic or digital control devices. The software may reside in a software memory (not shown) in a suitable electronic processing component or system, such as a processor or a processor. Figures 1 to 10 One or more of the functional systems, controllers, devices, components, modules or submodules schematically shown in the figure. The software memory may include an ordered list of executable instructions for implementing logical functions (i.e., "logic" may be implemented in digital form such as digital circuits or source code, or in analog form such as analog sources (such as analog electrical, sound or video signals)). The instructions may be executed in a processing module or controller (e.g., Figure 2 The user interface generator 102, image acquirer 104, sequencer 106, image editor 108, image processor 110, machine learning system trainer 114 and machine learning system 112 are executed within the processing module or controller, and the processing module or controller includes, for example, one or more microprocessors, general-purpose processors, combinations of processors, digital signal processors (DSPs), field programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition, the schematic diagram describes a logical division of functions with physical (hardware and / or software) implementations that are not limited by the architecture or physical layout of the functions. The example systems described in this application can be implemented in various configurations and operated as hardware / software components in a single hardware / software unit, or as separate hardware / software units.

[0135] It will be apparent to those skilled in the art that one or more components of the image analysis system 100 may operate in a cloud environment, for example, using a distributed computing and storage system connected via a local area network or a wide area network (e.g., the Internet or a private network). For example, one or more of the user interface generator 102, the image acquirer 104, the sequencer 106, the image editor 108, the image processor 110, the machine learning system 112, and the machine learning system trainer 114 may operate on one or more computers remote from the user computer 122, the imaging system 124, and / or each other. In one such configuration, for example, the image acquirer 104 may communicate with the imaging system 124 via a local area network or a wide area network to acquire images from the imaging system 124 and store the images acquired in this manner in a cloud-based image data storage 116. The sequencer 106 may provide images stored in such a cloud-based or remote image data storage 116 to other components of the image analysis system 100 for processing in conjunction with performing one or more image analysis steps. A user may specify one or more paths to one or more images stored in such an image data storage 116 in the panel 210. A machine learning system trainer 114 operating locally or in the cloud can access such images to train the machine learning system 112 and store the parameters in a machine learning system parameter data store 120, which can also be a cloud-based storage system. The sequencer 106 can access the parameters stored in the machine learning system parameter data store 120 to train an untrained machine learning system 112 (operating on a local computer or in a cloud environment). Similarly, operation of the machine learning system 112 can be performed to analyze images acquired using a local or remote imaging system 124.

[0136] Executable instructions may be implemented as a computer program product having instructions stored therein, which, when executed by a processing module of an electronic system, directs the electronic system to execute the instructions. A computer program product may be selectively embodied in any non-transitory computer-readable storage medium for use by or in conjunction with an instruction execution system, apparatus, or device, such as an electronic computer-based system, a processor-containing system, or other system that can selectively obtain instructions from and execute instructions in an instruction execution system, apparatus, or device. In the context of this document, a computer-readable storage medium is any non-transitory device that can store a program used by or in conjunction with an instruction execution system, apparatus, or device. A non-transitory computer-readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device. A non-exhaustive list of more specific examples of non-transitory computer-readable media includes: an electrical connection having one or more conductors (electronic); a portable computer disk (magnetic); random access, i.e., volatile memory (electronic); read-only memory (electronic); erasable programmable read-only memory, such as flash memory (electronic); optical disk storage, such as CD-ROM, CD-R, CD-RW (optical); and digital versatile disk storage, i.e., DVD (optical).

[0137] It will also be understood that the reception and transmission of the signal or data as used herein means that two or more systems, equipment, components, modules or submodules can communicate with each other via the signal traveling on a certain type of signal path.Signal can be communication, electric power, data or energy signal, which can transmit information, electric power or energy from a first system, equipment, component, module or submodule to a second system, equipment, component, module or submodule along the signal path between the first and second systems, equipment, components, modules or submodules.Signal path can include physical, electrical, magnetic, electromagnetic, electrochemical, optical, wired or wireless connection.Signal path can also be included in additional systems, equipment, components, modules or submodules between the first and second systems, equipment, components, modules or submodules.

[0138] All references, including publications, patent applications, and patents, cited herein are hereby incorporated by reference to the same extent as if each reference was individually and specifically indicated to be incorporated by reference and were set forth in its entirety herein.

[0139] The terms "one" and "an" and "the" and similar references used in the context of describing the present invention (especially in the context of the following claims) are interpreted as covering the singular and plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise specified herein, the description of the numerical range herein is intended only to be used as a shorthand method for individually referring to each individual value falling within the range, and each individual value is incorporated into the specification as if it were individually described herein. Unless otherwise specified herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. Unless otherwise required, the use of any and all examples or exemplary language (e.g., "such as") provided herein is intended only to better illustrate the present disclosure, rather than to impose limitations on the scope of the present disclosure. The language in the specification should not be interpreted as indicating that any unclaimed element is necessary for the practice of the present disclosure.

[0140] In view of the foregoing description, many modifications to the present disclosure will be apparent to those skilled in the art.It should be understood that the illustrated embodiments are exemplary only and should not be considered to limit the scope of the present disclosure.

Claims

1. A system for analyzing an image, wherein: The system comprises a non-transitory programmable device, the system comprising: a user interface generator operating on the non-transitory programmable device, the user interface generator receiving a designation of a first image of a biological sample and a designation of a first image analysis step; a sequencer operating on the non-transitory programmable device, the sequencer applying the first image analysis step to the first image to develop annotated training data; a machine learning system trainer operating on the non-transitory programmable device, the machine learning system trainer training an untrained machine learning system to develop a trained machine learning system, wherein when the trained machine learning system is fed the first image as input, the trained machine learning system develops predictions for the annotated training data; wherein the sequencer receives a specification of a workflow including a second image analysis step, which specifies operating the trained machine learning system, and the sequencer executes the workflow on a second image to analyze the second image; The system also includes an image editor operating on the non-transitory programmable device, wherein the sequencer develops an intermediate image from the first image by applying the first image analysis step, the user interface generator receives image editing instructions, and the image editor applies the image editing instructions to the intermediate image to develop the annotated training data.

2. The system of claim 1 , wherein the annotated training data comprises first annotated training data and the predictions for the annotated training data comprise first predictions for the first annotated training data, the user interface generator receives a specification of a third image analysis step, the sequencer applies the third image analysis step to a third image of the biological sample to develop second annotated training data, and the machine learning system trainer trains the untrained machine learning system based on the first annotated training data and the second annotated training data to develop the trained machine learning system, such that when the trained machine learning system takes the first image as input, the trained machine learning system simultaneously develops the first prediction and the second prediction for the second annotated data.

3. The system according to claim 2, wherein: The second annotated training data includes a plurality of second annotated training data values, each of the plurality of second annotated training data values ​​corresponding to a pixel of the third image, and the third image analysis step causes the sequencer to identify groups of adjacent pixels in the third image having the same value and set the second annotated training data values ​​associated with the pixels comprising the group to a predetermined value associated with the number of the pixels comprising the group.

4. The system according to claim 2, wherein: The second annotated training data and the third image are the same.

5. The system according to claim 1, wherein: The intermediate image includes a plurality of pixels, each of the plurality of pixels corresponding to a pixel of the first image and having a value associated with a characteristic of an object represented by the pixel of the first image or a background value.

6. The system according to claim 5, wherein: The image editing instructions cause the image editor to modify the value of a pixel of the plurality of pixels from a value associated with the characteristic to a value associated with the background.

7. The system according to claim 1, wherein: The non-transitory programmable device includes a first non-transitory programmable device, the sequencer includes a first sequencer, the machine learning system trainer includes a first machine learning system trainer, the trained machine learning system includes a first trained machine learning system, and the workflow includes a first workflow, the system also includes a second sequencer and a second machine learning system trainer, the second machine learning system trainer operates on a second non-transitory programmable device remote from the first non-transitory programmable device, wherein the second machine learning system trainer uses parameters associated with the first trained machine learning system to configure an untrained machine learning system to develop a second trained machine learning system, the second sequencer receives a specification of a second workflow, which specifies operating the second trained machine learning system, and applying the second workflow to a third image of the biological sample to analyze the third image.

8. The system according to claim 1, wherein: The user interface generator receives a designation of a third image of the biological sample, the sequencer develops second annotated data from the third image, and the machine learning system trainer updates the training of the trained machine learning system using the third image and the second annotated training data.

9. The system according to claim 8, wherein: The trained machine learning system includes a neural network, and a machine learning system trainer uses transfer learning to update the training of the trained machine learning system.

10. The system according to claim 1, wherein: The first image is one of a first series of images representing a three-dimensional representation of the biological sample, and the annotated training data is developed from all of the second series of images, and the machine learning system trainer simultaneously provides all of the first series of images to the machine learning system as input to train the machine learning system to produce predictions of the annotated training data within a predetermined error.

11. The system according to claim 10, wherein: The annotated training data are confocal images developed from the first series of images.

12. The system of claim 1, wherein: The user interface generator receives a designation of a third image of the biological sample, and the sequencer develops the annotated training data from the first image and the third image.

13. The system of claim 1, wherein: The first image analysis step includes a first image analysis step, and the sequencer applies a second image analysis step to a third image of the biological sample to develop an annotated training dataset, wherein the first image and the third image are acquired using one or more of different illumination, optical, and imaging conditions.

14. The system of claim 1, wherein: The specifying of the first image includes specifying a location of the first image in an image repository.

15. The system of claim 14, wherein: The image repository is located remotely from the non-transitory programmable device.

16. A method for analyzing an image, comprising the steps of: Operate the non-transitory programmable device to perform the following steps: receiving a first image of a biological sample and a specification of a first image analysis step; developing annotated training data by applying the first image analysis step to the first image; training an untrained machine learning system to develop a trained machine learning system, wherein the trained machine learning system develops predictions of the annotated training data within a predetermined error when the trained machine learning system is fed with the first image as input; receiving a specification of a workflow including a second image analysis step, wherein the second analysis step includes operating the trained machine learning system; and executing the workflow on a second image to analyze the second image; Wherein, operating the non-transitory programmable device to perform developing the annotated training data includes: developing an intermediate image from the first image by applying the first image analysis step, receiving image editing instructions, and applying the image editing instructions to the intermediate image to develop the annotated training data.

17. The method according to claim 16, wherein: The annotated training data comprises first annotated training data, and the prediction of the annotated training data comprises a first prediction of the first annotated training data, and the non-transitory programmable device is operated to perform the following further steps: receiving a designation of a third image analysis step; applying the third image analysis step to a third image of the biological sample to develop second annotated training data; and training the untrained machine learning system to develop the trained machine system such that when the trained machine learning system is fed with the first image as input, the trained machine learning system simultaneously develops the first prediction and a second prediction for the second annotated data.

18. The method according to claim 17, wherein: The second annotated training data comprises a plurality of second annotated training data values, each of the plurality of second annotated training data values ​​corresponding to a pixel of the third image, and the non-transitory programmable device is operated to perform the further steps of identifying a group of adjacent pixels in the third image having a same predetermined value, and setting the second annotated training values ​​associated with the pixels comprising the group to a predetermined value associated with the number of the pixels comprising the group.

19. The method according to claim 17, wherein: The second annotated training data and the third image are the same.

20. The method according to claim 16, wherein: The intermediate image includes a plurality of pixels, each of which corresponds to a pixel of the first image and has a value associated with a characteristic of an object represented by the pixel of the second image or a background value.

21. The method according to claim 20, wherein: Operating the non-transitory programmable device to execute the image editing instructions causes the non-transitory programmable device to perform modification of a value of a pixel of the plurality of pixels from a value associated with the characteristic to a value associated with the background.

22. The method according to claim 16, wherein: The non-transitory programmable device includes a first non-transitory programmable device, the workflow includes a first workflow, and the trained machine learning system includes a first trained machine learning system, and the method also includes operating a second non-transitory programmable device remote from the first non-transitory programmable device to perform the following steps: configuring an untrained machine learning system using parameters associated with the first trained machine learning system to develop a second trained machine learning system; receiving a specification of a second workflow that specifies operating the second trained machine learning system; and applying the second workflow to a third image of the biological sample to analyze the third image.

23. The method of claim 16 further comprising operating the non-transitory programmable device to perform the additional steps of receiving a designation of a third image of the biological sample, developing second annotated data from the third image, and updating the training of the trained machine learning system using the third image and the second annotated training data.

24. The method according to claim 23, wherein: The trained machine learning system includes a neural network and includes using transfer learning to update the training of the trained machine learning system.

25. The method of claim 16, wherein: The first image is one of a first series of images representing a three-dimensional representation of the biological sample, developing the annotated training data includes developing the annotated training data from all of the first series of images, and training the machine learning system includes providing all of the first series of images as input to the machine learning system.

26. The method of claim 16, wherein: The step of developing the annotated training data comprises developing the annotated training data from the first image and a third image of the biological sample.

Images