SYSTEM AND METHOD FOR PROCESSING ELECTRONIC IMAGES OF SLIDE FOR DIGITAL PATHOLOGY WORKFLOW - Patent application

JP2024500100A5Active Publication Date: 2024-11-20PAIGE AI INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2023536426
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2020-12-18
Filing Date
2021-12-17
Publication Date
2024-11-20
Estimated Expiration
2041-12-17

AI Technical Summary

Benefits of technology

【0016】 上述の説明と以下の詳細な説明はともに、例示で単に説明をしているものであり、開示の実施形態を限定するものではないという旨を理解すべきである。

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 00000000_0000_ABST
    Figure 00000000_0000_ABST
Patent Text Reader

Abstract

A computer-implemented method of using a machine learning model to classify samples in digital pathology may include receiving one or more cases, each associated with a digital image of a pathology specimen, identifying the cases as ready to be viewed using the machine learning model, receiving a selection of a case, the case including a plurality of parts, determining whether the plurality of parts is suspicious or unsuspected using the machine learning model, receiving a selection of a part of the plurality of parts, determining whether a plurality of slides associated with the one part are suspicious or unsuspected, and determining a collection of suspicious slides of the plurality of slides using the machine learning model, the machine learning model being trained by processing a plurality of training images.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 127,846, filed December 18, 2020, which is incorporated by reference in its entirety.

[0002] Various embodiments of the present disclosure relate generally to image processing methods, and more specifically, certain embodiments of the present disclosure relate to systems and methods for classifying artificial intelligence (AI) visualization attributes to create a reusable and scalable framework for unique AI-enabled visualization and interaction, resulting in AI-enabled digital workflows that are helpful to pathologists in making diagnoses. [Background technology]

[0003] There are a wide variety of tissue and surgical specimen types that may require pathological review and diagnosis. Examples of reportable features and instances that a pathologist may need to view, review, and diagnose may include features and instances within areas such as the bladder, colon, lung, dermatology, and stomach. It may be useful for the pathologist to provide visualization of some of the areas (e.g., bladder, colon, lung, dermatology, and stomach features).

[0004] The background discussion provided herein is intended to generally provide a context for the present disclosure. Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application and is not admitted to be prior art or an indication of prior art by inclusion in this section. Summary of the Invention [Means for solving the problem]

[0005] According to certain aspects of the present disclosure, a system and method for processing electronic images of slides for a digital pathology workflow is disclosed.

[0006] A computer-implemented method of using a machine learning model to classify samples in digital pathology may include receiving, on a digital storage device, one or more cases, each associated with a digital image of a pathology specimen, using the machine learning model to identify a case of the one or more cases as ready to be viewed, receiving a selection of a case, the case including a plurality of parts, determining whether the plurality of parts is suspicious or unsuspected using the machine learning model, receiving a selection of a part of the plurality of parts, and determining whether a plurality of slides associated with the one part are suspicious or unsuspected, determining a set of suspicious slides of the plurality of slides using the machine learning model, the machine learning model being trained by processing a plurality of training images, and annotating the set of suspicious slides and / or generating a report based on the set of suspicious slides.

[0007] Identifying a case as ready to view may include verifying that all slides in the case have been processed and uploaded to a digital storage device. In the techniques presented herein, annotating the collection of suspect slides may further include outlining at least one region around the suspicious tissue (or non-suspicious tissue instead or in addition to the suspicious tissue), measuring the length and / or area of ​​the suspicious tissue (or non-suspicious tissue), and outputting the annotations for the collection of suspect slides.

[0008] The method may further include inputting information about the collection of suspicious slides (and / or non-suspicious slides) into a report and outputting the report to a user. The information about the collection of suspicious slides may include lesion areas, context areas, one or more measurements of the suspicious tissue, alphanumeric output, and / or a report compiled based on all of the lesion areas, context areas, and / or measurements. The alphanumeric output may include a binary indication of the presence of one or more biomarkers in the tissue. The report and / or visualization of the suspicious tissue may include a detection panel, a quantification panel, and / or an annotation log. The annotation log may be searchable at the case level, the section level, and / or the slide level.

[0009] The method may further include determining a complexity associated with each of the one or more cases and prioritizing the one or more cases based on the determined complexity. The method may further include sorting and / or filtering the plurality of moieties for display based on a determination of whether the plurality of moieties is suspicious or non-suspicious using a machine learning model. The method may further include determining a pathology type and generating a report based on the determined pathology type.

[0010] The method may further include using the machine learning model to determine a plurality of features based on a digital image associated with the case, using the machine learning model to determine a plurality of partial reports using the plurality of features, and determining a report based on the plurality of partial reports.

[0011] A system for using a machine learning model to classify samples in digital pathology may include at least one memory storing instructions and at least one processor configured to execute the instructions to perform operations that may include receiving one or more cases at a digital storage device, each associated with a digital image of a pathology specimen, identifying a case of the one or more cases as ready to be viewed using a machine learning model, receiving a selection of a case, the case including a plurality of parts, determining whether the plurality of parts is suspicious or unsuspected using a machine learning model, receiving a selection of a part of the plurality of parts, determining whether a plurality of slides associated with the one part are suspicious or unsuspected, determining a set of suspicious slides of the plurality of slides using a machine learning model, the machine learning model being trained by processing a plurality of training images, and annotating the set of suspicious slides and / or generating a report based on the set of suspicious slides.

[0012] Identifying a case as ready to view may include verifying that all slides in the case have been processed and uploaded to a digital storage device. Annotating the collection of suspicious slides may further include outlining at least one region around the suspicious tissue, measuring the length and / or area of ​​the suspicious tissue, and outputting annotations to the collection of suspicious tissue. The operations may further include entering information about the collection of suspicious slides into a report and outputting the report to a user. The information about the collection of suspicious slides may include a lesion area, a context area, one or more measurements of the suspicious tissue, and / or an alphanumeric output.

[0013] The operations may further include determining a complexity associated with each of the one or more cases and prioritizing the one or more cases based on the determined complexity. The operations may further include determining a plurality of features based on digital images associated with the cases using a machine learning model, determining a plurality of partial reports using the plurality of features using the machine learning model, and determining a report based on the plurality of partial reports.

[0014] A non-transitory computer-readable medium may store instructions that, when executed by a processor, perform a method of using a machine learning model to output a task-specific prediction. The method may include receiving, at a digital storage device, one or more cases, each associated with a digital image of a pathology specimen, using the machine learning model to identify a case of the one or more cases as ready to be viewed, receiving a selection of a case, the case including a plurality of parts, determining, using the machine learning model, whether the plurality of parts is suspicious or unsuspected, receiving a selection of a part of the plurality of parts, determining whether a plurality of slides associated with the one part are suspicious or unsuspected, determining, using the machine learning model, a set of suspicious slides of the plurality of slides, the machine learning model being trained by processing a plurality of training images, and annotating the set of suspicious slides and / or generating a report based on the set of suspicious slides.

[0015] Annotating the suspicious slide collection may further include outlining at least one region around the suspicious tissue, measuring a length and / or an area of ​​the suspicious tissue, and outputting an annotation to the suspicious tissue collection.

[0016] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed embodiments.

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments. [Brief description of the drawings]

[0018] [Figure 1A] FIG. 1 illustrates an example block diagram of a system and network for classifying artificial intelligence (AI) visualization attributes to create an AI-powered digital workflow framework, according to an example embodiment of the present disclosure. [Figure 1B] FIG. 1 illustrates an exemplary block diagram of a disease detection platform, according to an exemplary embodiment of the present disclosure. [Figure 1C] FIG. 1 shows an exemplary block diagram of a slide analysis tool, according to an exemplary embodiment of the present disclosure. [Diagram 2] 1 illustrates an exemplary AI user clinical workflow, according to an exemplary embodiment of the present disclosure. [Diagram 3] 1 shows an exemplary diagram of a slide level visualization according to an exemplary embodiment of the present disclosure. [Figure 4A] 1 shows an exemplary report of various reportable features and examples that a pathologist may need to view, review, and diagnose, according to an exemplary embodiment of the present disclosure. [Figure 4B] 1 shows an exemplary report of various reportable features and examples that a pathologist may need to view, review, and diagnose, according to an exemplary embodiment of the present disclosure. [Figure 4C] 1 shows an exemplary report of various reportable features and examples that a pathologist may need to view, review, and diagnose, according to an exemplary embodiment of the present disclosure. [Figure 4D]1 shows an exemplary report of various reportable features and examples that a pathologist may need to view, review, and diagnose, according to an exemplary embodiment of the present disclosure. [Figure 4E] 1 shows an exemplary report of various reportable features and examples that a pathologist may need to view, review, and diagnose, according to an exemplary embodiment of the present disclosure. [Diagram 5] FIG. 1 illustrates an example diagram of an overall framework for AI-enabled visualization, according to an example embodiment of the present disclosure. [Figure 6A] FIG. 1 shows an example diagram of how common visualization and interaction of different tissue types can be applied, according to an example embodiment of the present disclosure. [Figure 6B] FIG. 1 shows an example diagram of how common visualization and interaction of different tissue types can be applied, according to an example embodiment of the present disclosure. [Figure 6C] FIG. 1 shows an example diagram of how common visualization and interaction of different tissue types can be applied, according to an example embodiment of the present disclosure. [Figure 6D] FIG. 1 shows an example diagram of how common visualization and interaction of different tissue types can be applied, according to an example embodiment of the present disclosure. [Figure 7] 1 illustrates an exemplary diagram of reportable features according to an exemplary embodiment of the present disclosure. [Figure 8] 1 illustrates an example system that can implement the techniques presented herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0020] The systems, devices, and methods disclosed herein are described in more detail, by way of example, with reference to the figures. The examples described herein are merely examples and are provided to aid in the explanation of the apparatus, devices, systems, and methods described herein. None of the features or components illustrated in the drawings or described below should be construed as essential to any particular implementation of any of these devices, systems, or methods, unless specifically designated as essential.

[0021] Also, for any method described, whether or not the method is described in conjunction with a flow diagram, unless otherwise specified or required by context, it should be understood that any explicit or implicit ordering of steps performed in execution of the method does not imply that the steps must be performed in the order they are presented, but instead may be performed in a different order or in parallel.

[0022] As used herein, the term "exemplary" is used in the sense of "example" rather than "ideal." Additionally, the terms "a" and "an" do not denote a limitation of quantity herein, but rather denote the presence of one or more of the referenced item.

[0023] FIG. 1A illustrates an example block diagram of a system and network for classifying artificial intelligence (AI) visualization attributes to create an AI-powered digital workflow framework, according to an example embodiment of the present disclosure.

[0024] Specifically, Figure 1A illustrates an electronic network 120 that may be connected to servers, such as at a hospital, laboratory, and / or clinic. For example, physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125, etc., may each be connected to electronic network 120, such as the Internet, via one or more computers, servers, and / or handheld mobile devices. According to an exemplary embodiment of the present application, electronic network 120 may also be connected to server system 110, which may include a processing device configured to implement disease detection platform 100, which includes slide analysis tool 101 for determining specimen properties, or image property information regarding digital pathology image(s), and using machine learning to determine whether a disease or infectious agent is present, according to an exemplary embodiment of the present disclosure. The slide analysis tool 101 may enable rapid assessment of "suitability" in liquid-based tumor preparations, facilitate diagnosis of liquid-based tumor preparations (cytology, hematology / hematopathology), and predict molecular findings likely to be detected in various tumors detected by liquid-based preparations.

[0025] The doctor server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125 may create or otherwise acquire one or more patient's cytology specimen(s), histopathology specimen(s), images of slide(s) of cytology specimen(s), digitized images of slide(s) of histopathology specimen(s), or any combination thereof. The doctor server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125 may also acquire any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, past biomedical or cytological information, etc. The doctor server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125 may transmit the digitized slide images and / or patient-specific information to the server system 110 via the electronic network 120. The server system(s) 110 may include one or more storage devices 109 for storing images and data received from at least one of the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125. The server system 110 may also include a processing unit for processing the images and data stored on the storage device 109. The server system 110 may further include one or more machine learning tool(s) or machine learning capabilities. For example, the processing unit may include machine learning tools for the disease detection platform 100, according to one embodiment. Alternatively or additionally, the present disclosure (or portions of the systems and methods of the present disclosure) may be executed on a local processing unit (e.g., a laptop).

[0026] Physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125 refer to systems used by pathologists to review images of slides. In a hospital environment, tissue type information may be stored in the laboratory information system 125.

[0027] 1B illustrates an exemplary block diagram of a disease detection platform 100 for determining specimen characteristics or image characteristic information for digital pathology image(s) using machine learning. The disease detection platform 100 may include a slide analysis tool 101, a data capture tool 102, a slide capture tool 103, a slide scanner 104, a slide manager 105, storage 106, a laboratory information system (e.g., laboratory information system 125), and a viewing application tool 108.

[0028] The slide analysis tool 101, as described below, refers to processes and systems for determining data variable characteristic or health variable characteristic information for digital pathology image(s). Machine learning may be used to classify images according to exemplary embodiments. The slide analysis tool 101 may also predict future relationships as described in the following embodiments.

[0029] The data capture tools 102, according to an example embodiment, may facilitate the transfer of digital pathology images to various tools, modules, components, and devices used to classify and process the digital pathology images.

[0030] A slide capture tool 103, according to an exemplary embodiment, can scan a pathology image and convert the pathology image into a digital format. Slides can be scanned with a slide scanner 104, and a slide manager 105 can process the images on the slides into digitized pathology images and store the digitized images in storage 106.

[0031] The viewing application tool 108, according to an exemplary embodiment, may provide information to a user regarding specimen characteristics or image characteristics related to the digital pathology image(s). The information may be provided via a variety of output interfaces (e.g., a screen, a monitor, a storage device, and / or a web browser, etc.).

[0032] The slide analysis tool 101, and one or more of its components, may transmit and / or receive digitized slide images and / or patient information via the network 120 to the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125. Additionally, the server system 110 may include a storage device for storing images and data received from at least one of the slide analysis tool 101, the data capture tool 102, the slide capture tool 103, the slide scanner 104, the slide manager 105, and the viewing application tool 108. The server system 110 may also include a processing device for processing images and data stored in the storage device. The server system 110 may further include one or more machine learning tool(s) or machine learning capabilities, e.g., for the processing device. Alternatively or in addition, the present disclosure (or portions of the systems and methods of the present disclosure) may be executed on a local processing device (e.g., a laptop).

[0033] Any of the above devices, tools, and modules may be located on one or more components, servers, and / or devices that may be connected to an electronic network, such as the Internet or a cloud service provider, via a handheld mobile device.

[0034] 1C illustrates an example block diagram of a slide analysis tool 101 according to an example embodiment of the present disclosure. The slide analysis tool 101 may include a training image platform 131 and / or a target image platform 135.

[0035] According to one embodiment, the training image platform 131 may include a training image capture module 132 , a data analysis module 133 , and a tissue identification module 134 .

[0036] The training data platform 131, according to one embodiment, may create or receive training images used to train the machine learning model to effectively analyze and classify digital pathology images. For example, the training images may be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125. The images used for training may be obtained from real sources (e.g., humans, animals, etc.) or synthetic sources (e.g., graphic rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains, such as (but not limited to) H&E, hematoxylin alone, IHC, molecular pathology, etc., and / or (b) digitized tissue samples from 3D imaging devices such as microCT.

[0037] The training image capture module 132 may create or receive a dataset including one or more training datasets corresponding to one or more specimen tissues. For example, the training dataset may be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125. The dataset may be stored on a digital storage device. The data analysis module 133 may identify whether a set of individual cells belongs to the cells of interest or to the background of the digitized image. The tissue identification module 134 may analyze the digitized image and determine whether individual cells in the cytology sample require further analysis. It may be useful to identify whether individual cells require further analysis and consolidate these regions, and such identification may trigger an alert to the user.

[0038] According to one embodiment, the target image platform 135 may include a target image capture module 136, a specimen detection module 137, and an output interface 138. The target image platform 135 may receive a target image and apply a machine learning model to the received target image to determine characteristics of a target dataset. For example, the target data may be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125. The target image capture module 136 may receive a target dataset corresponding to a target tissue specimen. The specimen detection module 137 may apply a machine learning model to the target dataset to determine characteristics of the tissue specimen. For example, the specimen detection module 137 may detect suspicious (and / or non-suspicious) tissue regions within the tissue specimen. The specimen detection module 137 may use an artificial intelligence (AI) system that is trained to identify the presence and / or absence of various features. The AI ​​system may output a map (e.g., a tissue map) highlighting all regions associated with the identified feature or instance of the feature. U.S. Application No. 17 / 313,617 is incorporated herein by reference in its entirety. The specimen detection module 137 may determine whether an identified feature is suspicious or non-suspicious based on certain features such as size, shape, location, proximity to other identified features, disease type, color, stain type, biomarker type, gene signature, protein type, blood marker, tissue type, tissue texture, presence or level of calcification, presence or level of inflammation, etc.

[0039] The specimen detection module 137 may also apply the machine learning model to the target data set to determine a quality score of the target tissue specimen. Additionally, the specimen detection module 137 may apply the machine learning model to the target image to determine whether a target element is present within the tissue specimen.

[0040] The output interface 138 may be used to output information about the target tissue sample and tissue region of interest (eg, to a screen, monitor, storage device, web browser, etc.).

[0041] One or more of the exemplary embodiments described below may provide a set of AI visualization types and a set of interaction types. One or more exemplary embodiments may provide visualization types sorted by tissue type and any interaction types possibly required for reportable features extracted from College of American Pathologists (CAP) synopsis. One or more exemplary embodiments may also apply to biomarker products that may incorporate clinical use cases but may utilize one or more defined outputs.

[0042] Section-level and case-level aggregations may be considerations when looking at a pathologist's workflow. A pathologist may use section-level aggregations to report and diagnose tissue specimens at the section level. However, a pathology report may not include slide-level observations. In some cases, reporting fields may not be displayed at the slide level. Some reporting fields may only be displayed at the section level, for example, the number of slides with sections of ductal carcinoma in situ (DCIS).

[0043] The following definitions are illustrative only and are not intended to be limiting. Whole slide image (WSI) may refer to one or more images of stained or unstained tissue. Reportable features may refer to classified or labeled areas or lesions of suspicious tissue and any observations of tissue used for diagnosis. Feature instances may be citations of reportable features defined by diagnostic and anatomical features that may be unique to that reportable feature. Visualization types may refer to display categories of feature instances. Interaction types may refer to ways in which a user may navigate between two or more feature instances. Instance definitions may refer to one or more anatomical features that may be required for inclusion in an instance display. An AI system or AI module may refer to one or more modules that implement AI, machine learning techniques, and / or machine learning algorithms, and may not necessarily be limited to a single module, device, system, platform, etc. Aspects disclosed herein may be used in any type or configuration of AI or machine learning systems, modules, platforms, and / or image analysis and / or process systems. Various outputs may be integrated. A machine learning model may refer to a model or process that implements machine learning techniques (e.g., to recognize patterns) and may not be limited to one model. Aspects disclosed herein may use multiple machine learning models that may be combined and / or produce individual outputs that may later be combined and / or integrated.

[0044] 2 illustrates an exemplary AI user workflow 200 in a clinical setting in accordance with an exemplary embodiment. The exemplary workflow may begin at step 202 where case triage and / or prioritization may occur (Clinical User Workflow Description). The user may wish to be alerted when a case is ready for viewing, i.e., when all slides associated with the case have been processed and / or uploaded into the system. The user may further wish to know which cases should be prioritized, for example, based on complexity, and / or which cases may require additional instructions, such as sending for molecular testing or resectioning.

[0045] Complexity may be determined by diagnosis (e.g., an AI system may classify a slide as Gleason 3+4=7 with relatively low confidence, but may determine that the percentage likelihood of Gleason pattern 3 is 49.9% and the percentage of Gleason pattern 4 is 50.1%, indicating a borderline case), and / or by the need for additional testing beyond the H&E slide to determine a diagnosis (e.g., if the condition is thought to be invasive and requires additional stain images for review, such as estrogen receptor (ER), progesterone receptor (PR), human epidural growth factor receptor 1 (HER2)).

[0046] Complexity may be determined based on a combination of AI output and known workflow and testing steps required by pathology. Complex cases may be prioritized earlier in the day and / or routed to more experienced pathologists.

[0047] At step 204, the workflow may include, for example, selection of a case from a user.

[0048] At step 206, the workflow may include a triage and / or case summary. Similar to step 202, the user may wish to see a brief summary of the case or triage information. During and / or prior to this summary step of 202, the identified features and any other metadata (e.g., user-generated annotations) may be summarized.

[0049] At step 208, the workflow may include selection of a portion. The selection may be accompanied by an indicator that may indicate whether the portion appears suspicious, not suspicious, or "other," which may indicate a preprogrammed quality or an unknown variable.

[0050] At step 210, the workflow may include a triage and / or part overview step. This overview may include a part index. The user may want to see an overview of what is in the part, but may not need to see the slides. By focusing on the part level, the user may be able to see important areas of the lesion and / or images / visualization of important areas.

[0051] At step 212, the workflow may include selecting a slide. An accompanying indicator may be available that may indicate whether the portion looks suspicious, not suspicious, or "other," which may indicate preprogrammed characteristics or unknown variables.

[0052] At step 214, the workflow may include diagnosing at a part level. The diagnosis may include a part index along with the diagnosis. The AI ​​may aggregate parts of the pathology specimen to create a part index. Since users need to create reports at the part level, the aggregated part level index may streamline the user's process for generating reports.

[0053] At step 216, the workflow may end with case sign-out.

[0054] During execution of the case workflow, the AI ​​may further generate a report on the slides examined and the resulting diagnosis. At step 218, the workflow may include displaying a visualization and / or quantification panel based on the results of the part-level diagnosis.

[0055] At step 220, the visualization and / or quantification panels may be edited and / or added to the report.

[0056] At step 222, the workflow may send a report to the user. All reporting may occur at the section level. Reportable features may vary by tissue type and between biopsy and resection within tissue type. Tissue type and procedure may determine what slide-level data may be sent to the section-level report. Reporting may require integrating across sections by averaging data that varies across slides. Additionally, the presence or absence of features may be reported. There may be some features of reporting that are not present at the slide level. For example, in a breast tissue exam, a pathologist may report how many slides in a section have DCIS. Annotations or output of the AI ​​workflow, such as relevant screenshots and / or identified areas, may be added to the report.

[0057] Additionally, in step 224, the report may be used to populate a Laboratory Information System (LIS) report in an exportable format, such as PDF.

[0058] Table 1 shows an example of a slide-level visualization framework. Lesions on a slide may be described using a rectangular indicator overlay on the slide itself, and context regions are described using tissue maps or otherwise outlined regions. Anatomical elements within the lesion may aid in the effectiveness of the display, as well as a set of coordinates and sizes and / or zoom levels associated with the lesion region. Metadata may accompany the slide-level visualization and may include the name of the lesion region (e.g., "calcification"), metrics and / or text output, and size of the lesion region.

[0059] Since the number of instances may be significant for some visualization types, the context area may aggregate any or all instances across slides. Potential accompanying metadata may include the name of the context area (e.g., "Breast Subtype"), as well as metrics and / or textual output that may include the name of the subtype, the size and / or measurements of the context area, a grade associated with the context area, the architecture of the context area, etc.

[0060] Measurements of the tissue specimen may be shown in the visualization as lines with clearly labeled endpoints. Measurements may be accompanied by a name (such as "Prostate" or "Tumor Length") and a metric and / or text output (such as a numerical measurement of the line).

[0061] Also, text or numerical output may be included in the slide-level visualization or displayed directly in the visualization. The text or numerical output may include metadata, such as names or other metrics and / or text output, associated with the tissue specimen or independent annotations added by the user during review. The metadata may include, for example, biomarkers detected.

[0062] A compiled report of the visualization may be included in the slide-level visualization and may be formatted by an external vendor. The compiled report may be associated with a WSI or a particular area of ​​interest. A report, such as a PDF report, may include, among other relevant information, the slide ID and the overall score for the slide. [Table 1]

[0063] 3 shows an example diagram of an overall framework for AI-enabled visualization. The slide-level visualization 300 may include a case summary 302, a part index 304, a detection panel 306, a quantification panel 308, and one or more exportable reports 310. Additionally, the framework may include an annotation log 312 and a report builder 316, which together may form a report enabler.

[0064] In case summary 302, the framework may include AI-driven case summary and triage information available to the user before the user begins reviewing slides. The case summary may include, for example, an acquisition identification number or other type of identifier, a status (e.g., ready, reviewed, not ready), an access date, a patient name, a medical record number or MRN, a tissue type (e.g., breast, dermis or derm, gastrointestinal or GI, or prostate), a specimen type (e.g., biopsy), and a number of slides. Another part of the case summary may include a visualization, snapshot, or "slide tray" of slides, which may include an identifier (e.g., "right breast"), a type of stain (e.g., H&E), and other information.

[0065] The portion index 304 may include AI-driven portion summary and triage information available to the user before the user begins reviewing slides.

[0066] The detection panel 306 may be a templated panel that may allow for binary classification of tissue as suspicious or non-suspicious.

[0067] The quantification panel 308 may include quantification, sorting, and / or filtering of tissue features and / or other reportable features. The quantification, sorting, and / or filtering may be based on suspicious and non-suspicious judgments and / or other judgments. As an example of quantification, in the case of prostate, tumor volume may be quantified as a percentage and / or as a metric of distance or area, for example, in millimeters (mm). Individual instances of features (e.g., perineural invasion) may also be sorted and / or filtered according to probability or likelihood of severity based on an AI system that detects the instances. For example, during filtering, individual instances with low (e.g., below a threshold) probability of severity may be removed before the sorted and / or filtered features are output. The quantification panel 308 may be joined with the detection panel 306 as one panel.

[0068] One or more exportable reports 310 may be output from the digitization panel 308 and the detection panel 306. The exportable reports 310 may be in the form of a PDF.

[0069] The annotation log 312 may include an activity log of all user-generated notes associated with a tissue specimen. The annotation log 312 may be searchable at both the case and section level, and each annotation or user-generated note may include a timestamp, the user's name and role, a thumbnail image, and / or one or more additional user comments.

[0070] The annotation log 312 may be combined with a report builder 316 to create a report enabler 314. The report builder 316 may be valuable with an AI component since it may be pre-populated. Additionally, the report builder 316 may be editable and may pull information and annotations from the annotation log 312.

[0071] The report enabler 314 may include the partial level sets of user-generated annotations and AI pre-populated reports. The partial level sets and / or AI pre-populated reports may be integrated with clinical systems and recorded in the patient's medical record. A pathologist may review the partial level sets and / or AI pre-populated reports and make edits or annotations to the on-canvas visualization. Reportable feature categories may be associated with summary report fields so that as the pathologist makes these edits or annotations, the AI ​​pre-populated report may be automatically updated based on the edits. Additionally, as the pathologist makes these edits or annotations, other features related to the edited features (e.g., child features) may be automatically updated or adjusted based on the edits.

[0072] There are a wide variety of tissue and surgical specimen types that may require pathological review and diagnosis. Figures 4A-4E show exemplary reports of various reportable features and examples that a pathologist may need to view, review, and diagnose.

[0073] 4A shows an exemplary report of a bladder tissue specimen. At least one first feature is reported, such as classification, invasion presence or absence, and subtype, including, for example, whether the tissue specimen is benign, in situ, invasive, etc. One or more second features, such as tumor grade, binary index of muscularis propria, and / or invasion depth, may also be included in the report. User notes may also be included in the exemplary report.

[0074] Similarly, FIG. 4B illustrates an exemplary report of a colon tissue specimen reporting a first feature and a second feature. The first feature may include detection of cancer, identification of granulomas and acute inflammation, and a non-cancerous subtype. The non-cancerous subtype may be further reported as normal, polyp, or inflammatory, and additional information such as the subtype polyp may also be included in the report. The second feature may include a prediction of the MMR status of the H&E.

[0075] 4C, 4D, and 4E show example reports for lung tissue, skin tissue, and stomach tissue, respectively. The example reports may include various first and second features reported depending on the tissue specimen and tissue classification or subtype.

[0076] Figure 5 shows an example diagram of the overall framework of the AI-enabled visualization, which may include two different sections, some slide-level findings, and the aggregation of findings by the AI ​​system to create an overall case report.

[0077] A case report 500 may include n parts for the particular case at hand. Each part may have its own corresponding partial report 502. A partial report may include slide level findings and may include several reportable features 504 with associated metadata. If there are relationships between the reportable features that may optimize the pathologist's review, they may be noted for display in the visualization. A reportable feature may include one or more feature instances 506 with accompanying metadata. Each feature instance may include a visualization type 508 and / or an interaction type 510.

[0078] A feature instance 506 may be an instance of a reportable feature 504 that is not connected to another reportable feature by tissue. The instances for each reportable feature 504 may be defined by diagnosis and anatomy. The feature instances 506 within a slide may be aggregated in several forms. A user may view some or all instances of a reportable feature 504 in one place (e.g., all tumors and invasive ductal carcinomas (IDC) as a tissue map, or all lesions of DCIS in a gallery view). If the feature instance 506 is associated with metadata, the metadata may be aggregated to create metadata for the reportable feature (e.g., Pattern 3 at 5mm and Pattern 4 at 5mm are aggregated for the entire tumor linear range (10mm)).

[0079] Table 2 further illustrates relevant information about feature instances such as feature instances 506 and instance collections. Within a feature instance, a user may "tissue hop" through instances or jump between instances on a slide. Instances viewed as isolated instances in the context of a slide may be viewed at high power settings. The corresponding visualization may be two or more lesion regions (if multiple elements are required to contain the instance) and surrounding context regions. An example may be a disease infiltration on tissue where multiple feature instances may be required to view. A user may also view feature instances using a gallery view. The gallery may include thumbnails of the feature instances displayed for quick review with the option for the user to jump to a particular instance by selecting a thumbnail. The gallery view may accommodate one or more lesion regions (although only one lesion region may be required to contain an instance).

[0080] For instance collections, all regions of an organizational type may be included in one aggregation group. All instances of that type may be viewed together at low power. The corresponding visualization type may include a context region. [Table 2]

[0081] Sub- (or specimen) level sets can be important to pathology workflows, especially when considering report validation.

[0082] All visualizations may have tissue hops as described in Table 2. Tissue hops may allow the user to "hop" between any visualizations of tissue types found across parts. For example, if a part contains three instances of an IDC, even though the IDC is an aggregate, tissue hops allow the user to move between those aggregate instances.

[0083] The case report 500 may be editable. Apart from any biomarkers, all alphanumeric output (either metadata or otherwise) may be editable. For example, measured endpoints may be editable and measurements may be updated during the reporting and / or editing states. Similarly, tissue map regions of future states may be editable. Calculated percentages or measured lengths may be updated during the reporting and / or editing states. Other metadata may not be recalculated.

[0084] 5 is not limited to a particular AI or image analysis and / or processing system, and the embodiments disclosed herein may be used in any type of AI or image analysis system. For example, multiple systems and / or modules may be used in the overall framework, and multiple system outputs may be visualized and integrated for an image, part, case, and / or patient.

[0085] 6A-6D show an exemplary diagram of how common visualization and interaction of different tissue types can be applied.

[0086] FIG. 6A shows an example of visualization of a general tissue specimen and organization of a prostate tissue specimen. Within the general tissue specimen, small regions of interest are identified as foci. Alternatively, small and large regions of interest may be identified as foci. A foci may be a single point of interest with some surrounding region of context. The context region of the general tissue specimen may include regions of interest of interest and / or tissue maps. This may be shown as a large region of interest, such as the length of an adjacent tumor on a slide (which may be more suitable for tissue biopsy). The ruler tool may be used to measure either large or small regions of interest with numerical output. The ruler may be used to measure one or more biomarkers within the specimen tissue. In slide-level visualization, the alphanumeric display may include textual and numerical outputs overlaying the slide. These outputs may be binary, discrete scores, continuous scores, categories, probabilities, numerical values ​​(including percentages), and / or report outputs in portable document format (PDF) or other formats. In the case of a PDF report, the PDF report may be a static report that cannot be easily edited. The PDF report may be associated with, for example, a molecular test, a CLIA laboratory test result, or other test or laboratory result that is intended to remain unedited or unmanipulated.

[0087] In the case of the prostate tissue specimen example of FIG. 6A, the visualization of multiple lesions may include the entire tumor and one or more patterns. The context region may vary depending on the lesion in question, for example, the entire tumor may include the entire length of the tumor, and the user may measure the summed tumor length if desired. The patterns may be described or measured by the user to include the percentage of tissue covered by the pattern. Additionally, the alphanumeric display may output the primary Gleason score and / or the secondary Gleason score.

[0088] Figure 6B shows an exemplary visualization of a breast tissue specimen. As in Figure 6A, a lesion area is selected, measured, and an alphanumeric output is obtained. Further examples of bladder tissue, colon tissue, and lung tissue are shown in Figures 6C and 6D.

[0089] FIG. 7 shows an exemplary diagram of reportable features according to an exemplary embodiment of the present disclosure. For example, in the case of an invasive cancer feature, the reportable feature may include a grade, including an indication of whether the grade is in a range considered good, moderate, or poor, and the length of the measured feature. The resulting resection indicator may instruct the user to resect only the edges of the tissue specimen. FIG. 7 includes various other exemplary reportable features, a more detailed description of which is provided in Tables 3-6 below.

[0090] As shown in FIG. 8, device 800 may include a central processing unit (CPU) 820. CPU 820 may be any type of processor device, including, for example, any type of special-purpose or general-purpose microprocessor device. As will be appreciated by those skilled in the art, CPU 820 may also be a single processor in a multi-core / multi-processor system, such systems operating alone or within a cluster of computing devices operating within a cluster or server farm. CPU 820 may be connected to a data communications infrastructure 810, such as a bus, a message queue, a network, or a multi-core message passing scheme.

[0091] The device 800 may also include a main memory 840, e.g., random access memory (RAM), and may also include a secondary memory 830. The secondary memory 830, e.g., read only memory (ROM), may be, for example, a hard disk or a removable storage drive. Such removable storage drives may include, for example, floppy disk drives, magnetic tape drives, optical disk drives, flash memory, and the like. The removable storage drives in this example read from and / or write to the removable storage units in a well-known manner. The removable storage may include floppy disks, magnetic tapes, optical disks, and the like that are read and written by the removable storage drives. As will be appreciated by those skilled in the art, such removable storage units generally include computer usable storage media having computer software and / or data stored thereon.

[0092] In alternative implementations, secondary memory 830 may include similar means for allowing computer programs or other instructions to be loaded into device 800. Examples of such means may include program cartridges and cartridge interfaces (such as those found in video game machines), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units, and interfaces that allow software and data to be transferred from the removable storage units to device 800.

[0093] Device 800 may also include a communications interface ("COM") 860. Communications interface 860 allows software and data to be transferred between device 800 and external devices. Communications interface 860 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. The software and data transferred via communications interface 860 may be in the form of signals, which may be electrical, electromagnetic, optical, or other signals capable of being received by communications interface 860. These signals may be provided to communications interface 860 over a communications path in device 800, which may be implemented using, for example, wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channel.

[0094] The hardware elements, operating systems, and programming languages ​​of such equipment are conventional in nature and are presumed to be sufficiently familiar to those skilled in the art. The device 800 may also include input / output ports 850 for connecting input / output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Of course, the various server functions may be implemented in a distributed manner on several similar platforms to distribute the processing load. Alternatively, the server may be implemented by appropriate programming of one computer hardware platform.

[0095] Throughout this disclosure, references to components or modules generally refer to items that can be logically grouped together to perform a function or group of related functions. Like reference numbers are intended to generally refer to the same or similar components. Components and modules may be implemented in software, hardware, or a combination of software and hardware.

[0096] The tools, modules, and functions described above may be executed by one or more processors. A "storage" type medium may include any or all of the tangible memory of a computer, processor, or associated modules thereof (such as various semiconductor memories, tape drives, disk drives, etc.), which may provide non-transitory storage for software programming at any time.

[0097] The software may be communicated over the Internet, a cloud service provider, or other telecommunications network. For example, the communication allows the software to be loaded from one computer or processor to another. As used herein, unless limited to non-transitory, tangible "storage" media, terms such as computer or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0098] The foregoing general description is exemplary and explanatory only and is not intended to limit the disclosure. Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only. [Table 3] [Table 4-1] [Table 4-2] [Table 4-3] [Table 5] [Table 6-1] [Table 6-2]

Table 6-3

Table 6-4

Claims

1. A computer-implemented method for processing electronic images associated with a pathology specimen, comprising: receiving one or more cases associated with a pathology specimen on a digital storage device; receiving a selection of a case of the one or more cases; Dividing the case into a plurality of portions and dividing one of the plurality of portions into a plurality of slides; applying a machine learning model to generate an interactive visualization of the plurality of slides, the machine learning model being trained by processing a plurality of training images; and determining image characteristics associated with the plurality of slides; determining at least one report for the image features; aggregating a plurality of feature instances of a reportable feature into a group, the reportable feature corresponding to at least one of the image features; aggregating the at least one report for the image feature into a partial report; Integrating at least one partial report into a case report; 4. A computer-implemented method comprising:

2. The computer-implemented method of claim 1, wherein the image features include classified and / or labeled areas or lesions of suspicious tissue, observations of the suspicious tissue, and / or feature instances.

3. The computer-implemented method of claim 2, wherein the image features include diagnostic features and / or anatomical features specific to reportable features, and associated metadata.

4. Displaying at least one thumbnail of the image features in a gallery view that includes reportable features; Selecting a thumbnail to jump to the image feature; The computer-implemented method of claim 2 , further comprising:

5. Displaying the group of the at least one feature instance together; and outputting a visualization of said group together with associated context regions; The computer-implemented method of claim 2 , further comprising:

6. Determining a relationship between two reportable features; detailing said relationship in said case report; The computer-implemented method of claim 1 , further comprising:

7. Identifying at least a metric associated with a reportable characteristic; including said at least one metric in said case report; The computer-implemented method of claim 1 , further comprising:

8. Moving on a slide between a selection of a visualization of a first feature instance or collection of feature instances and a selection of a visualization of a second feature instance or collection of feature instances; displaying said selection to a user; and The computer-implemented method of claim 1 , further comprising:

9. The computer-implemented method of claim 8, wherein the visualization of the feature instances includes one or more lesion regions.

10. The computer-implemented method of claim 1, wherein the report, the partial report, and / or the case report for the image features are editable.

11. The computer-implemented method of claim 2, wherein the feature instances are editable.

12. A system for using at least one machine learning model to process electronic images associated with a pathology specimen, comprising: at least one memory for storing instructions; at least one processor configured to execute the instructions to perform operations; Equipped with The operation includes: receiving one or more cases associated with a pathology specimen on a digital storage device; receiving a selection of a case of the one or more cases; Dividing the case into a plurality of portions and dividing one of the plurality of portions into a plurality of slides; applying a machine learning model to generate an interactive visualization of the plurality of slides, the machine learning model being trained by processing a plurality of training images; and determining image characteristics associated with the plurality of slides; determining at least one report for the image features; aggregating at least one feature instance of a plurality of reportable features into a group; aggregating the at least one report for the image feature into a partial report; Integrating at least one partial report into a case report; The system comprising:

13. The system of claim 12, wherein the image features include classified and / or labeled areas or lesions of suspicious tissue, observations of the suspicious tissue, and / or feature instances.

14. The system of claim 13, wherein the image features include diagnostic features and / or anatomical features specific to reportable features and associated metadata.

15. The operation displaying at least one thumbnail of the image features in a gallery view containing the reportable features; Selecting a thumbnail to jump to the image feature; The system of claim 13 further comprising:

16. The operation displaying the group of the at least one feature instance together; and outputting a visualization of said group together with associated context regions; The system of claim 13 further comprising:

17. The operation Determining a relationship between two reportable characteristics; detailing said relationship in said case report; The system of claim 12 further comprising:

18. A non-transitory computer-readable medium storing instructions that, when executed by a processor, perform a method for processing an electronic image associated with a pathology specimen, the method comprising: receiving one or more cases associated with a pathology specimen on a digital storage device; receiving a selection of a case of the one or more cases; Dividing the case into a plurality of portions and dividing one of the plurality of portions into a plurality of slides; applying a machine learning model to generate an interactive visualization of the plurality of slides, the machine learning model being trained by processing a plurality of training images; and determining image characteristics associated with the plurality of slides; determining at least one report for the image features; Aggregating at least one feature instance of a reportable feature into a group; aggregating the at least one report for the image feature into a partial report; Integrating at least one partial report into a case report; The non-transitory computer-readable medium.

19. The non-transitory computer-readable medium of claim 18, wherein the image features include classified and / or labeled areas or lesions of suspicious tissue, observations of the suspicious tissue, and / or feature instances.

20. The method of claim 1, moving on the slide between a selection of a visualization of a first feature instance or collection of feature instances and a selection of a visualization of a second feature instance or collection of feature instances; displaying said selection to a user; and 20. The computer readable medium of claim 18, further comprising: