System and method for processing electronic images of slides for digital pathology workflows

Abstract

A computer-implemented method of using a machine learning model to classify samples in digital pathology may include receiving one or more cases, each associated with a digital image of a pathology specimen, identifying the cases as ready to be viewed using the machine learning model, receiving a selection of a case, the case including a plurality of parts, determining whether the plurality of parts is suspicious or unsuspected using the machine learning model, receiving a selection of a part of the plurality of parts, determining whether a plurality of slides associated with the one part are suspicious or unsuspected, and determining a collection of suspicious slides of the plurality of slides using the machine learning model, the machine learning model being trained by processing a plurality of training images.

Description

Technical Field

[0001] Related Applications This application claims the priority of U.S. Provisional Patent Application No. 63 / 127,846, filed on December 18, 2020, the entire disclosure of which is incorporated herein by reference.

[0002] Various embodiments of the present disclosure generally relate to methods of image processing. More specifically, certain embodiments of the present disclosure classify artificial intelligence (AI) visualization attributes to create a reusable and scalable framework for AI-enabled visualization and interaction, resulting in a digital workflow that utilizes AI and is useful for pathologists making diagnoses.

Background Art

[0003] There are a wide variety of tissue and surgical specimen types that may require pathological review and diagnosis. Examples of reportable features and instances that a pathologist may need to view, review, and diagnose may include features and examples within areas such as bladder, colon, lung, dermatology, and stomach. It may be useful to provide visualization of some areas (e.g., bladder, colon, lung, dermatology, and stomach features) for pathologists.

[0004] The background description provided herein is for the purpose of generally presenting the context of the present disclosure. Unless otherwise indicated herein, the materials described in this section are not prior art to the claims of the present application and are not admitted to be prior art or suggestions of prior art by inclusion in this section.

Summary of the Invention

Means for Solving the Problems

[0005] According to certain aspects of the present disclosure, a system and method for processing an electronic image of a slide for a digital pathology workflow are disclosed.

[0006] A computer implementation of using a machine learning model to classify samples in digital pathology may include receiving one or more cases in a digital storage device, each associated with a digital image of a pathological specimen; using a machine learning model to identify one of the one or more cases as ready for viewing; receiving a selection of cases, where the case includes multiple parts; using a machine learning model to determine whether the multiple parts are suspicious or not; receiving a selection of one part from the multiple parts; determining whether the multiple slides associated with that one part are suspicious or not; using a machine learning model to determine a set of suspicious slides from the multiple slides, where the machine learning model has been trained by processing multiple training images; annotating the set of suspicious slides; and / or generating a report based on the set of suspicious slides.

[0007] Identifying a case as ready for viewing may include verifying that all slides in the case have been processed and uploaded to a digital storage device. Annotating a collection of suspicious slides using the techniques presented herein may further include outlining at least one area around the suspicious tissue (or, instead of or in addition to the suspicious tissue), measuring the length and / or area of ​​the suspicious tissue (or non-suspicious tissue), and outputting annotations to the collection of suspicious slides.

[0008] The method may further include entering information about a collection of suspicious slides (and / or non-suspicious slides) into a report and outputting the report to the user. Information about a collection of suspicious slides may include lesion areas, context areas, one or more measurements of the suspicious tissue, alphanumeric output, and / or a report compiled based on all of the lesion areas, context areas, and / or measurements. The alphanumeric output may include a binary representation of the presence of one or more biomarkers in the tissue. The report and / or visualization of the suspicious tissue may include detection panels, quantification panels, and / or annotation logs. The annotation logs may be searchable at the case level, sub-level, and / or slide level.

[0009] The method may further include determining the complexity associated with each of one or more cases and prioritizing one or more cases based on the determined complexity. The method may further include using a machine learning model to sort and / or filter multiple parts for display based on a determination of whether multiple parts are suspicious or not. The method may further include determining the pathological type and generating a report based on the determined pathological type.

[0010] The method may further include using a machine learning model to determine multiple features based on digital images associated with a case, using a machine learning model to determine multiple partial reports using the multiple features, and determining a report based on the multiple partial reports.

[0011] A system for using machine learning models to classify samples in digital pathology may include at least one memory for storing instructions and at least one processor configured to execute instructions and perform operations. The operations may include receiving one or more cases in a digital storage device, each associated with a digital image of a pathological specimen; using a machine learning model to identify one of the one or more cases as ready for viewing; receiving a selection of a case, wherein the case comprises multiple parts; using a machine learning model to determine whether the multiple parts are suspicious or not; receiving a selection of one of the multiple parts; determining whether the multiple slides associated with that one part are suspicious or not; using a machine learning model to determine a set of suspicious slides from the multiple slides, wherein the machine learning model has been trained by processing multiple training images; annotating the set of suspicious slides; and / or generating a report based on the set of suspicious slides.

[0012] Identifying a case as ready for viewing may include verifying that all slides in the case have been processed and uploaded to a digital storage device. Annotating a collection of suspicious slides may further include outlining at least one area around the suspicious tissue, measuring the length and / or area of ​​the suspicious tissue, and outputting annotations to the collection of suspicious tissue. The operation may further include entering information about the collection of suspicious slides into a report and outputting the report to the user. Information about the collection of suspicious slides may include lesion area, context area, one or more measurements of the suspicious tissue, and / or alphanumeric output.

[0013] The operation may further include determining the complexity associated with each of one or more cases, and prioritizing one or more cases based on the determined complexity. The operation may further include using a machine learning model to determine multiple features based on digital images associated with the cases, using a machine learning model to determine multiple partial reports using the multiple features, and determining a report based on the multiple partial reports.

[0014] Non-temporary computer-readable media may store instructions for a method that, when executed by a processor, uses a machine learning model to output task-specific predictions. The method may include receiving one or more cases in a digital storage device, each associated with a digital image of a pathological specimen; using a machine learning model to identify one of the one or more cases as ready for viewing; receiving a selection of a case, wherein the case comprises multiple parts; using a machine learning model to determine whether the multiple parts are suspicious or not; receiving a selection of one of the multiple parts; determining whether the multiple slides associated with that one part are suspicious or not; using a machine learning model to determine a set of suspicious slides from the multiple slides, wherein the machine learning model has been trained by processing multiple training images; annotating the set of suspicious slides; and / or generating a report based on the set of suspicious slides.

[0015] Annotating a collection of suspicious slides may further include outlining at least one area around the suspicious tissue, measuring the length and / or area of ​​the suspicious tissue, and outputting annotations to the collection of suspicious tissue.

[0016] It should be understood that both the foregoing description and the following detailed description are merely illustrative and not restrictive of the disclosed embodiments.

[0017] The accompanying drawings, which are incorporated herein and form a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments. The present invention provides, for example, the following: (Item 1) A computer implementation method for using at least one machine learning model to classify samples in digital pathology, Each of these involves receiving one or more cases associated with digital images of pathological specimens in a digital storage device, Using the aforementioned machine learning model, one of the one or more cases is identified as ready for viewing. Receiving the selection of the aforementioned case, wherein the aforementioned case includes multiple parts, Using the aforementioned machine learning model, determine whether the multiple parts are suspicious or not, Receiving the selection of one of the aforementioned multiple parts, Determining whether multiple slides associated with the aforementioned single section are suspicious or not, The process involves using the aforementioned machine learning model to determine a set of suspicious slides from among the multiple slides, wherein the machine learning model is trained by processing multiple training images. Annotating the aforementioned collection of suspicious slides, and / or generating a report based on the aforementioned collection of suspicious slides. The computer implementation method, including the above. (Item 2) The computer implementation method described in item 1, wherein identifying a case as ready for viewing includes verifying that all slides within the case have been processed and uploaded to a digital storage device. (Item 3) Annotating the aforementioned collection of suspicious slides is Outline at least one area around the suspicious organization, Measuring the length and / or area of ​​the suspected tissue, Outputting annotations to the aforementioned collection of suspicious slides The computer implementation method described in item 1, further including the method described in item 1. (Item 4) Enter the information about the aforementioned collection of suspicious slides into the report, Outputting the aforementioned report to the user The computer implementation method described in item 1, further including the method described in item 1. (Item 5) The computer implementation method described in item 4, wherein the information relating to the aforementioned collection of suspicious slides includes one or more measurements of the lesion area, context area, and / or alphanumeric output. (Item 6) The computer implementation method according to item 5, wherein the alphanumeric output includes a binary representation of the presence of one or more biomarkers in the tissue. (Item 7) The computer implementation method described in item 4, wherein the report and / or visualization of the suspicious organization includes a detection panel, a quantification panel, and / or an annotation log. (Item 8) The computer implementation method described in item 7, wherein the annotation log is searchable at the case level and at the partial level. (Item 9) To determine the complexity associated with each of the one or more cases, The computer implementation method according to item 1, further comprising prioritizing one or more cases based on the complexity determined above. (Item 10) Using the machine learning model, sort and / or fill in the parts for display based on the determination of whether the parts are suspicious or not. The computer implementation method described in item 1, further including the method described in item 1. (Item 11) Determining the pathological type, To generate the report based on the determined pathological type. The computer implementation method described in item 1, further including the method described in item 1. (Item 12) Using the aforementioned machine learning model, multiple features are determined based on the digital images associated with the case, Using the aforementioned machine learning model, multiple partial reports are determined using the aforementioned multiple features, The report is determined based on the aforementioned multiple partial reports. The computer implementation method described in item 1, further including the method described in item 1. (Item 13) A system for using at least one machine learning model to classify samples in digital pathology, At least one memory to store instructions, The system comprises at least one processor configured to execute the aforementioned instructions and perform an operation, The aforementioned operation is, Each of these involves receiving one or more cases associated with digital images of pathological specimens in a digital storage device, Using the aforementioned machine learning model, one of the one or more cases is identified as ready for viewing. Receiving the selection of the aforementioned case, wherein the aforementioned case includes multiple parts, Using the aforementioned machine learning model, determine whether the multiple parts are suspicious or not, Receiving the selection of one of the aforementioned multiple parts, Determining whether multiple slides associated with the aforementioned single section are suspicious or not, The process involves using the aforementioned machine learning model to determine a set of suspicious slides from among the multiple slides, wherein the machine learning model is trained by processing multiple training images. Annotating the aforementioned collection of suspicious slides, and / or generating a report based on the aforementioned collection of suspicious slides. The system including the above. (Item 14) The system described in item 13, which identifies a case as ready for viewing, includes verifying that all slides in the case have been processed and uploaded to a digital storage device. (Item 15) Annotating the aforementioned collection of suspicious slides is Outline at least one area around the suspicious organization, Measuring the length and / or area of ​​the suspected tissue, Outputting annotations to the aforementioned collection of suspicious slides The system described in item 13, which further includes the system described in item 13. (Item 16) The aforementioned operation, Enter the information about the aforementioned collection of suspicious slides into the report, Outputting the report to the user, wherein the information about the collection of suspicious slides includes a lesion area, a context area, one or more measurements of the suspicious tissue, and / or alphanumeric output. The system described in item 13, which further includes the system described in item 13. (Item 17) The aforementioned operation, To determine the complexity associated with each of the one or more cases, The system according to item 13, further comprising prioritizing one or more cases based on the complexity determined above. (Item 18) The aforementioned operation, Using the aforementioned machine learning model, multiple features are determined based on the digital images associated with the case, Using the aforementioned machine learning model, multiple partial reports are determined using the aforementioned multiple features, The report is determined based on the aforementioned multiple partial reports. The system described in item 13, which further includes the system described in item 13. (Item 19) A non-temporary computer-readable medium that stores instructions for a method that, when executed by a processor, uses at least one machine learning model to produce task-specific predictions, wherein the method Each of these involves receiving one or more cases associated with digital images of pathological specimens in a digital storage device, Using the aforementioned machine learning model, one of the one or more cases is identified as ready for viewing. Receiving the selection of the aforementioned case, wherein the aforementioned case includes multiple parts, Using the aforementioned machine learning model, determine whether the multiple parts are suspicious or not, Receiving the selection of one of the aforementioned multiple parts, Determining whether multiple slides associated with the aforementioned single section are suspicious or not, The process involves using the aforementioned machine learning model to determine a set of suspicious slides from among the multiple slides, wherein the machine learning model is trained by processing multiple training images. Annotating the aforementioned collection of suspicious slides, and / or generating a report based on the aforementioned collection of suspicious slides. The non-temporary computer-readable media, including the above. (Item 20) Annotating the aforementioned collection of suspicious slides is Outline at least one area around the suspicious organization, Measuring the length and / or area of ​​the suspected tissue, Outputting annotations to the aforementioned collection of suspicious slides Non-temporary computer-readable media as described in item 19, including, furthermore. [Brief explanation of the drawing]

[0018] [Figure 1A] This disclosure illustrates an exemplary block diagram of a system and network for classifying artificial intelligence (AI) visualization attributes and creating an AI-powered digital workflow framework, according to an exemplary embodiment of this disclosure. [Figure 1B] An exemplary block diagram of a disease detection platform according to an exemplary embodiment of the present disclosure is shown. [Figure 1C] An exemplary block diagram of a slide analysis tool according to an exemplary embodiment of the present disclosure is shown. [Figure 2] This disclosure illustrates an exemplary AI user clinical workflow according to an exemplary embodiment of this disclosure. [Figure 3] An exemplary diagram of slide-level visualization according to an exemplary embodiment of the present disclosure is shown. [Figure 4A] The exemplary embodiments of this disclosure provide an exemplary report of various reportable features and examples that a pathologist may need to see, review, and diagnose. [Figure 4B] The exemplary embodiments of this disclosure provide an exemplary report of various reportable features and examples that a pathologist may need to see, review, and diagnose. [Figure 4C] The exemplary embodiments of this disclosure provide an exemplary report of various reportable features and examples that a pathologist may need to see, review, and diagnose. [Figure 4D] The exemplary embodiments of this disclosure provide an exemplary report of various reportable features and examples that a pathologist may need to see, review, and diagnose. [Figure 4E] The exemplary embodiments of this disclosure provide an exemplary report of various reportable features and examples that a pathologist may need to see, review, and diagnose. [Figure 5] This disclosure illustrates an exemplary diagram of an overall framework for AI-enabled visualizations based on exemplary embodiments of this disclosure. [Figure 6A] This diagram illustrates how common visualizations and interactions across different organizational types may be applied according to exemplary embodiments of the present disclosure. [Figure 6B] This diagram illustrates how common visualizations and interactions across different organizational types may be applied according to exemplary embodiments of the present disclosure. [Figure 6C] This diagram illustrates how common visualizations and interactions across different organizational types may be applied according to exemplary embodiments of the present disclosure. [Figure 6D] This diagram illustrates how common visualizations and interactions across different organizational types may be applied according to exemplary embodiments of the present disclosure. [Figure 7] The following are illustrative diagrams of reportable features according to exemplary embodiments of this disclosure. [Figure 8] This specification shows an exemplary system capable of performing the technologies presented herein. [Modes for carrying out the invention]

[0019] The following describes exemplary embodiments of the present disclosure in detail, examples of which are shown in the accompanying drawings. Where possible, the same reference numerals are used throughout the drawings to indicate the same or similar parts.

[0020] The systems, devices, and methods disclosed herein are described in more detail, by reference to the drawings, as examples. The examples described herein are for illustrative purposes only and are provided to aid in the description of the apparatus, devices, systems, and methods described herein. None of the features or components illustrated in the drawings or described below should be construed as essential for any particular implementation of any of these devices, systems, or methods unless specifically designated as essential.

[0021] Furthermore, with respect to any method described, regardless of whether the method is described in relation to a flowchart, unless otherwise specified or required by the context, any explicit or implicit ordering of the steps performed in the execution of the method does not imply that those steps must be performed in the order presented, but rather that they may be performed in a different order or in parallel.

[0022] As used herein, the term “exemplary” is used in the sense of “example” rather than “ideal.” Furthermore, the terms “a” and “an” herein do not imply a limitation of quantity, but rather that one or more of the items mentioned exist.

[0023] Figure 1A shows an exemplary block diagram of a system and network for classifying artificial intelligence (AI) visualization attributes and creating an AI-powered digital workflow framework, according to an exemplary embodiment of the present disclosure.

[0024] Specifically, Figure 1A shows an electronic network 120 that may be connected to servers in a hospital, laboratory, and / or clinic, etc. For example, a physician server 121, a hospital server 122, a clinical trial server 123, a laboratory server 124, and / or a laboratory information system 125, etc., may each be connected to the electronic network 120, such as the Internet, via one or more computers, servers, and / or handheld mobile devices. According to exemplary embodiments of the present application, the electronic network 120 may also be connected to a server system 110 which may include a processing unit configured to implement a disease detection platform 100, which includes a slide analysis tool 101 for determining information on specimen properties or image properties relating to digital pathology images(s), and for using machine learning to determine whether a disease or infectious agent is present, in accordance with exemplary embodiments of the present disclosure. The slide analysis tool 101 can enable rapid assessment of the "appropriateness" of liquid-based tumor formulations, facilitate the diagnosis of liquid-based tumor formulations (cytology, hematology / hematopathology), and predict molecular findings that are likely to be detected in various tumors detected by liquid-based formulations.

[0025] The physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125 may create or otherwise obtain images of one or more cytological specimens, histopathological specimens, slides of cytological specimens, digitized images of slides of histopathological specimens, or any combination thereof from one or more patients. The physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, and past biomedical or cytological information. The physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125 may transmit digitized slide images and / or patient-specific information to the server system 110 via the electronic network 120. The server system 110 may include one or more storage devices 109 for storing images and data received from at least one of the physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125. The server system 110 may also include a processing unit for processing the images and data stored in the storage devices 109. The server system 110 may further include one or more machine learning tools or machine learning functions. For example, the processing unit may include a machine learning tool for a disease detection platform 100, according to one embodiment. Alternatively or further, the present disclosure (or parts of the systems and methods of the present disclosure) may be run on a local processing unit (e.g., a laptop).

[0026] The physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125 refer to systems used by pathologists to review slide images. In a hospital environment, tissue type information may be stored in the laboratory information system 125.

[0027] Figure 1B shows an exemplary block diagram of a disease detection platform 100 for determining image characteristic information regarding specimen characteristics or digital pathology images(s) using machine learning. The disease detection platform 100 may include a slide analysis tool 101, a data acquisition tool 102, a slide acquisition tool 103, a slide scanner 104, a slide manager 105, storage 106, a laboratory information system (e.g., laboratory information system 125), and a browsing application tool 108.

[0028] The slide analysis tool 101 refers to a process and system for determining information on data variable characteristics or health variable characteristics related to digital pathology images(s), as described below. Machine learning may be used to classify images according to exemplary embodiments. The slide analysis tool 101 may also predict future relationships, as described in the embodiments below.

[0029] According to exemplary embodiments, the data acquisition tool 102 can facilitate the transfer of digital pathology images to various tools, modules, components, and devices used for classifying and processing digital pathology images.

[0030] According to an exemplary embodiment, the slide acquisition tool 103 can scan pathological images and convert them into a digital format. The slides may be scanned by a slide scanner 104, and the slide manager 105 may process the images on the slides into digitized pathological images and store the digitized images in storage 106.

[0031] According to exemplary embodiments, the browsing application tool 108 may provide the user with information on the characteristics of specimens or image characteristics of digital pathology images. This information may be provided through various output interfaces (e.g., screens, monitors, storage devices, and / or web browsers).

[0032] The slide analysis tool 101, and one or more of its components, may transmit and / or receive digitized slide images and / or patient information via the network 120 to the server system 110, physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125. Furthermore, the server system 110 may include a storage device for storing images and data received from at least one of the slide analysis tool 101, data acquisition tool 102, slide acquisition tool 103, slide scanner 104, slide manager 105, and browsing application tool 108. The server system 110 may also include a processing unit for processing the images and data stored in the storage device. The server system 110 may further include one or more machine learning tools or machine learning functions, for example, for the processing unit. Alternatively or further, the present disclosure (or parts of the systems and methods of the present disclosure) may be run on a local processing unit (e.g., a laptop).

[0033] Any of the above devices, tools, and modules may be located on a device that can be connected to an electronic network, such as the Internet or a cloud service provider, via one or more components, servers, and / or handheld mobile devices.

[0034] Figure 1C shows an exemplary block diagram of a slide analysis tool 101 according to an exemplary embodiment of the present disclosure. The slide analysis tool 101 may include a training image platform 131 and / or a target image platform 135.

[0035] According to one embodiment, the training image platform 131 may include a training image acquisition module 132, a data analysis module 133, and a tissue identification module 134.

[0036] According to one embodiment, the training data platform 131 may create or receive training images used to train a machine learning model to effectively analyze and classify digital pathology images. For example, training images may be received from any one or any combination of the server system 110, physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125. Images used for training may be obtained from real sources (e.g., humans, animals, etc.) or from synthetic sources (e.g., graphic rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains such as H&E, hematoxylin alone, IHC, molecular pathology, etc., and / or (b) digitized tissue samples from a 3D imaging device such as microCT.

[0037] The training image acquisition module 132 may create or receive a dataset containing one or more training datasets corresponding to one or more specimen tissues. For example, training datasets may be received from one or any combination of the server system 110, physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125. This dataset may be stored on a digital storage device. The data analysis module 133 may identify whether a set of individual cells belongs to cells of interest or to the background of the digitized image. The tissue identification module 134 may analyze the digitized image and determine whether individual cells in a cytological sample require further analysis. Identifying whether individual cells require further analysis and integrating these regions is useful, and the identification of such cells may trigger an alert to the user.

[0038] According to one embodiment, the target image platform 135 may include a target image acquisition module 136, a sample detection module 137, and an output interface 138. The target image platform 135 may receive target images and apply a machine learning model to the received target images to determine the features of the target dataset. For example, the target data may be received from any one or any combination of the server system 110, physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125. The target image acquisition module 136 may receive a target dataset corresponding to a target tissue sample. The sample detection module 137 may apply a machine learning model to the target dataset to determine the features of the tissue sample. For example, the sample detection module 137 may detect suspicious (and / or non-suspicious) tissue regions within the tissue sample. The sample detection module 137 may use an artificial intelligence (AI) system trained to identify the presence and / or absence of various features. The AI ​​system may output a map (e.g., a tissue map) that highlights all areas associated with the identified feature or instance of the feature. U.S. Patent Application No. 17 / 313,617 is incorporated herein by reference in its entirety. The specimen detection module 137 may determine whether the identified feature is suspicious or not based on specific features such as size, shape, location, proximity to other identified features, disease type, color, type of stain, type of biomarker, gene signature, type of protein, blood marker, tissue type, tissue texture, presence or level of calcification, or presence or level of inflammation.

[0039] The sample detection module 137 can also apply a machine learning model to a target dataset to determine the quality score of a target tissue sample. Furthermore, the sample detection module 137 can apply a machine learning model to a target image to determine whether a target element is present in the tissue sample.

[0040] The output interface 138 may be used to output information about the target tissue sample and the tissue area of ​​interest (for example, to a screen, monitor, storage device, web browser, etc.).

[0041] One or more of the exemplary embodiments described below may provide a set of AI visualization types and a set of interaction types. One or more exemplary embodiments may be sorted by tissue type and provide visualization types and any interaction types that may be required for reportable features extracted from the College of American Pathologists (CAP) overview. One or more exemplary embodiments may incorporate clinical use cases but may also be applied to biomarker products that can utilize one or more defined outputs.

[0042] Partial and case-level sets may be considerations when looking at a pathologist's workflow. Pathologists can use partial-level sets to report and diagnose tissue specimens at a partial level. However, pathology reports may not include slide-level observations. In some cases, reporting fields may not be displayed at the slide level. Some reporting fields may only be displayed at the partial level, such as the slide number for a portion of ductal carcinoma in situ (DCIS).

[0043] The following definitions are for illustrative purposes only and are not intended to be restrictive. A whole slide image (WSI) may refer to one or more images of stained or unstained tissue. A reportable feature may refer to a classified or labeled area or lesion of suspected tissue, and any observation of tissue used for diagnosis. A feature instance may be a reference to a reportable feature defined by diagnostic and anatomical features that may be specific to that reportable feature. A visualization type may refer to a display category for a feature instance. An interaction type may refer to a way in which a user can navigate between two or more feature instances. An instance definition may refer to one or more anatomical features that may be required to be included in an instance display. An AI system or AI module may refer to one or more modules that implement AI, machine learning techniques, and / or machine learning algorithms, and is not necessarily limited to a single module, device, system, platform, etc. The embodiments disclosed herein may be used in any type or configuration of AI or machine learning systems, modules, platforms, and / or image analysis and / or process systems. Various outputs may be integrated. A machine learning model may refer to a model or process that implements machine learning techniques (for example, to recognize patterns), and may not be limited to a single model. The embodiments disclosed herein may use multiple machine learning models that can be combined and / or produce individual outputs that can subsequently be combined and / or integrated.

[0044] Figure 2 shows an exemplary AI user workflow 200 in a clinical setting according to an exemplary embodiment. The exemplary workflow may begin with step 202 (Clinical User Workflow Description), where case triage and / or prioritization may be performed. The user may wish to be alerted when a case is ready for viewing, i.e., when all slides associated with the case have been processed and / or uploaded to the system. The user may also wish to know which cases should be prioritized, for example, based on complexity, and / or which cases may require additional instructions, such as sending to molecular testing or re-cutting.

[0045] Complexity may be determined by the diagnosis (for example, an AI system might classify a slide as Gliason 3+4=7 with relatively low confidence, or it might determine that the percentage probability of Gliason pattern 3 is 49.9%, while the percentage of Gliason pattern 4 is 50.1%, indicating a borderline case), and / or by the need for additional testing beyond H&E slides to determine the diagnosis (for example, if the symptoms are thought to be invasive and additional stained images are needed for review, such as estrogen receptor (ER), progesterone receptor (PR), and human epidural growth factor receptor 1 (HER2)).

[0046] Complexity may be determined based on the combination of AI output and known workflows, as well as the examination steps required by the pathologist. Complex cases may be prioritized earlier on the day and / or assigned to a more experienced pathologist.

[0047] In step 204, the workflow may include, for example, the selection of a case from the user.

[0048] In step 206, the workflow may include a triage and / or case summary. Similar to step 202, the user may wish to see a brief summary of the case or triage information. During and / or before this summary step in 202, identified features and any other metadata (e.g., user-generated annotations) may be summarized.

[0049] In step 208, the workflow may include a selection of a portion. The selection may have an indicator that tells you whether the portion looks suspicious, not suspicious, or “other,” where “other” may indicate a pre-programmed quality or an unknown variable.

[0050] Step 210 may include a triage and / or sub-summary step. This summary may include a sub-index. Users may want to see an overview of what is in a sub-summary but do not necessarily need to view slides. By focusing at the sub-level, users may be able to see images / visualizations of key areas of the lesion and / or key areas.

[0051] In step 212, the workflow may include selecting a slide. There may be accompanying indicators available that can tell you whether the part looks suspicious, not suspicious, or "other," where "other" may indicate a pre-programmed characteristic or an unknown variable.

[0052] In step 214, the workflow may include diagnosing at a partial level. The diagnosis may include a partial index along with the diagnosis. The AI ​​may consolidate parts of the pathology specimen to create a partial index. Since the user needs to create a report at a partial level, the consolidated partial-level index can streamline the user's process for generating the report.

[0053] In step 216, the workflow may end with the case sign-out.

[0054] During the execution of the case workflow, the AI ​​may further generate a report on the examined slides and the resulting diagnoses. In step 218, the workflow may include displaying visualization and / or quantification panels based on the results of the partial-level diagnosis.

[0055] In step 220, the visualization and / or quantification panel can be edited and / or added to the report.

[0056] In step 222, the workflow may send a report to the user. All reporting may occur at a partial level. Reportable features may vary by histological type and between biopsy and excision within the same histological type. The histological type and procedure may determine which slide-level data can be sent to a partial-level report. For reporting, it may be necessary to integrate the data across the entire partial by averaging the data that varies across the slides. In addition, the presence or absence of features may be reported. There may be some reporting features that are not present at the slide level. For example, in a breast histological examination, the pathologist may report how many slides in the partial have DCIS. Annotations or outputs from the AI ​​workflow, such as relevant screenshots and / or identified areas, may be added to the report.

[0057] Furthermore, in step 224, the report can be used to input the Laboratory Information System (LIS) report in an exportable format such as PDF.

[0058] Table 1 shows an example of a slide-level visualization framework. Lesion areas on a slide may be described using a rectangular indicator overlay above the slide itself, and contextual areas may be described using a tissue map or, otherwise, a contour area. Anatomical elements within the lesion, as well as a set of coordinates, size, and / or zoom levels associated with the lesion area, may be helpful for the effectiveness of the display. Metadata may accompany the slide-level visualization and may include the name of the lesion area (e.g., "calcification"), metrics, and / or text output, and the size of the lesion area.

[0059] Since the number of instances can be important for some visualization types, a context area may consolidate any or all instances across the entire slide. Potential accompanying metadata may include the name of the context area (e.g., "Breast Subtype"), as well as metrics and / or text output that may include the name of the subtype, the size and / or measurements of the context area, the grade associated with the context area, and the architecture of the context area.

[0060] Measurements from a tissue sample may be visualized as lines with clearly labeled endpoints. Measurements may be accompanied by a name (e.g., "prostate" or "tumor length") as well as a metric and / or text output (e.g., numerical measurements of the line).

[0061] Additionally, text or numerical output may be included in slide-level visualizations and displayed directly in the visualizations. The text or numerical output may include metadata such as names or other metrics and / or text output associated with tissue samples or separate annotations added by the user during the review. Metadata may include, for example, detected biomarkers.

[0062] Edited reports of visualizations may be included in slide-level visualizations and may be formatted by external vendors. Edited reports may be associated with WSI or specific areas of interest. For example, a report such as a PDF report may include slide IDs and overall scores for slides, among other relevant information. [Table 1]

[0063] Figure 3 shows an illustrative diagram of the overall framework for AI-enabled visualization. A slide-level visualization 300 may include a case summary 302, a partial index 304, a detection panel 306, a quantification panel 308, and one or more exportable reports 310. Furthermore, the framework may include an annotation log 312 and a report builder 316, which together may form a report enabler.

[0064] In Case Summary 302, the framework may include AI-driven case summary and triage information available to the user before the user begins reviewing the slides. The case summary may include, for example, an acquisition identification number or other type of identifier, status (e.g., ready, reviewed, not ready), access date, patient name, medical record number or MRN, tissue type (e.g., breast, dermis or skin, gastrointestinal or digestive tract, or prostate), specimen type (e.g., biopsy), and the number of slides. Another part of the case summary may include visualizations, snapshots, or a “slide tray” of slides, which may include an identifier (e.g., “right breast”), stain type (e.g., H&E), and other information.

[0065] Partial index 304 may include AI-driven partial summaries and triage information available to the user before they begin reviewing the slides.

[0066] The detection panel 306 may be a templated panel that enables a binary classification of the organization as either suspicious or not suspicious.

[0067] The quantification panel 308 may include quantification, sorting, and / or filtering of tissue features and / or other reportable features. The quantification, sorting, and / or filtering may be based on suspicious and non-suspicious judgments and / or other judgments. As an example of quantification, in a prostate case, tumor volume may be quantified as a percentage and / or as a metric such as distance or area in millimeters (mm). Individual instances of a feature (e.g., perineural infiltration) may also be sorted and / or filtered according to the probability or likelihood of severity based on an AI system that detects the instances. For example, during filtering, individual instances with a low (e.g., below a threshold) probability of severity may be removed before the sorted and / or filtered features are output. The quantification panel 308 may be joined with the detection panel 306 as a single panel.

[0068] One or more exportable reports 310 may be output from the digitization panel 308 and the detection panel 306. The exportable reports 310 may be in PDF format.

[0069] The annotation log 312 may contain an activity log of all user-generated notes associated with the tissue specimen. The annotation log 312 may be searchable at both the case level and the sub-level, and each annotation or user-generated note may include a timestamp, the user's name and role, a thumbnail image, and / or one or more additional user comments.

[0070] The annotation log 312 can be combined with the report builder 316 to create the report enabler 314. The report builder 316 can be pre-populated and may be useful when used with AI components. Furthermore, the report builder 316 may be editable and can extract information and annotations from the annotation log 312.

[0071] Report Enabler 314 may include a subset of user-generated annotations and an AI pre-filled report. The subset and / or AI pre-filled report may be integrated with a clinical system and recorded in the patient's medical record. Pathologists may review the subset and / or AI pre-filled report and edit or annotate it in an on-canvas visualization. Reportable feature categories may be associated with the summary report field, so when a pathologist makes these edits or annotations, the AI ​​pre-filled report may be automatically updated based on the edits. Furthermore, when a pathologist makes these edits or annotations, other features related to the edited features (e.g., child features) may be automatically updated or adjusted based on the edits.

[0072] There are a wide variety of histological and surgical specimen types that may require pathological review and diagnosis. Figures 4A–4E show illustrative reports of various reportable features and examples that a pathologist may need to examine, review, and diagnose.

[0073] Figure 4A shows an exemplary report of a bladder tissue specimen. At least one primary feature is reported, such as classification, presence or absence of invasion, and subtype, including whether the tissue specimen is benign, in situ, or invasive. One or more secondary features, such as tumor grade, binary index of the muscularis propria, and / or depth of invasion, may also be included in the report. User notes may also be included in the exemplary report.

[0074] Similarly, Figure 4B shows an exemplary report of a colon tissue specimen reporting the first and second features. The first features may include the detection of cancer, identification of granulomas and acute inflammation, and non-cancerous subtypes. Non-cancerous subtypes may be further reported as normal, polyp, or inflammatory, and additional information such as subtype polyps may also be included in the report. The second features may include a prediction of the H&E MMR status.

[0075] Figures 4C, 4D, and 4E show exemplary reports of lung tissue, skin tissue, and gastric tissue, respectively. The exemplary reports may include various reported primary and secondary features, depending on the tissue specimen and the classification or subtype of the tissue.

[0076] Figure 5 shows an illustrative diagram of the overall framework for AI-enabled visualization. The overall framework may include two different sections, several slide-level findings, and a collection of findings by the AI ​​system for creating an overall case report.

[0077] A case report 500 may contain n parts in a particular case at hand. Each part may have its own corresponding part report 502. A part report may contain slide-level findings and several reportable features 504 with associated metadata. Where there are relationships between reportable features that can optimize the pathologist's review, they may be noted for display in the visualization. A reportable feature may contain one or more feature instances 506 with accompanying metadata. Each feature instance may contain a visualization type 508 and / or interaction type 510.

[0078] Feature instance 506 may be an instance of a reportable feature 504 that is not linked to another reportable feature by the tissue. Instances for each reportable feature 504 may be defined by diagnosis and anatomy. Feature instance 506 within a slide may be integrated in several forms. A user may view some or all instances of a reportable feature 504 in one place (e.g., whole tumor and invasive ductal carcinoma (IDC) as a tissue map, or all lesions of DCIS in a gallery view). If feature instance 506 is associated with metadata, the metadata may be integrated to create metadata for the reportable feature (e.g., pattern 3 and pattern 4 at 5 mm are integrated for the whole tumor linear range (10 mm)).

[0079] Table 2 provides further relevant information about feature instances, such as feature instance 506 and instance sets. Within a feature instance, users can "tissue hop" through instances or jump between instances on a slide. Instances viewed as isolated instances within the slide context may be viewed with high output settings. The corresponding visualization may consist of two or more lesion regions (if multiple elements are required to contain the instance) and the surrounding context region. An example might be a disease infiltration on tissue, where multiple feature instances may be required to be viewed. Users may also view feature instances using a gallery view. The gallery may contain thumbnails of feature instances displayed for quick review, with the option for the user to jump to a specific instance by selecting a thumbnail. The gallery view may correspond to one or more lesion regions (however, only one lesion region may be required to contain the instance).

[0080] In the case of an instance set, all regions of an organization type may be contained within a single aggregate group. All instances of that type can be viewed together with low output. The corresponding visualization type may include a context region. [Table 2]

[0081] Partial (or specimen) level abstracts can be important for pathology workflows, especially when considering the effectiveness of reports.

[0082] All visualizations may have organization hops, as described in Table 2. Organization hops can allow users to "hop" between any visualization of an organization type detected throughout a part. For example, if one part contains three instances of an IDC, even if the IDC is an aggregate, organization hops allow users to move between those aggregate instances.

[0083] Case report 500 may be editable. Apart from any biomarkers, all alphanumeric output (either metadata or otherwise) may be editable. For example, measurement endpoints may be editable, and measurements may be updated during reporting and / or editing. Similarly, tissue map areas for future states may be editable. Calculated percentages or measured lengths may be updated during reporting and / or editing. Other metadata may not be recalculated.

[0084] The overall framework shown in Figure 5 is not limited to any specific AI or image analysis and / or processing system, and the embodiments disclosed herein may be used in any type of AI or image analysis system. For example, multiple systems and / or modules may be used in the overall framework, and the outputs of multiple systems may be visualized and integrated with respect to images, parts, cases, and / or patients.

[0085] Figures 6A to 6D illustrate how common visualizations and interactions can be applied to different organizational types.

[0086] Figure 6A shows examples of visualization of a general tissue specimen and organization of a prostate tissue specimen. Within a general tissue specimen, small regions of interest are identified as lesions. Alternatively, both small and large regions of interest may be identified as lesions. A lesion may be a single point of interest with some surrounding region of context. The context region of a general tissue specimen may include important regions of interest and / or tissue maps. This may be shown as a large region of interest, such as the length of adjacent tumors on a slide (which may be more appropriate for a tissue biopsy). The ruler tool may be used to measure either large or small regions of interest, along with numerical output. The ruler may be used to measure one or more biomarkers within the specimen tissue. In slide-level visualizations, alphanumeric display may include text and numerical output that overlays the slide. These outputs may be binary, discrete scores, continuous scores, categories, probabilities, quantifications (including percentages), and / or report outputs in Portable Document Format (PDF) or other formats. In the case of PDF reports, they may be static reports that cannot be easily edited. PDF reports may be associated with, for example, molecular testing, CLIA laboratory test results, or other test or laboratory results that are intended to remain unedited or unmanipulated.

[0087] In the example of a prostate tissue specimen in Figure 6A, the visualization of multiple lesions may include the entire tumor and one or more patterns. The context area may vary depending on the lesion in question; for example, the entire tumor may include the total length of the tumor, and the user may measure the total length of the tumor if necessary. Patterns may be described or measured by the user to include the percentage of tissue covered by the pattern. Furthermore, alphanumeric display may output primary and / or secondary Gleason scores.

[0088] Figure 6B shows an exemplary visualization of a breast tissue specimen. As in Figure 6A, the lesion area is selected, measured, and an alphanumeric output is obtained. Further examples of bladder, colon, and lung tissue are shown in Figures 6C and 6D.

[0089] Figure 7 shows an exemplary diagram of reportable features according to exemplary embodiments of the present disclosure. For example, in the case of invasive cancer features, reportable features may include a grade indicating whether the grade is within a range considered good, moderate, or poor, and the length of the measured feature. The resulting excision indicator may instruct the user to excise only the edges of the tissue specimen. Figure 7 includes various other exemplary reportable features, but a more detailed description is provided in Tables 3 to 6 below.

[0090] As shown in Figure 8, device 800 may include a central processing unit (CPU) 820. The CPU 820 may be any type of processor device, including, for example, any type of special-purpose or general-purpose microprocessor device. As will be recognized by those skilled in the art, the CPU 820 may also be a single processor in a multicore / multiprocessor system, such a system operating independently or in a cluster of computing devices operating in a cluster or server farm. The CPU 820 may be connected to a data communication infrastructure 810, such as a bus, message queue, network, or multicore message passing scheme.

[0091] Device 800 may also include main memory 840, for example, random access memory (RAM), and may also include secondary memory 830. Secondary memory 830, such as read-only memory (ROM), may be, for example, a hard disk or a removable storage drive. Such a removable storage drive may include, for example, a floppy disk drive, a magnetic tape drive, an optical disk drive, flash memory, etc. The removable storage drive in this example reads from and / or writes to a removable storage unit in a well-known manner. Removable storage may include floppy disks, magnetic tapes, optical disks, etc., which are read from and written to by the removable storage drive. As will be recognized by those skilled in the art, such a removable storage unit generally includes a computer-usable storage medium on which computer software and / or data is stored.

[0092] In alternative implementations, the secondary memory 830 may include similar means for enabling the loading of computer programs or other instructions into device 800. Examples of such means may include program cartridges and cartridge interfaces (such as those found in video game machines), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units, as well as interfaces that enable the transfer of software and data from the removable storage units to device 800.

[0093] Device 800 may also include a communication interface ("COM") 860. The communication interface 860 enables the transfer of software and data between Device 800 and external devices. The communication interface 860 may include a modem, a network interface (such as an Ethernet® card), a communication port, a PCMCIA slot and card, etc. The software and data transferred via the communication interface 860 may be in the form of electrical signals, electromagnetic signals, optical signals, or other signals that can be received by the communication interface 860. These signals may be provided to the communication interface 860 via a communication path of Device 800, which may be implemented using, for example, wires or cables, optical fibers, telephone lines, cell phone links, RF links, or other communication channels.

[0094] The hardware elements, operating systems, and programming languages ​​of such devices are essentially conventional, and it is presumed that those skilled in the art are quite familiar with them. Device 800 may also include input / output ports 850 for connecting to input / output devices such as keyboards, mice, touchscreens, monitors, and displays. Needless to say, various server functions may be implemented in a distributed manner on several similar platforms to distribute the processing load. Alternatively, the server may be implemented by appropriate programming on a single computer hardware platform.

[0095] Throughout this disclosure, references to components or modules generally refer to items that can be logically grouped together to perform a function or a group of related functions. Similar reference numbers are generally intended to refer to the same component or similar components. Components and modules may be implemented in software, hardware, or a combination of software and hardware.

[0096] The tools, modules, and functions described above may be executed by one or more processors. “Storage” type media may include any or all of the following: tangible memory such as computers and processors, or their associated modules (various semiconductor memories, tape drives, disk drives, etc.), which may provide non-temporary storage for software programming at any time.

[0097] Software may be communicated via the Internet, cloud service providers, or other telecommunications networks. For example, communication may make it possible to load software from one computer or processor to another. As used herein, unless limited to non-temporary, tangible “storage” media, terms such as computer or machine “readable media” refer to any medium involved in giving instructions to a processor for execution.

[0098] The general description above is illustrative and descriptive only and does not limit the present disclosure. Other embodiments of the present invention will become apparent to those skilled in the art from consideration herein and the practice of the present invention disclosed herein. The specification and examples are intended to be considered illustrative only. [Table 3] [Table 4-1] [Table 4-2] [Table 4-3] [Table 5] [Table 6-1] [Table 6-2] Table 6-3 Table 6-4

Claims

1. A computer implementation method for processing electronic images associated with pathological specimens, Receiving one or more cases associated with pathological specimens in a digital storage device, Receiving the selection of one case from the one or more cases mentioned above, The aforementioned case is divided into multiple parts, and a selected part of these multiple parts is divided into multiple slides, with each part corresponding to an individual tissue sample from the patient. The method involves applying a machine learning model to generate an interactive visualization of the aforementioned multiple slides, wherein the machine learning model is trained by processing multiple training images. To determine the image features associated with the aforementioned multiple slides, To determine at least one report for the aforementioned image features, The method involves integrating multiple feature instances of a reportable feature into a single group, wherein the reportable feature corresponds to at least one of the image features. Integrating the at least one report for the image features from the plurality of slides associated with the selected portion into a partial report showing a diagnosis for the individual tissue specimen, Integrate at least one sub-report into the case report. Computer implementation methods, including those mentioned above.

2. The computer implementation method according to claim 1, wherein the image features include classified and / or labeled regions or lesions of suspected tissue, observations of the suspected tissue, and / or feature instances.

3. The computer implementation method according to claim 2, wherein the image features include diagnostic and / or anatomical features specific to the reportable features, and associated metadata.

4. Displaying at least one thumbnail of the image feature in a gallery view that includes the reportable features, Selecting a thumbnail to jump to the aforementioned image features and The computer implementation method according to claim 2, further comprising:

5. Displaying the group of at least one feature instance together, Outputting the visualization of the group along with the related context area The computer implementation method according to claim 2, further comprising:

6. Determining the relationship between two reportable features, The aforementioned case report will detail the aforementioned relationship. The computer implementation method according to claim 1, further comprising:

7. Identify at least one metric associated with a reportable feature, The case report shall include at least one of the aforementioned measurement criteria. The computer implementation method according to claim 1, further comprising:

8. On the slide, you can move between selecting a visualization of the first feature instance or set of feature instances and selecting a visualization of the second feature instance or set of feature instances. To display the aforementioned selection to the user and The computer implementation method according to claim 1, further comprising:

9. The computer implementation method according to claim 8, wherein the visualization of feature instances includes one or more lesion regions.

10. The computer implementation method according to claim 1, wherein the report, the partial report, and / or the case report for the image features are editable.

11. The computer implementation method according to claim 2, wherein the feature instance is editable.

12. A system for using at least one machine learning model to process electronic images associated with pathological specimens, At least one memory to store instructions, At least one processor configured to execute the aforementioned instructions and perform an operation, Equipped with, The aforementioned operation is, Receiving one or more cases associated with pathological specimens in a digital storage device, Receiving the selection of one case from the one or more cases mentioned above, The aforementioned case is divided into multiple parts, and a selected part of these multiple parts is divided into multiple slides, with each part corresponding to an individual tissue sample from the patient. The method involves applying a machine learning model to generate an interactive visualization of the aforementioned multiple slides, wherein the machine learning model is trained by processing multiple training images. To determine the image features associated with the aforementioned multiple slides, To determine at least one report for the aforementioned image features, Integrating at least one feature instance of a reportable feature into one group, Integrating the at least one report for the image features from the plurality of slides associated with the selected portion into a partial report showing a diagnosis for the individual tissue specimen, Integrate at least one sub-report into the case report. A system that includes this.

13. The system according to claim 12, wherein the image features include classified and / or labeled regions or lesions of suspected tissue, observations of the suspected tissue, and / or feature instances.

14. The system according to claim 13, wherein the image features include diagnostic and / or anatomical features specific to the reportable features, and associated metadata.

15. The aforementioned operation, Displaying at least one thumbnail of the image feature in a gallery view that includes the reportable features, Selecting a thumbnail to jump to the aforementioned image features and The system according to claim 13, further comprising:

16. The aforementioned operation, Displaying the group of at least one feature instance together, Outputting the visualization of the group along with the related context area The system according to claim 13, further comprising:

17. The aforementioned operation, Determining the relationship between two reportable features, The aforementioned case report will detail the aforementioned relationship. The system according to claim 12, further comprising:

18. A non-temporary computer-readable medium that stores instructions for a method of processing an electronic image associated with a pathological specimen when executed by a processor, wherein the method is Receiving one or more cases associated with pathological specimens in a digital storage device, Receiving the selection of one case from the one or more cases mentioned above, The aforementioned case is divided into multiple parts, and a selected part of these multiple parts is divided into multiple slides, with each part corresponding to an individual tissue sample from the patient. The method involves applying a machine learning model to generate an interactive visualization of the aforementioned multiple slides, wherein the machine learning model is trained by processing multiple training images. To determine the image features associated with the aforementioned multiple slides, To determine at least one report for the aforementioned image features, Integrating at least one feature instance of a reportable feature into one group, Integrating the at least one report for the image features from the plurality of slides associated with the selected portion into a partial report showing a diagnosis for the individual tissue specimen, Integrate at least one sub-report into the case report. Non-temporary computer-readable media, including [specific examples of such media].

19. The computer-readable medium according to claim 18, wherein the image features include classified and / or labeled regions or lesions of suspected tissue, observations of the suspected tissue, and / or feature instances.

20. The method described above is On the slide, you can move between selecting a visualization of the first feature instance or set of feature instances and selecting a visualization of the second feature instance or set of feature instances. To display the aforementioned selection to the user and The computer-readable medium according to claim 18, further comprising:

Images