Image processing system and method for determining case optimization

Abstract

Described herein are systems and methods for processing electronic medical images to optimize an examination order of pathology cases. For example, a plurality of pathology cases may be received along with a plurality of variables and one or more constraints. Each case of the plurality of pathology cases may include one or more medical images of at least one pathology specimen associated with a patient. The medical images from each case, the plurality of variables, and the one or more constraints may be provided as inputs to a trained system. An order of user examination of the plurality of cases to optimize one or more of the plurality of variables based on the one or more constraints may be received as an output of the trained system. Each case of the plurality of cases may be automatically provided to a user for examination according to the order.

Description

[Technical field]

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 290,479, filed December 16, 2021, the entire disclosure of which is incorporated herein by reference in its entirety.

[0002] Various embodiments of the present disclosure relate generally to image processing systems and methods, and more specifically, certain embodiments of the present disclosure relate to image processing systems and methods for determining case optimization. [Background technology]

[0003] In pathology, in a manual workflow, pathologists review cases primarily based on the order in which physical slide trays are stacked on their desks. During a manual workflow, there may be some cases marked with a physical indicator such as “STAT” that need urgent review, and they are taken out for review first. In a digital workflow, pathologists have access to a digital worklist or laboratory information system that notes, for example, the status of different cases, which cases are assigned to them, which cases are awaiting further information, etc. However, prioritization of these cases assigned for review can be time-consuming and inconsistent.

[0004] The background discussion provided herein is intended to provide a general context for the disclosure. Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application and is not admitted to be prior art or suggested by inclusion in this section. Summary of the Invention [Means for solving the problem]

[0005] According to an aspect of the present disclosure, a system and method for processing electronic medical images are disclosed. In one aspect, a computer-implemented method for processing electronic medical images to optimize the examination order of pathology cases is disclosed. The method may include receiving a plurality of variables and one or more constraints; receiving a plurality of pathology cases, each of the plurality of pathology cases including one or more medical images of at least one pathology specimen associated with a patient; providing one or more medical images from each case, the plurality of variables, and the one or more constraints as input to a trained system; receiving an order of user examination of the plurality of cases to optimize one or more of the plurality of variables based on the one or more constraints as output of the trained system; and automatically providing each case of the plurality of cases to a user for examination according to the order.

[0006] The trained system may determine a plurality of potential sequences and a score for each of the plurality of potential sequences indicative of an optimization level of one or more of the plurality of variables, and the sequence provided as an output is one of the plurality of potential sequences having the highest score.

[0007] When at least a first variable and a second variable of the plurality of variables are optimized, the trained system may determine a first score for the first variable and a second score for the second variable for each of the plurality of potential orders, and the order provided as output may be one of the plurality of potential orders having an overall score that is maximized based on the first score and the second score.

[0008] One or more of the multiple variables to be optimized may be user-selected variables. The trained system may be a trained machine learning system or a trained rule-based system.

[0009] The method may further include receiving one or more determined characteristics of the medical image as output from a trained machine learning system configured to process one or more medical images from each of the plurality of pathology cases to determine one or more characteristics of the medical image, and providing the determined one or more characteristics of the medical image as a further input to the trained system, where one determined characteristic includes case complexity, case type, number of regions of interest per medical image or per case, amount of tissue per medical image, and image quality.

[0010] The method may further include receiving one or more additional pathology cases, each of the one or more additional pathology cases including one or more medical images of at least one pathology specimen associated with the patient, providing the one or more medical images from each of the one or more additional pathology cases as further inputs to the trained system, and receiving the updated order as an output of the trained system.

[0011] The trained system may be further configured to assign a subset of the plurality of cases to each of the plurality of users, and the order received as output from the trained system includes an order of user testing of each subset of the plurality of cases.

[0012] The method may further include generating a notification to prompt the user to take one or more breaks to increase optimization of one or more of the variables based on the information received from the training system.

[0013] Automatically providing each case of the plurality of cases to the user for review in a sequence may include automatically navigating from a first case to a subsequent case in the sequence based on an indication that review of the first case is complete.

[0014] The indication may be an event associated with an input received from a user or an automatically detected case exam completion.

[0015] The method may further include monitoring values ​​associated with the plurality of variables as the user examines the plurality of cases, and providing the values ​​to a trained system, where the trained system is retrained based on the values ​​for future optimization.

[0016] According to an aspect of the present disclosure, a system and method for processing electronic medical images are disclosed. In another aspect, a system for processing electronic digital medical images may include at least one memory storing instructions and at least one processor configured to execute the instructions to perform operations. The at least one processor may include: receiving a plurality of variables and one or more constraints; receiving a plurality of pathology cases, each of the plurality of pathology cases including one or more medical images of at least one pathology specimen associated with a patient; providing the one or more medical images from each case, the plurality of variables, and the one or more constraints as inputs to a trained system; receiving an order of user examination of the plurality of cases to optimize one or more of the plurality of variables based on the one or more constraints as an output of the trained system; and automatically providing each case of the plurality of cases to a user for examination according to the order.

[0017] The trained system may determine a plurality of potential sequences and a score for each of the plurality of potential sequences indicative of an optimization level of one or more of the plurality of variables. The sequence provided as an output may be one of the plurality of potential sequences having the highest score.

[0018] When at least a first variable and a second variable of the plurality of variables are optimized, the trained system may determine a first score for the first variable and a second score for the second variable for each of the plurality of potential orderings, and the ordering provided as output is one of the plurality of potential orderings having an overall score that is maximized based on the first score and the second score.

[0019] One or more of the multiple variables to be optimized may be user-selected variables. The trained system may be a trained machine learning system or a trained rule-based system.

[0020] The system may further include receiving one or more determined characteristics of the medical image as output from a trained machine learning system configured to process one or more medical images from each of the plurality of pathology cases to determine one or more characteristics of the medical image, and providing the determined one or more characteristics of the medical image as a further input to the trained system, where one determined characteristic includes case complexity, case type, number of regions of interest per medical image or per case, amount of tissue per medical image, and image quality.

[0021] According to an aspect of the present disclosure, a system and method for processing electronic medical images is disclosed. In another aspect, a non-transitory computer-readable medium is disclosed that stores instructions that, when executed by a processor, perform operations for processing electronic digital medical images. The operations may include receiving a plurality of variables and one or more constraints, receiving a plurality of pathology cases, each of the plurality of pathology cases including one or more medical images of at least one pathology specimen associated with a patient, providing the one or more medical images from each case, the plurality of variables, and the one or more constraints as inputs to a trained system, receiving an order of user examination of the plurality of cases to optimize one or more of the plurality of variables based on the one or more constraints as an output of the trained system, and automatically providing each case of the plurality of cases to a user for examination according to the order.

[0022] The trained system may determine a plurality of potential sequences and a score for each of the plurality of potential sequences indicative of an optimization level of one or more of the plurality of variables. The sequence provided as an output may be one of the plurality of potential sequences having the highest score.

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and, together with the description, serve to explain the principles of the disclosed embodiments. [Brief description of the drawings]

[0024] [Figure 1A] 1 shows an example block diagram of a system and network for processing images to determine optimal case sequence according to the techniques presented herein.

[0025] [Figure 1B] 1 shows an exemplary block diagram of a tissue observation platform in accordance with the technology presented herein.

[0026] [Figure 1C] FIG. 1 shows an exemplary block diagram of a slide analysis tool in accordance with the technology presented herein.

[0027] [Diagram 2] FIG. 1 is a flow diagram illustrating an example process for determining an optimized case sequence using a trained system in accordance with the techniques presented herein.

[0028] [Figure 3A] 1 is a flowchart of an exemplary method for training a system to determine optimized case sequencing in a clinical setting, according to the techniques presented herein.

[0029] [Figure 3B]1 is a flowchart illustrating an example method for determining case sequence optimization in a clinical setting using a trained system, according to one or more example embodiments herein.

[0030] [Figure 4A] 1 is a flowchart of an exemplary method for training a system to determine optimized case sequencing in a research setting, according to the techniques presented herein.

[0031] [Figure 4B] 1 is a flow chart illustrating an example method for determining case sequence optimization in a research setting using a trained system, according to one or more example embodiments herein.

[0032] [Figure 5A] 1 is a flowchart of an exemplary method for training a system to determine optimized case sequencing in an educational context, according to the techniques presented herein.

[0033] [Figure 5B] 1 is a flowchart illustrating an example method for determining case sequence optimization in an educational context using a trained system, according to one or more example embodiments herein.

[0034] [Figure 6] 1 is a flow chart illustrating an exemplary method for optimizing an order of cases displayed to one or more users.

[0035] [Figure 7] 1 illustrates an example of a computing device that can execute the techniques presented herein, according to one or more embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] Reference will now be made in detail to the exemplary embodiments of the present disclosure, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

[0037] The systems, devices, and methods disclosed herein are described in detail by way of example and with reference to the drawings. The examples described herein are merely examples and are provided to aid in the explanation of the apparatus, devices, systems, and methods described herein. No feature or component shown in the drawings or described below should be considered essential to any particular embodiment of these devices, systems, or methods, unless specifically designated as essential.

[0038] Also, for any method described, whether or not the method is described with reference to a flow diagram, unless otherwise specified or required by context, it should be understood that any explicit or implicit ordering of steps performed in the performance of the method does not imply that the steps must be performed in the order presented, but may be performed in a different order or in parallel.

[0039] As used herein, the term "exemplary" is used in the sense of "example" rather than "ideal." Furthermore, the words "a" and "an" do not denote a limitation of quantity herein, but rather denote the presence of one or more of the referenced item.

[0040] As used herein, a "machine learning model" generally encompasses instructions, data, and / or models configured to receive an input and apply one or more of weights, biases, classifications, or analyses to the input to generate an output. The output may include, for example, a classification of the input, an analysis based on the input, a design, process, prediction, or recommendation associated with the input, or any other suitable type of output. Machine learning models are generally trained using training data, e.g., samples of empirical data and / or input data, that are fed to the model to establish, adjust, or modify one or more aspects of the model, such as weights, biases, or criteria for forming classifications or clusters. Deep learning techniques may also be employed. Aspects of the machine learning model may operate linearly on the input, in parallel, through a network (e.g., a neural network), or through any suitable configuration.

[0041] Execution of the machine learning model may include deployment of one or more machine learning techniques, such as linear regression, logistic regression, random forest, gradient boosting machine (GBM), deep learning, and / or deep neural networks. Supervised and / or unsupervised training may be employed. For example, supervised learning may include providing training data and labels corresponding to the training data, e.g., as ground truth. Unsupervised techniques may include clustering or classification, etc. K-means clustering or K-nearest neighbors may also be used, which may be supervised or unsupervised. A combination of K-nearest neighbors and unsupervised cluster techniques may also be used. Any suitable type of training may be used, e.g., stochastic, gradient boosting, random seed, recurrent, epoch or batch-based.

[0042] In the field of pathology, pathologists may access digital medical images through a digital workflow. In the digital workflow, pathologists may have access to a digital worklist or laboratory information system that notes the status of different cases, which cases are assigned to them, and which cases are awaiting further information. This may again require input from the pathologist into filters for a case. For example, a pathologist may filter based on the pathologist's specific behavior or experience (e.g., a pathologist may want to view resections first, since resections take more time). As another example, a pathologist may filter based on site (e.g., biopsies should always be pushed to the top of the worklist, given the tight response time).

[0043] Pathologists may be under pressure to examine more specimens within shorter response times. Using the techniques described herein, pathologists may save significant time because they may not need to go back to their worklist to find the case and spend seconds opening the case. Additionally, the techniques disclosed herein may ensure that cases are presented to pathologists in a manner that maximizes efficiency, reduces fatigue, and takes into account pathologist and administrator preferences.

[0044] Using the techniques described herein, a pathologist may not need to manually curate or customize a worklist or to review the worklist each time before moving on to the next case. After completing an examination and creating a report or initial evaluation of the case, the pathologist may automatically begin reviewing the next case.

[0045] Pathology departments and / or laboratories are also under pressure to respond quickly and efficiently with reports to their clients (e.g., clinicians). The system can create optimal case sequencing for testing based on pathologist preferences, time of day, clinician expectations, specimen type, diagnosis, etc., allowing for faster response times, which can be advantageous for the laboratory and its clients.

[0046] FIG. 1A illustrates a block diagram of a system and network for processing images to determine optimal case sequence, according to an exemplary technique of this disclosure.

[0047] Specifically, FIG. 1A illustrates an electronic network 120 that may be connected to servers of a hospital, laboratory, and / or clinic, etc. For example, physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125, etc. may each be connected to electronic network 120, such as the Internet, through one or more computers, servers, and / or handheld mobile devices. According to an exemplary embodiment of the present disclosure, electronic network 120 may also be connected to server system 110, which may include processing devices 111. One or more of processing devices 111 may be configured to implement tissue viewing platform 100, which includes slide analysis tool 101 for determining specimen property information or image property information on digital pathology image(s) and classifying specimens using machine learning according to the exemplary techniques described in this disclosure. The tissue review platform 100 may also include a case order optimization tool 141 for determining an order in which to present cases to one or more users (e.g., one or more pathologists) to optimize one or more variables subject to one or more constraints in accordance with the example techniques described herein. In other examples, the case order optimization tool 141 may be operated separately (e.g., by a different platform) from the tissue review platform 100.

[0048] The doctor server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125 may create or obtain images of one or more patients' cytology specimen(s), images of histopathology specimen(s), images of slide(s) of cytology specimen(s), digitized images of slide(s) of histopathology specimen(s), or any combination thereof. The doctor server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125 may also obtain any combination of patient-specific information, such as age, medical history, cancer treatment history, family history, previous biopsy or cytology information, etc. The doctor server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125 may transmit the digitized slide images and / or the patient-specific information to the server system 110 via the electronic network 120. The server system 110 may include one or more storage devices 109 for storing images and data received from at least one of the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125. The server system 110 may also include a processing device 111 for processing the images and data stored in the one or more storage devices 109. The server system 110 may further include one or more machine learning tool(s) or machine learning capabilities. For example, according to one embodiment, the processing device 111 may include machine learning tools for the tissue observation platform 100. Alternatively or in addition, the present disclosure (or parts of the systems and methods of the present disclosure) may be performed on a local processing device (e.g., a laptop).

[0049] Physician server 121, hospital server 122, clinical trial server 123, laboratory server 124, and / or laboratory information system 125 refer to systems used by pathologists to review images of slides. In a hospital environment, tissue type information may be stored in one of the laboratory information systems 125.

[0050] 1B illustrates an exemplary block diagram of tissue review platform 100. For example, tissue review platform 100 can include a slide analysis tool 101, a case sequence optimization tool 141, a data capture tool 102, a slide import tool 103, a slide scanner 104, a slide manager 105, storage 106, and a viewing application tool 108.

[0051] According to an exemplary embodiment, slide analysis tool 101 refers to a process and system for processing digital pathology slides (e.g., digitized images of tissue or cytology specimens mounted on a slide) and analyzing a given slide using machine learning, as described in more detail below.

[0052] Case sequence optimization tool 141 refers to processes and systems for using machine learning or rule-based systems to process digital pathology slides (e.g., digitized images of historical or cytology specimens mounted on slides) and determine the order in which to present cases to one or more users (e.g., one or more pathologists) to optimize one or more variables subject to one or more constraints, as described in more detail below.

[0053] According to an exemplary embodiment, data capture tools 102 refer to processes and systems for facilitating the transfer of digital pathology images to various tools, modules, components, and devices used to classify and process the digital pathology images.

[0054] According to an exemplary embodiment, slide capture tool 103 refers to the process and system for scanning pathology slides and converting them into digital format. Slides may be scanned with slide scanner 104, and slide manager 105 may process the images on the slides into digitized pathology images and store the digitized images in storage 106.

[0055] According to an exemplary embodiment, the viewing application tool 108 refers to processes and systems for providing specimen property information or image property information regarding digital pathology image(s) to a user (e.g., a pathologist). The information may be provided through a variety of output interfaces (e.g., a screen, a monitor, a storage device, and / or a web browser, etc.).

[0056] The slide analysis tool 101 and one or more of its components may transmit and / or receive digitized slide images and / or patient information to the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125 via the electronic network 120. Additionally, the server system 110 may include one or more storage devices 109 for storing images and data received from at least one of the slide analysis tool 101, the data capture tool 102, the slide import tool 103, the slide scanner 104, the slide manager 105, and the viewing application tool 108. The server system 110 may also include a processing device 111 for processing images and data stored in the storage device 109. The server system 110 may further include one or more machine learning tool(s) or capabilities, e.g., by the processing device 111. Alternatively, or in addition, the present disclosure (or parts of the systems and methods of the present disclosure) may be performed on a local processing device (eg, a laptop).

[0057] Any of the above devices, tools, and modules may be located on devices that may be connected to an electronic network 120, such as the Internet or a cloud service provider, through one or more computers, servers, and / or handheld mobile devices.

[0058] 1C illustrates an example block diagram of a slide analysis tool 101, according to an example embodiment of the present disclosure. The slide analysis tool 101 may include a training image platform 131 and / or a target image platform 135.

[0059] The training image platform 131, according to one embodiment, may create or receive training images used to train the machine learning system to effectively analyze and classify digital pathology images. For example, the training images may be received from any one or any combination of the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125. The images used for training may come from real sources (e.g., humans, animals, etc.) or may come from synthetic sources (e.g., graphics rendering engines, 3D models, etc.). Examples of digital pathology images may include (a) digitized slides stained with various stains, such as, but not limited to, H&E, hematoxylin alone, IHC, molecular pathology, etc., and / or (b) digitized image samples from 3D imaging devices, such as micro-CT.

[0060] The training image capture module 132 may create or receive a dataset including one or more training images corresponding to one or both of images of human tissue and graphically rendered images. For example, the training images may be received from any one or any combination of the server system 110, the physician server 121, and / or the laboratory information system 125. The dataset may be held on a digital storage device. In some examples, the dataset may be composed of multiple data subsets, each data subset corresponding to a training case from the multiple training cases and including one or more training images from the training cases. The training slide characteristics module 133 may include, for example, one or more computing devices capable of determining whether the training images have a sufficient quality level to train the machine learning model. The training slide characteristics module 133 may further include, for example, one or more computing devices capable of identifying whether a set of individual cells belongs to the cells of interest or belongs to the background of the digitized image.

[0061] The target image platform 135 may include one or more computing devices capable of receiving a target dataset and applying a machine learning model to the received target dataset to determine one or more characteristics of the target dataset. In some examples, the target dataset may include one or more target images included in a case. The characteristics of the target dataset may include, but are not limited to, case complexity, case type, number of regions of interest per slide, amount of tissue per slide, and / or image quality. For example, the target dataset may be received from the server system 110, the physician server 121, the hospital server 122, the clinical trial server 123, the laboratory server 124, and / or the laboratory information system 125. The target image capture module 136 may include one or more computing devices capable of, for example, receiving the target dataset. The slide characteristics determination module 137 may include one or more computing devices capable of, for example, applying a machine learning model to the target dataset to determine case complexity, case type, number of regions of interest, amount of tissue per slide, and / or image quality. The slide characteristics module decision 137 may also be responsible for identifying the amount of images associated with the input case that may need to be reviewed by one or more pathologists. For example, the slide characteristics module decision 137 may identify all previous cases the patient has that may need to be reviewed / reviewed.

[0062] The output interface 138 may include, for example, one or more computing devices capable of outputting (e.g., to a screen, monitor, storage device, web browser, etc.) information regarding the target dataset and the determined relationships. In some examples, the output interface 138 may provide case complexity, case type, number of regions of interest, amount of tissue per slide, and / or image quality to the case sequence optimization tool 141 for use, for example, as input for one or more other processes.

[0063] 2 is a flow diagram 200 illustrating an example process for determining an optimized case order using a trained system. The trained system may be implemented by a case order optimization tool 141 of the tissue observation platform 100. The case order optimization tool 141 may utilize the trained system to determine an order for presenting each case of a plurality of cases under review that optimizes one or more variables 202 within the bounds of one or more constraints 204. Each case may include one or more case images 205, and the determined order may be further based on the case images 205 and / or output from an analysis thereof (e.g., output from the slide analysis tool 101).

[0064] For example, the case sequence optimization tool 141 may receive one or more variables 202. The variables 202 may include important factors to consider to inform case sequence. The variables 202 may vary by pathologist, site, institution, etc. The variables 202 may include, but are not limited to, pathologist preferences, number of cases to be examined per day, pathologist expertise, diagnostic quality, time of day, day of week, pathologist behavior, client expectations, and / or laboratory considerations, each of which is described in turn below. The variables 202 may be input as a numeric value (e.g., time of day) or as text. For variables 202 input as text, a rules-based text extraction system may be used to extract information from the received data set or data sets. The case sequence optimization tool 141 may allow a user (e.g., a pathologist, department manager, administrator) to input any of the variables 202 described within this application.

[0065] A pathologist's preferences may refer to the types of cases that a pathologist prefers and / or does not prefer to analyze. For example, a pathologist may prefer not to examine cases of prostate cancer. In some embodiments, the case order optimization tool 141 may display a list of case types, and a user may select from the list one or more of the case types that the user would like to examine and / or one or more of the case types that the user would not like to examine. Additionally, the user may be able to rank the case types by highest and lowest preference. The number of cases to examine per day may refer to the desired quantity of cases that a particular pathologist may examine in a day or within a certain period of time, which may be entered by a system administrator or a particular pathologist.

[0066] A pathologist's expertise may refer to the level of expertise a pathologist has with one or more case types. The case order optimization tool 141 may allow a pathologist to select from all available case types and select those cases in which the pathologist has expertise. Additionally, the pathologist may be able to provide a score (e.g., a number from 1 to 10) indicating the level of expertise for each case type. Additionally, the case order optimization tool 141 may track or log cases reviewed by a particular pathologist to update the expertise of that particular individual. Diagnostic quality may refer to the percentage of cases of a particular type that are overturned and / or denied by a second reviewing pathologist. Diagnostic quality variables may be an overall assessment of all diagnoses made and / or an assessment per case type.

[0067] The time may refer to a time of day (e.g., morning or afternoon) and / or a day of the week (e.g., Monday or Friday). The case sequence optimization tool 141 may receive and / or generate time information regarding how the quality and / or speed of a pathologist's examination is affected depending on the time of day and / or the day of the week. The case metadata may refer to data associated with a patient's case, such as the type of case, the number of digital slides, the amount of tissue on each digital slide, the number of regions of interest in each digital slide, and / or the number of previous cases associated with the same patient. In some examples, a portion of the case metadata may be received from the storage device 109, the clinical trial server 123, the physician server 121, the laboratory information system 125, the laboratory server 124, and / or the hospital server 122. Additionally or alternatively, a portion of the case metadata may be received as an output of the slide analysis tool 101.

[0068] Pathologist behavior may refer to data related to how a pathologist interacts with the viewing application tool 108, for example, to review cases. The data may include general usage of the viewing application tool 108, including fatigue and / or user burnout, relative speed of review based on total time to review a particular type of slide, total time to review a particular type of case taken based on input speed (e.g., mouse, keyboard, touch, eye tracking, etc.). Client expectations may refer to response time for a pathologist's report and / or initial assessment from the time a case is forwarded to a department or lab for review. In some examples, the response time may be a standard period across cases. However, certain individual cases or case types may be of higher importance and / or more time-sensitive. Thus, these individual cases or case types may be digitally marked to indicate a higher priority and therefore a shorter response time expected by the client. In one example, this digital mark may be equivalent to certain cases being marked as "Stat" during manual review. In other examples, cases may be automatically identified as a higher priority based on the type of case, for example, regardless of whether the case is digitally marked. For example, biopsies may be automatically identified as a higher priority given that expected response times are typically more stringent. Laboratory considerations may refer to considerations for a particular case, such as diagnosis, number of pathologists available that day, or further testing requirements. For example, if a case is likely to require additional testing based on the diagnosis, and the pathologist may be notified of this by the AI ​​before seeing the case, the need for further testing may result in the case being prioritized over some other cases and added to the pathologist's queue.

[0069] The case sequence optimization tool 141 may further receive one or more constraints 204. The constraints 204 may include goals or key performance indicators (KPIs) that the pathologists may need to meet. Examples of constraints may include, but are not limited to, available pathologists, pathologist qualifications, case deadlines, time remaining in a shift or day, and / or department goals. Available pathologists may refer to a list of all potential pathologists and / or at least the number of all potential pathologists available at a particular time or on a particular day, where all potential pathologists may be generally available to perform case reviews. Pathologist qualifications may include the particular types of cases that a pathologist (e.g., from the available pathologists) is trained and / or has expertise to handle. Case deadlines may refer to a particular date and / or time by which a case needs to be analyzed. In one example, case deadlines may be extracted as metadata included with a particular case. Case deadlines may also be automatically determined based on standard response times for a particular case type. Deadlines may be adjusted to account for cases marked as high priority (e.g., cases must be analyzed within 24 hours of receiving the case in the system for higher priority cases). Additionally, case deadlines may be entered and / or modified externally by pathologists as cases / case load is entered.

[0070] The time remaining in a shift or day may refer to the time remaining for one or more of the available and / or qualified pathologists to analyze cases that day. For example, a shift schedule may be received that includes the times each available and / or qualified pathologist starts and finishes work, and the case sequence optimizer 141 may track the time remaining for that day until the end time. Additionally, the case sequence optimizer 141 may subtract break times, such as set lunch break times and / or other mandated breaks, from the time remaining in the shift to accurately reflect how much work time each pathologist has left in the shift. Department goals may refer to certain types of cases, or high priority cases of certain patients to be analyzed within a certain time period. Department goals may also refer to the total number of cases a department may want to analyze over a period of time, such as the total number of cases for individual pathologists and across the entire laboratory. Department goals may refer to the average accuracy a laboratory may seek to achieve across all pathologists in the laboratory.

[0071] Although variables 202 are described herein as optional factors to be optimized and constraints 204 are described herein as requirements to be met that limit or define the boundaries of case allocation and sequencing, the particular examples or types of constraints 204 described herein may, in some cases, be specified as variables 202 by a user and / or administrator, or vice versa. For example, a department goal may be a constraint 204 that must be met or may be specified as a variable 202 to be optimized relative to other variables 202. For example, in some cases, the number of pathologists available to review cases may be a constraint 204. In other cases, other pathologists may be available to be called in for assistance, and thus the number of pathologists may be selected or indicated as a variable 202 to be optimized.

[0072] The case order optimization tool 141 may further receive case images 205 for the multiple cases to be analyzed. The case images 205 may include digitized pathology images. As described above, examples of digital pathology images may include (a) digitized slides stained with various stains, such as (but not limited to) H&E, hematoxylin alone, IHC, molecular pathology, and / or (b) digitized image samples from a 3D imaging device, such as a micro-CT. The case images 205 may also be referred to as whole slide images (WSIs). The case order optimization tool 141 may receive one or more case images 205 for each case. Thus, the case order optimization tool 141 may receive multiple sets of one or more WSIs, each set corresponding to each case of the multiple cases. In some examples, in addition to the case images 205, other case-related data, such as patient data, may be received. Case-related data including at least the case images 205 may be imported from the storage device 109, the clinical trial server 123, the physician server 121, the laboratory information system 125, the laboratory server 124, the hospital server 122, or an external network. Additionally, these case images 205 may be imported through an external storage device such as a flash drive. The case sequence optimizer 141 may be capable of continually receiving new case images 205 corresponding to new cases and updating the case sequence to incorporate the new cases as the new case images 205 are received.

[0073] In some examples, the case sequence optimization tool 141 may also receive output from an analysis or evaluation of the case images 205 performed, for example, by the slide analysis tool 101. These analysis and / or evaluation outputs may optionally be received and / or implemented as variables 202. Exemplary analysis or evaluation outputs may include AI evaluation of slides that may determine the time required to evaluate a particular case, the type of case, the number of regions of interest, the amount of tissue per slide, the number of slides in a case, and / or the number of past cases of a patient that a pathologist must review, relative to the complexity of the case. The complexity of a case may be determined in part based on the number of regions of interest (e.g., salient regions). Salient regions may be determined according to techniques described in U.S. Patent Application Serial No. 17 / 313,617, filed May 6, 2021, which is incorporated herein by reference in its entirety.

[0074] The case order optimization tool 141 may utilize a trained system to determine an order for presenting each case of the multiple cases to be reviewed while optimizing one or more variables 202 within the bounds of the constraints 204. For example, the case order optimization tool 141 may provide one or more variables 202 and constraints 204 along with case images 205 (and optionally corresponding case image analysis or evaluation output from the slide analysis tool 101) as inputs to the trained system. The trained system may determine and output an order for presenting each case of the multiple cases to be reviewed. For example, the cases may be assigned to one or more pathologists, and an order for reviewing the cases may be determined per the case load assigned to the pathologists.

[0075] When making a case ordering determination, the trained system may optimize one or more of the variables 202 received as input based on one or more constraints 204 received as input. In some examples, the variables 202 may be optimized automatically. In other examples, the variables to be optimized may be selected by a user (e.g., a pathologist, department manager, administrator, etc.). The user may also select and / or modify the constraints 204 for the determination. As an example, the trained system may generate a number of possible case assignments and case orders for each assignment, all within the constraints 204. The trained system may determine a score for each of the multiple possible assignments and a case order for each assignment. The score may be based on the degree or level to which the variables 202 to be optimized are satisfied. The case assignments and the order of the cases with the highest scores within the assignments may be output by the trained system.

[0076] When multiple variables 202 (e.g., more than one variable) are optimized, a score may be assigned to each optimized variable 202 of the multiple variables 202, and the trained system may output the case assignment and ordering therein that maximizes the overall score. For example, when one set of assignments and case orderings has a high score for a first variable but a low score for a second variable, and a second set of assignments and case orderings has a low score for the first variable but a high score for the second variable, the trained system may select to output whichever set has the highest overall score. The trained system may further consider the indicated priority, if any, of the first and second variables when selecting. The indicated priority may also be considered in terms of the constraints 204. For example, if a pathologist's preference is not to test for prostate cancer (e.g., a first variable), but there is a department goal to have a large number of prostate cancer cases and complete them on the same day (e.g., a constraint), the trained system may still assign some prostate cancer cases to the pathologist to ensure that the constraint is met (e.g., not optimizing the first variable as a result). In such a case, the trained system may instead optimize a second variable, such as the best time of day for the pathologist to test, or the type of prostate cancer cases to assign to the pathologist. The trained system may follow any number of constraints 204, for example, if a particular case is an emergency case, immediate testing of the case may be included as a constraint and the case may be prioritized accordingly.

[0077] In some examples, the trained system may be a trained machine learning (ML) system. Exemplary types of ML systems that may be trained and implemented for optimized case sequence determination may include, but are not limited to, reactive machines and memory-limited machines. In some examples, the ML system may include a neural network for objective function approximation. Training and implementation of ML systems are described in more detail below. Additionally or alternatively, the trained system may be a rule-based system (e.g., applying rule-based logic) for determining optimized case sequence. In some examples, whether the trained system is ML-based or rule-based may depend on the optimization.

[0078] Once the optimized case order is determined, the trained system may output one or more lists of cases 208 based on the determination. For example, each list of cases 208 may be specific to (e.g., assigned to) a particular pathologist within a reviewing department or laboratory. The order of the cases 208 in the list may represent an optimized case order in which the cases 208 should be reviewed by the pathologist. In some examples, the list of cases 208 may identify the cases based on a case number or other identifier to enable automated retrieval and presentation of one or more case images 205 for each case, as described in more detail below. In other examples, the list of cases 208 may include one or more case images 205 for each case. The trained system may output an updated list of cases 208 as additional cases, their respective case images 205 are received. In some examples, the update may be performed each time a new case is received. In other examples, the update may be performed at regular time intervals. Additionally, the trained system may output an updated list of cases 208 once one or more variables 202 and / or constraints 204 are modified. For example, if a pathologist who was previously available in a particular lab is now unable to work in a given lab, the list of cases 208 may be updated, for example, to reassign cases to other pathologists who are available in the particular lab and update case ordering.

[0079] In some examples, in the process 210, the case images 205 of each case may be automatically provided for display to the user according to the order of the cases provided in the list of cases 208. In some examples, the case images 205 may be displayed via a user interface of the viewing application tool 108. For example, when the examination or analysis of the case image 205 for the current case (e.g., before the next case in the list) is completed, the case image 205 for the next case in the list of cases 208 may be automatically displayed in the user interface. In some examples, the pathologist may select a single user interface option such as a "next case" button or control element (e.g., by touching or clicking a mouse), may select a hot key (e.g., on a keyboard), and / or may provide other gestures or indications to indicate that the examination or analysis of the case image 205 of the current case is completed and the case image 205 of the next case in the list may be displayed. In other examples, direct input from the user may not be required and the viewing application tool 108 may automatically detect that the examination or analysis of the current case is completed (e.g., may automatically detect that an event associated with case examination completion has occurred). For example, a determination that review of the current case is complete may be made based on the user having viewed all case images 205 of the current case and / or having entered a report or initial assessment. Although the user interface of the view application tool 108 is described above as displaying cases in order for a pathologist to review, the user interface of any system used by a pathologist to review and / or analyze cases or WSIs may be configured to include the same functionality, including other user interfaces of tissue viewing platform 100 (e.g., aside from the view application tool 108).

[0080] In some examples, when one or more case images 205 from a case are displayed to a user, the user may receive specific images from within the case in a previously specified order. As one non-limiting example, when a case is displayed to a user, the images for that case may be ordered according to the prioritization techniques described in U.S. patent application Ser. No. 16 / 887,855, filed May 29, 2020, which is incorporated by reference herein in its entirety.

[0081] In a further example, as discussed above, a separate machine learning system (e.g., of the slide analysis tool 101) may be utilized to analyze images within a case before displaying the images to one or more users. For example, the separate machine learning system may identify regions of interest for one or more users, which may be provided as input to the case sequence optimization tool 141 and / or other processes. The regions of interest may be determined according to the techniques described in U.S. Patent Application No. 17 / 014,532, filed September 8, 2020, which is incorporated herein by reference in its entirety. Additionally, the images themselves may be displayed to the user according to the techniques described in U.S. Patent Application No. 17 / 014,532.

[0082] The case sequence optimization tool 141 can be operated in a number of different contexts, including clinical, research, and education contexts. In some examples, the training and implementation of the trained system to determine optimized case sequences may depend on the context. The training and implementation of the trained system is described below for each of the clinical, research, and education contexts in turn.

[0083] 3A is a flowchart of an example method 300 for training a system to determine optimized case sequences in a clinical context in accordance with the techniques presented herein. The method 300 of FIG. 3A illustrates steps that may be performed, for example, by the case sequence optimization tool 141. Alternatively, the method 300 may be performed by an external system and a trained system may be provided to the case sequence optimization tool 141 for implementation.

[0084] In step 302, multiple training data sets may be received. The training data sets may include variables 202, constraints 204, and / or case images 205. Additional training data sets may include data related to general usage of the application, and the usage data may also include tracking data (e.g., variables 202) related to pathologist fatigue and / or user burnout. For example, the relative review speed of individual pathologists (e.g., as indicated by mouse speed, keystroke speed, eye tracking speed, total time reviewing a particular type of slide, total time reviewing a particular type of case, etc.) may be determined based on case type, experience level with that type of case, time of day, time since the pathologist's last break, experience level, etc. By utilizing this type of data for training, the trained system may, for example, predict the fatigue level of pathologists, either collectively or individually, moment by moment, or at any point throughout the day, which may be taken into account in the optimization.

[0085] The received data set may also include data (e.g., variables 202) from the viewing application tool 108 that tracks, for example, how the pathologist navigates the pathology slides. In one example, the data may include a measurement of the time the pathologist examines each slide. The data may further include tracking whether the user is using shortcuts and / or hotkeys to perform actions or tasks associated with the tissue viewing platform 100 and / or the amount of time the user gazes at the pathology slide. As more data is collected about the pathologist, the system may be retrained to allow for greater specificity to individual pathologists to improve case assignment and case sequencing accuracy. If no data has been collected about the pathologist (e.g., if the pathologist is new to the lab or site), the trained system may be applied in general. At this point, the system may begin receiving and retraining data for the new pathologist. For example, this data may be utilized as input for variables 202 to update the case assignment and sequencing output for a particular user.

[0086] The received data set (e.g., variables 202) may further include survey data and historical data. Survey data may include data regarding satisfaction, pathologist demographics, experience level, and any other data received from pathologists' preferences and dislikes. Historical data may include data regarding pathologists' past shifts (e.g., days and hours worked), years of experience, general experience level, experience with specific types of slides, general pathologist preferences, and initial quality assessment of diagnoses (e.g., whether an initial diagnosis was refuted by a second test). Historical data may also include general sample type distribution (e.g., average number of samples and / or samples of a given type received per day) as well as distribution of sample types at a particular site (e.g., a laboratory depending on its location near a specialized hospital or clinical center may generally receive more cases and / or more cases of a particular type).

[0087] In addition, the training data set may include information associated with any of the types of variables 202 and / or constraints 204 described above with reference to FIG.

[0088] Optionally, the slide analysis tool 101 may utilize, for example, one or more machine learning systems to evaluate slides to determine, for example, the time spent evaluating against the complexity of the case, the type of case, the number and amount of regions of interest, the amount of tissue per slide, the number of slides in the case, and / or the number of previous cases associated with the same patient that the pathologist must review. In some examples, the slide analysis tool 101 may determine the complexity of a case, at least in part, by determining the number of regions of interest (e.g., determining salient regions).

[0089] In step 304, the system may be trained using one or more of the data sets. For example, the system may train the system using the AI-assessed images and / or the received data set to be configured to determine a case sequence that optimizes one or more of a plurality of variables given one or more constraints in a clinical environment. In a clinical environment or setting, the optimized variable or variables may be related to pathologist efficiency. For example, the optimized variables may include number of cases completed per day, quality of initial assessment / quality of diagnosis (e.g., to achieve a high percentage of first-time correct diagnosis), pathologist retention rate, and / or number of cases per day with minimal pathologist burnout. As described in detail above with reference to FIG. 2, the trained system may be a trained ML system or may apply rule-based logic in response to optimization.

[0090] 3B is a flow chart illustrating an example method 350 for determining case sequence optimization in a clinical context using a trained system, according to one or more example embodiments herein. The example method 350 of FIG. 3B (e.g., steps 352-358) illustrates steps that may be performed, for example, by the case sequence optimization tool 141. These steps may be performed automatically or in response to a request from a user (e.g., a pathologist, a department or lab manager, an administrator, etc.). Alternatively, the method 350 may be performed by any computer process system, such as device 700, capable of receiving image input and capable of storing and executing the trained system described in FIG. 3A.

[0091] At step 352, the trained system may receive a plurality of electronic medical images of pathology slides associated with a pathology case. The received electronic medical images of pathology slides may correspond to and be the same types of slides as described for case images 205 from FIG. 2. At step 354, the trained system may further receive any constraint data 204 and / or variable data 202. The constraint data 204 may be any of the constraint data 204 described in FIG. 2. The variable data 202 may be any of the variable data 202 described in FIG. 2.

[0092] In step 356, the trained system may determine one or more orders of pathology cases for review by one or more users. In some examples, a subset of cases may be assigned to each available user (e.g., a pathologist) at the site, and an order of review of the cases in each subset may be determined. Optionally, the order may also define an order for the images of each case. The determined order may optimize one or more variables from the received variable data, subject to one or more constraints from the received constraint data. In some examples, the trained system may receive user-selected variables and / or variables to optimize. In other examples, the trained system may automatically select the variables to be optimized.

[0093] In step 358, the trained system may output the determined order of the pathology cases (e.g., in a list) for review by one or more pathologists. In some examples, the electronic medical images for each case may be automatically displayed (e.g., automatically navigated) in order, as described in more detail with reference to process 210 of FIG.

[0094] In some examples, input electronic medical images of pathology slides for new cases may be provided to the trained system as they are received by the case sequence optimization tool 141. The trained system may update the optimized sequence continuously or at predefined intervals.

[0095] In one example, the trained system may optimize case ordering for one or more individual pathologists. In another example, the trained system may optimize case ordering for a site or group of pathologists, rather than for individual pathologists within a site or group. In a further example, the trained system may optimize case ordering more generally, without considering individual pathologist or site data.

[0096] In some examples, the case sequence optimization tool 141 may also be capable of generating and displaying notifications to one or more users to prompt them to take a break while they are reviewing cases. For example, the tool may provide a warning that a certain time would be a good time to conduct research if a certain portion or portions of the day should be devoted to research (e.g., constraint 204). Additionally, if the tool notices a decline in the quality of testing throughout the day, the tool may prompt for a break to attempt to prevent the user from declining in quality of testing.

[0097] In a further example, the case sequence optimization tool 141 may monitor, for each user, the quality of diagnosis, the number of cases examined per day, how the examination rate drops throughout the day, survey data regarding satisfaction, and user demographics. Additionally, the case sequence optimization tool 141 may automatically map individual variables 202 and feed them to the trained system for retraining, which may occur at predefined and / or user-defined intervals. For example, the case sequence optimization tool 141 may obtain user data from case examination sessions and feed that data back to the trained system, which may then be used to tune and / or modify the trained system so that the output is more accurate or results in better optimization when used to determine case sequences for new pathologists and any particular repeat pathologists. This may include leveraging the output from a machine learning system (e.g., of the slide analysis tool 101) that analyzed the slides and giving the appropriate cases to the appropriate pathologists. For example, if a user / type of slides typically receives second opinions / examinations, the trained system may provide these cases earlier in the optimized case sequence list. The system may additionally take into account which pathologists / users are available and what level of expertise they may have. A trained system may further identify which cases require a second opinion or examination.

[0098] 4A is a flow chart of an exemplary method 400 for training a system to determine optimized case sequences in a research setting according to the techniques presented herein. The method 400 of FIG. 4A illustrates steps that may be performed, for example, by the case sequence optimization tool 141 described above in FIG. 1C. Alternatively, the method 400 may be performed by an external system and a trained system may be provided to the case sequence optimization tool 141 for implementation.

[0099] In research situations, including clinical research and clinical trial environments, it may not be necessary to render a diagnosis as part of case examination. For example, a user may be tasked with identifying certain aspects of a pathology slide (e.g., certain features or patterns), but the diagnosis associated with these features or patterns may not be important or may already be known. As one illustrative example, in a molecular diagnostics laboratory, a user may inspect a pathology slide to determine what should be machine scraped rather than circling a tumor and diagnosing the tumor. Thus, in a research situation, a trained system may perform case sequence optimization for some performance metric rather than for diagnostic quality. For example, a speed variable may be prioritized over an accuracy variable in case sequence optimization.

[0100] In general, the training phase detailed in steps 402-404 of method 400 may incorporate any of the steps or features described in method 300 (e.g., the training phase in a clinical setting). For example, the trained system may receive a dataset in step 402 that includes any of the datasets received in step 302 of method 300 described with reference to FIG. 3A.

[0101] In step 404, the system may be trained using one or more of the data sets. For example, the system may utilize the AI-assessed images and / or the received data set to train the system so that it is configured to determine a case sequence that optimizes one or more of the multiple variables given one or more constraints in a research environment. For example, the system may be trained in step 404 similar to step 304, except that the optimized variable may be different. For example, in a clinical research and / or clinical trial environment or setting, the optimized variable or variables may be related to pathologist efficiency or any other optimal variable related to clinical research. For example, the optimized variables may include the number of cases completed, the number of salient areas examined within a given time period, and / or the number of slides examined in a day. As described in detail above with reference to FIG. 2, the trained system may be a trained ML system or may apply rule-based logic in response to the optimization.

[0102] 4B is a flow chart illustrating an example method 450 for determining case sequence optimization in a research setting using a trained system, according to one or more example embodiments herein. The example method 450 of FIG. 4B (e.g., steps 452-458) illustrates steps that may be performed, for example, by the case sequence optimization tool 141. These steps may be performed automatically or in response to a request from a user (e.g., a pathologist, a department or lab manager, an administrator, etc.). Alternatively, method 450 may be performed by any computer process system, such as device 700, capable of receiving image input and capable of storing and executing the trained system described in FIG. 4A.

[0103] In general, the manufacturing stages described in method 450 may incorporate any of the steps or features of method 350 described in FIG. 3B. In step 452, the trained system may receive a plurality of electronic medical images of pathology slides associated with pathology cases. In step 454, the trained system may further receive optional variable data 202 and / or optional constraint data 204. In step 456, the trained system may determine one or more orders of pathology cases for review by one or more users in a research environment. In some examples, a subset of cases may be assigned to each available user (e.g., pathologist) at a site, and an order of review of cases within each subset may be determined. Optionally, the order may also define an order for images of each case. The determined order may optimize one or more variables from the received variable data, subject to one or more constraints from the received constraint data. For example, the trained system may optimize for a pathologist's performance metric rather than quality of diagnosis in a research environment or setting. For example, a speed variable may be prioritized over an accuracy variable in case order optimization. In some instances, the user may specifically select which variables to optimize during this step, while in other instances the trained system may automatically select the variables to be optimized.

[0104] In step 458, the trained system may output the determined order of the pathology cases (e.g., in a list) for review by one or more pathologists. In some examples, the electronic medical images for each case may be automatically displayed (e.g., automatically navigated) in order, as described in more detail with reference to process 210 of FIG.

[0105] 5A is a flowchart of an example method 500 for training a system to determine optimized case sequences in an educational context in accordance with the techniques presented herein. The method 500 of FIG. 5A illustrates steps that may be performed by the case sequence optimization tool 141 for implementation, for example, as described above in FIG. 1C. Alternatively, the method 500 may be performed by an external system and a trained system may be provided to the case sequence optimization tool 141 for implementation.

[0106] The case sequence optimization tool 141 may be utilized in an educational context to train and / or test users (e.g., students in training to become pathologists and / or pathologists continuing their training). From an educational perspective, variation in case types may be necessary for general understanding or testing of users. Additionally, it may also be helpful to identify and / or present similar / identical case types to individual students based on areas needing improvement. Thus, the variables optimized for case sequence decisions in an educational context may be variables related to pathologist improvement rather than accuracy or speed. Additionally, the diagnosis associated with each case may be known by the case sequence optimization tool 141.

[0107] Additionally, the case sequence optimization tool 141 may allow the user to select certain case areas for practice. The case sequence optimization tool 141 may automatically suggest a break if quality is declining (e.g., the number of incorrect diagnoses or number of cases the user brings in within a predefined time period exceeds a threshold). The case sequence optimization tool 141 may further track a user's progress over time and store individual user strengths and weaknesses for future use. A user's strengths may refer to a particular area or pathology area where the user correctly identifies diagnoses at a rate above a certain threshold (e.g., 80% or more when the diagnosis is correct) or at a rate higher than the user's overall diagnosis rate. A user's weaknesses may refer to a particular area or pathology area where the user incorrectly identifies diagnoses at a rate below a certain threshold (e.g., 50% or less when the diagnosis is incorrect) or at a rate higher than the user's overall diagnosis rate.

[0108] In one exemplary case, the case sequence optimization tool 141 may utilize a trained system to provide an optimized case sequence for a test, such as a general exam or even a specific specialty exam. The trained system may optimize the case sequence for diagnostic quality and / or recognition of features known to be diagnostically important for the slides. Additionally, in a testing environment, the trained system may select cases for inclusion in the case sequence based on detected user weaknesses or strengths. Alternatively, the trained system may provide an even distribution of topics or a select set of topics while using new slides for all test takers.

[0109] In general, the training phase detailed in steps 502-504 of method 500 may incorporate any of the training systems described in other embodiments herein (e.g., in FIG. 3A and FIG. 4A). In step 502, the trained system may receive a dataset including pathogen slides that have been previously analyzed by either a user, a pathologist, and / or an AI to identify diagnoses and / or target characteristics, among other examples. The diagnosis data and / or characteristic data may be received along with the slides. That is, the dataset may be an annotated or labeled dataset. Additionally, any of the datasets received in step 302 of method 300 described with reference to FIG. 3A may also be received in step 502. Optionally, the dataset may further include information from a viewing application tool 108 that tracks how a user navigates through the slides. For example, the tracking data may include measurements such as the time a user spends examining a particular slide, the frequency of using or not using shortcuts / hotkeys, and / or a certain gaze duration.

[0110] In some examples, the trained system may categorize received pathogen slides with a subject category and subcategory. Training slides may be labeled with a subject category or subcategory when provided as input to the system.

[0111] In step 504, the system may be trained using one or more of the data sets. For example, the system may utilize the AI-assessed images and / or the received data set to train the system configured to determine a case sequence that optimizes one or more of the multiple variables given one or more constraints in an educational environment. For example, the system may be trained in step 504 similar to step 304, except that the variables optimized may be different. For example, in an educational environment, the one or more variables optimized may be improvement in diagnostic quality in one or more areas of pathology. In another example, the variables optimized may be improvement in identification of salient regions, maximization of improvement in correct diagnostic decisions, and / or maximization of improvement in pathologist efficiency. As described in detail above with reference to FIG. 2, the trained system may be a trained ML system or may apply rule-based logic in response to optimization.

[0112] 5B is a flow chart illustrating an example method for determining case sequence optimization in a teaching context using a trained system, according to one or more example embodiments herein. The example method 550 of FIG. 5B (e.g., steps 552-558) illustrates steps that may be performed, for example, by the case sequence optimization tool 141. These steps may be performed automatically or in response to a request from a user (e.g., a pathologist, a department or lab manager, an administrator, etc.). Alternatively, the method 550 may be performed by any computer process system capable of receiving image inputs, such as device 700, and capable of storing and executing the trained system described in FIG. 5A.

[0113] In general, method 550 may incorporate any of the steps or features described in other embodiments herein (e.g., FIG. 3B or FIG. 4B). In step 502, the trained system may receive a plurality of electronic medical images of pathology slides associated with a pathology case. These slides may already be pre-labeled to identify the type of pathology (e.g., diagnosis) and the region of interest. In another example, the received electronic images of pathology slides may correspond to and be the same types of slides as described for case image 205 of FIG. 2.

[0114] In step 554, the trained system may receive any variable data 202 and constraint data 204. For example, the variable data 202 may be related to training, such as identifying areas for improvement or increasing accuracy. Additionally, the constraint data 204 may include the amount of time a particular user may have to study, train, or undergo diet / testing. Additionally, during use of the tissue observation platform 100, further variable 202 data and constraint data 204 may be provided to the trained system and / or tracked by the case sequence optimization tool 141.

[0115] In step 556, the trained system may determine one or more sequences of pathology cases for review by one or more users (e.g., one or more pathologists-in-training or students) in an educational environment. In some examples, in an educational environment or setting, the case sequence may be optimized to maximize improvement in identification of salient regions, maximize improvement in correct diagnostic calls, and / or maximize improvement in pathologist efficiency. These variables 202 may be selected by a user / administrator or may already be set as preferred variables 202 to be optimized in an educational setting.

[0116] In step 558, the trained system may output the determined order of the pathology cases (e.g., in a list) for review by one or more pathologists. In some examples, the electronic medical images for each case may be automatically displayed (e.g., automatically navigated) in order, as described in more detail with reference to process 210 of FIG.

[0117] Optionally, the case sequence optimizer 141 may prompt the user to take a break. For example, the system may provide an alert when the day's teaching goals (e.g., constraints 204) may be met. The case sequence optimizer 141 may track total training time and user accuracy overall and within a category or subcategory. Additionally, the case sequence optimizer 141 may automatically map individual performance and feed that data to the trained system, which may be used to retrain the system at predefined time intervals / events. This may include taking user data from a training or test session where cases are presented to the user in the order in which they were adjudicated, and feeding it back to the trained system, which is then used to retrain the system to be better optimized, initially from the next user and / or for any particular user.

[0118] In one example, the trained system may be implemented to determine an optimized case order for training one or more individuals for a particular specialty within pathology, such as identifying the grade of a tumor located on a particular slide. In this case, the training data set of step 502 and the received medical images of step 552 may include various slides with tumors of different grades. Additionally, the training data and / or inserted medical images of step 552 may be labeled to indicate the location and grade of the tumor. The case order optimization tool 141 may output to the user various slides with tumors of different grades located on them according to the optimized case order determined by the trained system. As the user progresses (e.g., analyzes more slides), the case order optimization tool 141 may identify which grades of tumors the user has correctly identified and which types the user has not correctly identified. The case order optimization tool 141 may provide this information to the trained system, which may progress by updating the optimized case order to give the user more cases with tumor grades that the user failed to identify at a higher rate. The case sequence optimization tool 141 may continue to output cases with tumors of a certain grade until the user identifies tumors of a certain grade with a predetermined accuracy percentage.

[0119] 6 is a flow chart illustrating an example method 600 for optimizing an order of cases displayed to one or more users. At step 602, a number of variables and one or more constraints may be received. At step 604, a number of pathology cases may be received. Each case of the number of pathology cases may include one or more medical images of at least one pathology specimen associated with a patient.

[0120] In step 606, one or more medical images from each case, a number of variables, and one or more constraints may be input to a trained system. The trained system described herein may be a trained machine learning system. For example, the trained machine learning system may provide slides in dependency order using dependency constraints and a constraint solver, as opposed to a scoring system that ranks all possible orderings.

[0121] At step 608, an order for user testing of the multiple cases may be received. The order may be used to optimize one or more of the multiple variables based on one or more constraints. At step 610, each case of the multiple cases may be automatically provided to the user for testing according to the order.

[0122] As shown in Figure 7, device 700 may include a central processing unit (CPU) 720. CPU 720 may be any type of processor device, including, for example, any type of dedicated or general purpose microprocessor device. As will be appreciated by those skilled in the art, CPU 720 may also be a single processor in a multi-core / multi-processor system operating alone or in a cluster of computing devices operating in a cluster or server farm. CPU 720 may be connected to a data communications infrastructure 710, such as a bus, message queue, network, or multi-core message passing scheme.

[0123] The device 700 may also include a main memory 740, such as random access memory (RAM), and may also include a secondary memory 730, such as read only memory (ROM), which may be, for example, a hard disk drive or a removable storage drive. Such removable storage drives may include, for example, floppy disk drives, magnetic tape drives, optical disk drives, flash memory, and the like. The removable storage drive in this example reads from and / or writes to the removable storage unit in a well-known manner. The removable storage may include floppy disks, magnetic tapes, optical disks, and the like that are read by and written to the removable storage drive. As will be appreciated by those skilled in the art, such removable storage units typically include computer usable storage media having computer software and / or data stored thereon.

[0124] In alternative implementations, secondary memory 730 may include similar means for allowing computer programs or other instructions to be loaded into device 700. Examples of such means may include program cartridges and cartridge interfaces (such as those found in video game devices), removable memory chips (such as EPROMs or PROMs) and associated sockets, and other removable storage units and interfaces that allow software and data to be transferred from removable storage units to device 700.

[0125] Device 700 may also include a communications interface ("COM") 760. Communications interface 760 allows software and data to be transferred between device 700 and external devices. Communications interface 760 may include a modem, a network interface (such as an Ethernet card), a communications port, a PCMCIA slot and card, or the like. The software and data transferred via communications interface 760 may be in the form of signals, which may be electronic, electromagnetic, optical, or other signals receivable by communications interface 760. These signals may be provided to communications interface 760 via a communications path of device 700, which may be implemented using, for example, wire or cable, fiber optics, a phone line, a cellular phone link, an RF link, or other communications channel.

[0126] The hardware elements, operating systems, and programming languages ​​of such equipment are conventional in nature and are presumed to be sufficiently familiar to those skilled in the art. The device 700 may also include input / output ports 650 for connecting input / output devices such as a keyboard, mouse, touch screen, monitor, display, etc. Of course, the various server functions may be performed in a distributed manner on multiple similar platforms in order to distribute the processing load. Alternatively, the server may be implemented by suitable programming of one computer hardware platform.

[0127] Throughout this disclosure, references to components or modules generally refer to items that may be logically grouped together to perform a function or group of related functions. Like reference numbers are generally intended to refer to the same or similar components. The components and / or modules may be implemented in software, hardware, or a combination of software and / or hardware.

[0128] The aforementioned tools, modules, and / or functions may be performed by one or more processors. A "storage" type medium may include any or all of the tangible memory of a computer, processor, or associated modules such as various semiconductor memories, tape drives, disk drives, etc., which may provide non-transitory storage for software programming at any time.

[0129] The software may be communicated via the Internet, a cloud service provider, or other telecommunications network. For example, the communication may enable loading of the software from one computer or processor into another. As used herein, unless limited to non-transitory tangible "storage" media, terms such as computer "readable medium" or machine "readable medium" refer to any medium that participates in providing instructions to a processor for execution.

[0130] The foregoing general description is merely exemplary and explanatory and is not intended to limit the present disclosure. Other embodiments may be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only.

Claims

1. 1. A computer-implemented method for processing electronic medical images to optimize pathology case review sequencing, comprising: Receiving a plurality of variables and one or more constraints; receiving a plurality of pathology cases, each case of the plurality of pathology cases including one or more medical images of at least one pathology specimen associated with a patient; providing the one or more medical images from each case, the plurality of variables, and the one or more constraints as inputs to a trained system; receiving, as an output of the trained system, an order of user testing of the plurality of cases to optimize one or more of the plurality of variables based on the one or more constraints; automatically providing each case of the plurality of cases to a user for review according to the order; said computer-implemented method comprising:

2. 2. The method of claim 1, wherein the trained system determines a plurality of potential sequences and a score for each of the plurality of potential sequences indicative of an optimization level of the one or more of the plurality of variables, and the sequence provided as an output is one of the plurality of potential sequences having a highest score.

3. 3. The method of claim 2, wherein when at least a first variable and a second variable of the plurality of variables are optimized, the trained system determines a first score of the first variable and a second score of the second variable for each of the plurality of potential orderings, and the ordering provided as output is one of the plurality of potential orderings having a maximized overall score based on the first score and the second score.

4. The method of claim 1 , wherein the one or more of the plurality of variables to be optimized are user-selected variables.

5. The method of claim 1 , wherein the trained system is a trained machine learning system or a trained rule-based system.

6. receiving the determined one or more characteristics of the medical image as output from a trained machine learning system configured to process the one or more medical images from each case of the plurality of pathology cases to determine one or more characteristics of the medical image; providing the determined one or more characteristics of the medical image as a further input to the trained system, wherein the determined one or more characteristics include case complexity, case type, number of regions of interest per medical image or per case, amount of tissue per medical image, and image quality; The method of claim 1 further comprising:

7. receiving one or more additional pathology cases, each of the one or more additional pathology cases including one or more medical images of at least one pathology specimen associated with the patient; providing the one or more medical images from each of the one or more additional pathology cases as further inputs to the trained system; receiving an updated sequence as an output of the trained system; The method of claim 1 further comprising:

8. 2. The method of claim 1, wherein the trained system is further configured to assign a subset of the plurality of cases to each of a plurality of users, and the order received as output from the trained system comprises an order of user testing of each subset of the plurality of cases.

9. 10. The method of claim 1, further comprising generating a notification to prompt a user to take one or more breaks to increase optimization of one or more of the variables based on information received from the training system.

10. automatically providing each case of the plurality of cases to the user for review according to the order; The method of claim 1 , further comprising automatically navigating from a first case to a subsequent case in the sequence based on an indication that examination of the first case is complete.

11. The method of claim 10 , wherein the indication is an event associated with an input received from the user or an automatically detected case test completion.

12. monitoring values ​​associated with the plurality of variables as the user examines the plurality of cases; providing the values ​​to the trained system, the trained system being retrained based on the values ​​for future optimization; The method of claim 1 further comprising:

13. 1. A system for processing electronic medical images, comprising: at least one memory for storing instructions; and at least one processor configured to execute the instructions to perform operations, the operations comprising: Receiving a plurality of variables and one or more constraints; receiving a plurality of pathology cases, each case of the plurality of pathology cases including one or more medical images of at least one pathology specimen associated with a patient; providing the one or more medical images from each case, the plurality of variables, and the one or more constraints as inputs to a trained system; receiving, as an output of the trained system, an order of user testing of the plurality of cases to optimize one or more of the plurality of variables based on the one or more constraints; automatically providing each case of the plurality of cases to a user for review according to the order; The system comprising:

14. 14. The system of claim 13, wherein the trained system determines a plurality of potential sequences and a score for each of the plurality of potential sequences indicative of an optimization level of the one or more of the plurality of variables, and the sequence provided as an output is one of the plurality of potential sequences having a highest score.

15. 15. The system of claim 14, wherein when at least a first variable and a second variable of the plurality of variables are optimized, the trained system determines a first score of the first variable and a second score of the second variable for each of the plurality of potential orderings, and the ordering provided as output is one of the plurality of potential orderings having an overall score that is maximized based on the first score and the second score.

16. The system of claim 13 , wherein the one or more of the plurality of variables to be optimized are user-selected variables.

17. The system of claim 13 , wherein the trained system is a trained machine learning system or a trained rule-based system.

18. receiving the determined one or more characteristics of the medical image as output from a trained machine learning system configured to process the one or more medical images from each case of the plurality of pathology cases to determine one or more characteristics of the medical image; providing the determined one or more characteristics of the medical image as a further input to the trained system, wherein the determined one or more characteristics include case complexity, case type, number of regions of interest per medical image or per case, amount of tissue per medical image, and image quality; The system of claim 13 further comprising:

19. A non-transitory computer readable medium storing instructions that, when executed by a processor, perform operations for processing electronic medical images, the operations including: Receiving a plurality of variables and one or more constraints; receiving a plurality of pathology cases, each case of the plurality of pathology cases including one or more medical images of at least one pathology specimen associated with a patient; providing the one or more medical images from each case, the plurality of variables, and the one or more constraints as inputs to a trained system; receiving, as an output of the trained system, an order of user testing of the plurality of cases to optimize one or more of the plurality of variables based on the one or more constraints; automatically providing each case of the plurality of cases to a user for review according to the order; The computer-readable medium comprising:

20. 20. The computer-readable medium of claim 19, wherein the trained system determines a plurality of potential sequences and a score for each of the plurality of potential sequences indicative of an optimization level of the one or more of the plurality of variables, and the sequence provided as an output is one of the plurality of potential sequences having a highest score.

Images