Collaborative artificial intelligence method and system

Abstract

A method and system for audibly broadcasting a response to a user based on the user's query about a particular patient's molecular report, the method comprising: receiving an audible query from the user to a microphone coupled to a collaboration device; identifying at least one intent associated with the audible query; identifying at least one data action associated with the at least one intent; associating each of the at least one data action with a first set of data presented in the molecular report; performing each of the at least one data action on a second data set to generate response data; generating an audible response file associated with the response data; and providing the audible response file for broadcast via a speaker coupled to the collaboration device.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Patent Application No. 62 / 871,667, entitled "COLLABORATIVE ARTIFICIAL INTELLIGENCE METHOD AND SYSTEM," filed July 8, 2019, U.S. Provisional Patent Application No. 62 / 855,646, entitled "COLLABORATIVE ARTIFICIAL INTELLIGENCE METHOD AND APPARATUS," filed May 31, 2019, and U.S. Provisional Patent Application No. 62 / 835,339, entitled "COLLABORATIVE ARTIFICIAL INTELLIGENCE METHOD AND APPARATUS," filed April 17, 2019.

[0002] Applications Incorporated by Reference Each of the following U.S. patent applications is incorporated herein by reference in its entirety: (1) U.S. patent application Ser. No. 16 / 657,804, entitled “DATA BASED CANCER RESEARCH AND TREAMENT SYSTEMS AND METHODS,” filed on October 18, 2019. (2) U.S. patent application Ser. No. 16 / 671,165, entitled “USER INTERFACE, SYSTEM, AND METHOD FOR COHORT ANALYSIS,” filed on December 31, 2019. (3) U.S. patent application Ser. No. 16 / 732,168, entitled “A METHOD AND PROCESS FOR PREDICTING AND ANALYZING PATIENT COHORT RESPONSE, PROGRESSION, AND SURVIVAL,” filed on December 31, 2019.

[0003] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT Not applicable [Background technology]

[0004] The field of the disclosure is systems for accessing and manipulating large, complex data sets in a manner that enables system users to develop new insights and conclusions with minimal user interface friction that impedes access and manipulation.

[0005] The present disclosure describes innovations that are described in the context of an exemplary healthcare professional collaborating with a patient to diagnose disease conditions, prescribe treatments, and administer those treatments to improve overall patient health. Additionally, while many different types of healthcare professionals (e.g., physicians, psychologists, physical therapists, nurses, administrators, researchers, insurance professionals, pharmacists, etc.) in many different medical specialties (e.g., cancer, Alzheimer's, Parkinson's, psychiatric, cardiology, immunology, infectious diseases, and diabetes) will benefit from the disclosed innovations, unless otherwise noted, the innovations are described in the context of an exemplary oncologist / researcher (hereinafter "oncologist") who collaborates with patients to diagnose a cancer condition (e.g., all physiological, habitual, medical history, genetic, and treatment effect factors), understand and evaluate existing data and guidelines of patients similar to theirs, prescribe treatments, administer those treatments, and observe patient outcomes, all to improve overall patient health, and / or those conducting cancer medical research.

[0006] Many professions require complex thinking that requires considering many factors when selecting a solution to an encountered situation, hypothesizing new factors and solutions, and testing the new factors and solutions to ensure they are effective. For example, an oncologist considering a particular patient's cancer status must optimally consider many different factors when assessing the patient's cancer status, as well as many factors when creating and managing an optimized treatment plan. For example, these factors may include the patient's family history, past medical conditions, current diagnosis, the genomic / molecular profile of the patient's genetic DNA and the patient's tumor's DNA, current nationally recognized guidelines for standards of care within that cancer subtype, recently published research on that patient's condition, available clinical trials relevant to that patient, available pharmaceutical therapies, and other potential therapeutic interventions that may be good options for the patient, as well as data from similar patients. Additionally, cancer, and cancer treatment research, is rapidly evolving, requiring researchers to continually utilize data, new research, and new treatment guidelines to think critically about new factors and treatments when diagnosing the cancer status and optimized treatment plans.

[0007] In particular, it is no longer possible for oncologists to stay up to date on all new research in the field of cancer treatment. Similarly, it is extremely difficult for oncologists to manually analyze the medical records and outcomes of thousands or millions of cancer patients every time they wish to make a specific treatment recommendation regarding a particular patient being treated by that oncologist. As a first problem, oncologists often do not even have access to health information from institutions other than their own. In the United States, the implementation of a federal law known as the Health Insurance Portability and Accountability Act of 1996 ("HIPAA") has significantly limited the ability of one health care provider to access the medical records of another. Furthermore, health care systems face administrative, technical, and financial challenges in making data available to third parties for aggregation with similar data from other health care systems. To the extent that medical information from multiple patients seen at multiple providers is aggregated in a single repository, a system and method is needed to structure that information using a common data dictionary or library of data dictionaries. When multiple institutions are responsible for developing a single aggregated repository, significant discrepancies may arise regarding the structure of one or more data dictionaries, how the data is accessed, which individuals or other donors are authorized to access the data, and how much data is accessible. Furthermore, the scope of searchable data can be overwhelming for oncologists who want to perform manual reviews. Every patient has health information that contains hundreds or even thousands of data elements. Including sequencing information in the health information accessed and analyzed, such as from next-generation sequencing, greatly increases the amount of health information that can be analyzed. For example, a single FASTQ or BAM file generated during the process of whole-exome sequencing consumes gigabytes of storage, despite containing sequences of only the patient's exome, which is believed to be approximately 1-2% of the entire human genome.

[0008] In this regard, an oncologist may have a simple question – “What is the best drug for this particular patient?” – the answer of which requires a vast amount of health information, analytical software modules to analyze that information, and a hardware framework to enable those modules to run to provide an answer. Almost every query / idea / concept is a work in progress that evolves over time as critical thinking is applied and the relationships of one or more additional relevant factors are recognized and / or better understood. Every query begins as a hypothesis rooted in a set of interrelated raw materials (e.g., data). A hypothesis is typically tested by asking a question related to the hypothesis and determining whether the hypothesis is consistent and durable when considered in light of the raw materials and the answers to the question. A consistent / durable hypothesis becomes dependent on ideas (i.e., facts) and additional raw materials to generate the next iteration of the initial idea, not just entirely new ideas.

[0009] When considering a particular cancer condition, oncologists consider known factors (e.g., patient condition, previous treatments, treatment efficacy, etc.), develop a hypothesis regarding an optimized treatment, consider that hypothesis in light of previous data and previous studies relating similar cancer conditions to treatment efficacy, and may prescribe the hypothesized treatment to the patient if the previous data indicates high efficacy for the treatment hypothesis. If the data indicates low efficacy of the treatment, the oncologist will reconsider and generate a different hypothesis, continuing the cycle of iterative testing and conclusion until an effective treatment plan is identified. Cancer researchers perform similar iterative hypothesis, data testing, and conclusion processes to derive new cancer research insights.

[0010] Tools have been developed to help oncologists diagnose cancer conditions, select and manage optimized treatments, and explore and explore new cancer condition factors, new cancer conditions (e.g., diagnoses), new treatment factors, new treatments, and new efficacy factors. For example, large cancer databases have been developed and maintained for access and manipulation by oncologists to explore diagnostic and treatment options, as well as new insights and treatment hypotheses. Computers enable access to and manipulation of cancer data and its derivatives.

[0011] Because cancer data tends to be voluminous and multifaceted, many useful representations include a significant amount of detail and specific arrangement of the data or data derivatives that are best visually represented. For this reason, oncology and research computer workstations typically include traditional interface devices, such as one or more large flat panel display screens for presenting data representations, and a keyboard, mouse, or other mechanical input device for entering information, operating interface tools, and presenting many different data representations. Often, the workstation computer / processor runs an electronic medical record (EMR) or medical research application program (hereinafter "research application") that presents various data representations and on-screen cursor-selectable control icons for selecting various data access and manipulation options.

[0012] While traditional computers and workstations work well as interfaces for data access and manipulation, they suffer from several drawbacks. First, using a computer interface often requires oncologists to click multiple times through various interfaces to find specific information. This is a tedious and time-consuming process that often does not result in oncologists achieving the desired results and receiving answers to the questions they are trying to ask.

[0013] Second, it is often difficult to capture hypothetical queries as they arise, and ideas are not followed up on in a timely manner or are lost forever. Because queries are not restricted to a specific time schedule, they often arise at inconvenient times when the oncologist is not logged into their workstation and is using a research application that can be used to capture and test ideas. For example, an oncologist may be at home when they become curious about some aspect of a patient's cancer condition, or some statistics related to one of their patients, or when they first develop a treatment hypothesis for a particular patient's cancer condition. In this case, if the oncologist's workstation is in a remote medical facility, the oncologist cannot easily query the database or capture or test a hypothesis.

[0014] Also in this case, even if an oncologist can use a laptop or other home computer to access a research application from home, friction associated with using the application often gets in the way. In this regard, accessing the application requires the oncologist to retrieve the laptop or physically travel to a stationary computer at home, boot up the computer's operating system, log on to the computer (e.g., enter a username and password), select and start the research application, navigate through several application screenshots to the desired database access tool suite, and then enter query or hypothesis definition information to begin hypothesis testing. In many cases, this application access friction is enough to discourage immediate query or hypothesis capture and testing, especially if the oncologist simply assumes that the query or hypothesis will be remembered the next time he or she accesses a computer interface. As anyone with a lot of ideas knows, ideas are fleeting, and therefore ideas not captured immediately are often lost. More importantly, oncologists typically have limited time to spend on each patient case and need questions and queries resolved immediately while evaluating information specific to that patient.

[0015] Third, often times, a new query or hypothesis arises while an oncologist is engaged in other activities unrelated to oncology activities. Here, as with many, it is simply not considered for immediate consideration and testing via traditional research applications. Again, the idea gets lost if not captured immediately.

[0016] Fourth, oncology and research data activities often involve a series of successive questions or requests (hereafter "Requests") that focus on increasingly detailed data responses where the intermediate results are not particularly interesting and therefore require the oncologist / researcher to iteratively enter additional input to define the next level of request. Furthermore, while in many cases a visual representation of the data response to an oncology and research request is optimal, in other cases the visual representation tends to hinder usability and may even be overwhelming. In these cases, while a visual representation is usable, the representation may require significant time and effort to consume the presented information (e.g., reading the results, mentally summarizing the results, etc.). In short, traditional oncology interfaces are often cumbersome to use.

[0017] Furthermore, today oncologists and other specialists do not have an easy mechanism to query large, complex databases and receive answers in real time without interacting with electronic health record systems or other cumbersome software solutions. In particular, there is a need for systems and methods that allow a provider to query a device using his or her voice with questions related to the optimal care of his or her patient, with answers to those questions being generated from unique datasets that provide context and new information relevant to the patient, including vast amounts of real-world historical clinical information combined with other forms of medical data, such as molecular data from omics sequencing and imaging data, as well as data derived from such data using analytics to determine the optimal pathway for that single patient.

[0018] What is needed, therefore, is an intuitive interface to complex databases that allows oncologists, researchers, and other professionals and database users to access and manipulate the data in a variety of ways to generate queries, test hypotheses or new ideas, and thereby explore those ideas in the context of various data sets while minimizing the friction of access and manipulation. It would be advantageous for the interface to be always present, or at least portable, and therefore essentially always available. It would also be advantageous for the system associated with the interface to memorialize user interface interactions, so that an oncologist or researcher may later reconsider the interactions, so that they can be re-engaged with the purpose of continuing the line of questioning or hypothesis testing without losing previous thinking.

[0019] It would also be advantageous to have a system that captures the thoughts of an oncologist for several purposes, such as developing better medical aid systems, generating automated records and documentation, and providing services such as scheduling appointments, tests and procedures, and preparing prescriptions.

[0020] It would also be advantageous to have an interface available across several different form factors. [Prior art documents] [Patent documents]

[0021] [Patent Document 1] U.S. Patent Application No. 16 / 671,165, filed October 31, 2019 [Patent Document 2] U.S. Patent Application No. 16 / 732,168, filed December 31, 2019 [Patent Document 3] U.S. Patent Application No. 16 / 657,804, filed October 18, 2019 Summary of the Invention [Means for solving the problem]

[0022] It has been recognized that a relatively small and portable voice-activated and audio-responsive interface device (hereinafter, "collaborative device") can be provided that allows an oncologist to perform at least initial database access and manipulation activities. In at least some embodiments, the collaborative device includes a processor linked to each of a microphone, a speaker, and a wireless transceiver (e.g., a transmitter and a receiver). The processor executes software for capturing voice signals generated by the oncologist. An automatic speech recognition (ASR) system converts the voice signals into a text file, which is then processed by a natural language processor (NLP) or other artificial intelligence module (e.g., a natural language understanding module, etc.) to generate data operations (e.g., commands to perform some data access or manipulation process, such as query, filter, memorialize, clear previous query and filter results, make notes, etc.).

[0023] In at least some embodiments, the collaboration device is used within a collaboration system that includes a server that maintains and operates an industry-specific data repository. The data operations are received by the collaboration server and used to access and / or manipulate database data, thereby generating a data response. In at least some cases, the data response is returned to the collaboration device as an audio file that is broadcast to the oncologist as a result associated with the original query.

[0024] In some cases, the transcription of the voice signal to a text file is performed by the collaboration device processor, or in other cases, the voice signal is transmitted from the collaboration device to a collaboration server, which performs the transcription to the text file. In other cases, the text file is converted to a data operation by the collaboration device processor, or in other cases, the conversion is performed by the collaboration server. In some cases, the collaboration server maintains or has access to an industry-specific database, so that the server acts as an intermediary between the collaboration device and the industry-specific database.

[0025] In at least some embodiments, the collaboration device is a dedicated collaboration device that serves only as an interface to the collaboration server and industry-specific databases. In such cases, because the collaboration interface device is always on and capable of running only a single dedicated application program, the device requires no start-up time and can be essentially activated immediately via a single activation activity performed by the oncologist.

[0026] For example, in some cases, the collaboration device's motion sensors (e.g., accelerometer, gyroscope, etc.) are linked to the processor so that simply lifting the device can cause the processor to activate the application. In other cases, the collaboration device processor may be programmed to "listen" for the phrase "Hey query" and, when received, activate to capture the next voice signal utterance that serves as seed data for generating the text file. In other cases, the processor may be programmed to listen for a different activation phrase, such as the brand name of the system, or a combination of the brand name and a command display. For example, if the brand name of the system is "One," the activation phrase could be "One" or "Go One," etc. In still other cases, the collaboration device may simply listen for voice signal utterances that can be recognized as oncology queries and then automatically use the recognized queries as seed data for text generation.

[0027] In addition to providing audio responses to the data actions, at least in some cases, the system automatically records and stores the data actions (e.g., data defining the actions) and responses as collaboration records for subsequent access. The collaboration records may include one or the other or both of the original audio signal and the broadcast response, or the text file and text response corresponding to the data response. Here, the stored collaboration records provide details about the oncologist's search and data action activities that serve to automatically memorialize hypotheses or ideas the oncologist was considering. If the oncologist requests a series of queries, those queries and data responses may be stored as one-line questions that together provide more detailed information to characterize the oncologist's initial hypotheses or ideas. The system may then enable the oncologist to access the memorialized queries and data responses, re-entering the flow state associated therewith, allowing hypothesis testing and data manipulation to continue using a workstation-type interface, or other computing device that includes a display screen and perhaps an audio device such as a speaker or microphone, and that is better suited to presenting more complex data sets and data representations.

[0028] In addition to simple data search queries, other voice signal data action types are possible. For example, the system may support a filter action where an oncologist's voice signal message defines a subset of an industry-specific database set. For example, an oncologist can say, "Access all medical records of male patients over 45 years old who have had pancreatic cancer since 1990," and have the system generate an associated subset of relevant data that meets the specified criteria.

[0029] Importantly, some data responses to oncology queries will be "audio-friendly", i.e., the response can be fully understood and comprehended when broadcast as an audio message. In other cases, the data responses may not be well suited for presentation simply as audio output. For example, if the query contains the phrase "Who was the patient you saw during your last office visit last Thursday?", an audio-friendly response might be "Mary Brown." On the other hand, if the query is "List all medications prescribed since 1978 for men over 45 years old with pancreatic cancer" and the response contains a list of 225 medications, the list would not be suitable for audio because it would take a long time to broadcast each list entry and comprehension of all the list entries would be questionable at best.

[0030] If the data response is best presented visually, the system may take alternative or additional steps to provide the response in a user-friendly format. The system may simply indicate, as part of the audio response, that the response data is better presented in a visual format and then present the audio response. If there is a large display screen nearby, such as a computer monitor or a television (TV), such as a smart TV, the system may pair with that display to present the visual data with or without the audio data. The system may simply indicate that a suitable audio response is not available. In some embodiments, the system may pair with a computing device that includes a display, such as a smartphone, tablet computer, etc.

[0031] Thus, at least some embodiments of the present invention allow intuitive and rapid access to complex data sets essentially anywhere within a wireless communication zone, allowing oncologists to initiate thought processes in real time as they arise. By answering questions as they arise, the system allows oncologists to dig deeper into the data in the moment and continue thought processes through the progression of the query. Some embodiments memorialize the oncologist's queries and responses, allowing the oncologist to subsequently revisit that information and continue queries related thereto. If visual and audio responses are available, the system may be adapted to provide visual responses when visual capabilities are present, or may simply store the visual responses as part of a collaboration record for subsequent access when the oncologist has access to a workstation, etc.

[0032] In at least some embodiments, the disclosure includes a method of interacting with a database to access data therein, the method for use with a collaboration device including a speaker, a microphone, and a processor, the method comprising: associating distinct sets of condition-specific intents and supporting information with different clinical report types, the supporting information including at least one intent-specific data operation for each condition-specific intent; receiving a voice query via the microphone for information; identifying a particular patient associated with the query; identifying condition-specific clinical reports associated with the identified patient; upon selecting one of the condition-specific intents, attempting to select one of the condition-specific intents associated with the identified condition-specific clinical report as a match for the query; performing at least one data operation associated with the selected condition-specific intent to generate results; and broadcasting the query response via the speaker using the results to form a query response.

[0033] In some cases, the method is for use with at least a first database including information in addition to the clinical report, the method further including the step of obtaining at least a subset of the information in addition to the clinical report in response to the query, and using the results to form the query response includes using the results and the additionally obtained information to form the query response.

[0034] In some cases, the at least one data operation includes at least one data operation for accessing additional information from a database, and obtaining at least the subset includes obtaining data for each of the at least one data operation for accessing additional information from the database.

[0035] Some embodiments include a method of interacting with a database to access data therein, the method for use with a collaboration device including a speaker, a microphone, and a processor, the method comprising: associating distinct sets of condition-specific intents and supporting information with different clinical report types, the supporting information including at least one intent-specific primary data operation for each condition-specific intent; receiving a voice query via the microphone for information; identifying a particular patient associated with the query; identifying condition-specific clinical reports associated with the identified patient; upon selection of one of the condition-specific intents, attempting to select one of the condition-specific intents associated with the identified condition-specific clinical report as a match for the query; performing a primary data operation associated with the selected condition-specific intent to generate a result; performing supplemental data operations on data from a database including data in addition to the clinical report data to generate additional information; and broadcasting the query response via the speaker using the results and the additional information to form a query response.

[0036] Some embodiments include a method for audibly broadcasting a response to a user based on the user's query about a particular patient's molecular report, the method comprising: receiving an audible query from a user to a microphone coupled to a collaboration device; identifying at least one intent associated with the audible query; identifying at least one data action associated with the at least one intent; associating each of the at least one data action with a first set of data presented in the molecular report; executing each of the at least one data action against a second data set to generate response data; generating an audible response file associated with the response data; and providing the audible response file for broadcast via a speaker coupled to the collaboration device.

[0037] In at least some cases, the audible query includes a question regarding a nucleotide profile associated with the patient. In at least some cases, the nucleotide profile associated with the patient is a cancer profile of the patient. In at least some cases, the nucleotide profile associated with the patient is a germline profile of the patient. In at least some cases, the nucleotide profile is a DNA profile. In at least some cases, the nucleotide profile is an RNA expression profile. In at least some cases, the nucleotide profile is a mutational biomarker.

[0038] In at least some cases, the mutation biomarker is a BRCA biomarker. In at least some cases, the audible query includes a question regarding a treatment. In at least some cases, the audible query includes a question regarding a gene. In at least some cases, the audible query includes a question regarding clinical data. In at least some cases, the audible query includes a question regarding a next generation sequencing panel. In at least some cases, the audible query includes a question regarding a biomarker.

[0039] In at least some cases, the audible query includes a question regarding an immune biomarker. In at least some cases, the audible query includes a question regarding an antibody-based test. In at least some cases, the audible query includes a question regarding a clinical trial. In at least some cases, the audible query includes a question regarding an organoid assay. In at least some cases, the audible query includes a question regarding a pathology image. In at least some cases, the audible query includes a question regarding a type of disease. In at least some cases, the at least one intent is an intent related to a biomarker. In at least some cases, the biomarker is a BRCA biomarker. In at least some cases, the at least one intent is an intent related to a clinical condition. In at least some cases, the at least one intent is an intent related to a clinical trial.

[0040] At least in some cases, at least one intent is related to a pharmaceutical. At least in some cases, the pharmaceutical intent is related to a pharmaceutical that is a chemotherapy. At least in some cases, the pharmaceutical intent is an intent related to a PARP inhibitor intent. At least in some cases, at least one intent is related to a genetic. At least in some cases, at least one intent is related to immunology. At least in some cases, at least one intent is related to a knowledge database. At least in some cases, at least one intent is related to a testing method. At least in some cases, at least one intent is related to a gene panel. At least in some cases, at least one intent is related to reporting. At least in some cases, at least one intent is related to an organoid process. At least in some cases, at least one intent is related to imaging.

[0041] In at least some cases, the at least one intent is related to a pathogen. In at least some cases, the at least one intent is related to a vaccine. In at least some cases, the at least one data action includes an action for identifying at least one treatment option. In at least some cases, the at least one data action includes an action for identifying knowledge about a treatment. In at least some cases, the at least one data action includes an action for identifying knowledge related to at least one pharmaceutical agent (e.g., "Which pharmaceutical agent is associated with high CD40 expression?"). In at least some cases, the at least one data action includes an action for identifying knowledge related to mutation testing (e.g., "Was Dwayne Holder's sample tested for a KMT2D mutation?"). In at least some cases, the at least one data action includes an action for identifying knowledge related to the presence of a mutation (e.g., "Does Dwayne Holder have a KMT2C mutation?"). In at least some cases, the at least one data action includes an action for identifying knowledge related to a tumor characteristic (e.g., "Is Dwayne Holder's tumor likely to be a BRCA2-driven tumor?"). In at least some cases, at least one data action includes an action for identifying knowledge related to testing requirements (e.g., "What percentage of tumors does Tempus require for a TMB result?"). In at least some cases, at least one data action includes an action for querying definitional information (e.g., "What is PDL1 expression?"). In at least some cases, at least one data action includes an action for querying expert information (e.g., "What is the clinical relevance of PDL1 expression?", "What are the common risks associated with Whipple surgery?"). In at least some cases, at least one data action includes an action for identifying information related to a recommended treatment (e.g., "Dwayne Holder is in the 88th percentile for PDL1 expression, is he a candidate for immunotherapy?").In at least some cases, the at least one data operation includes an operation for querying information related to the patient (e.g., Dwayne Holder). In at least some cases, the at least one data operation includes an operation for querying information related to patients having one or more clinical characteristics similar to the patient (e.g., "What are the most common adverse events for patients similar to Dwayne Holder?").

[0042] In at least some cases, the at least one data operation includes operations for querying information related to a patient cohort (e.g., "What are the most common adverse events for pancreatic cancer patients?"). In at least some cases, the at least one data operation includes operations for querying information related to clinical trials (e.g., "Which clinical trial is best for Dwayne?").

[0043] In at least some cases, the at least one data operation includes an operation for querying for a feature associated with a genomic mutation. In at least some cases, the feature is loss of heterozygosity. In at least some cases, the feature reflects a cause of the mutation. In at least some cases, the cause is germline. In at least some cases, the cause is somatic. In at least some cases, the feature includes whether the mutation is a tumor driver. In at least some cases, the first set of data comprises a patient name.

[0044] In at least some cases, the first set of data comprises patient age. In at least some cases, the first set of data comprises a next generation sequence panel. In at least some cases, the first set of data comprises genomic variants. In at least some cases, the first set of data comprises somatic genomic variants. In at least some cases, the first set of data comprises germline genomic variants. In at least some cases, the first set of data comprises clinically actionable genomic variants. In at least some cases, the first set of data comprises loss-of-function variants. In at least some cases, the first set of data comprises gain-of-function variants.

[0045] At least in some cases, the first set of data comprises an immunological marker. At least in some cases, the first set of data comprises a mutational burden of the tumor. At least in some cases, the first set of data comprises a microsatellite instability status. At least in some cases, the first set of data comprises a diagnosis. At least in some cases, the first set of data comprises a treatment. At least in some cases, the first set of data comprises a treatment approved by the U.S. Food and Drug Administration. At least in some cases, the first set of data comprises a pharmaceutical therapy. At least in some cases, the first set of data comprises a radiation therapy. At least in some cases, the first set of data comprises a chemotherapy. At least in some cases, the first set of data comprises a cancer vaccine therapy. At least in some cases, the first set of data comprises an oncolytic virus therapy.

[0046] In at least some cases, the first set of data comprises an immunotherapy. In at least some cases, the first set of data comprises a pembrolizumab therapy. In at least some cases, the first set of data comprises a CAR-T therapy. In at least some cases, the first set of data comprises a proton beam therapy. In at least some cases, the first set of data comprises an ultrasound therapy. In at least some cases, the first set of data comprises a surgery. In at least some cases, the first set of data comprises a hormone therapy. In at least some cases, the first set of data comprises an off-label use. In at least some cases, the first set of data comprises a gene editing therapy. In at least some cases, the gene editing therapy can be a clustered regularly interspaced short palindromic repeats (CRISPR) therapy.

[0047] At least in some cases, the first set of data comprises an intra-indication use. At least in some cases, the first set of data comprises a bone marrow transplant event. At least in some cases, the first set of data comprises a cryoablation event. At least in some cases, the first set of data comprises radiofrequency ablation. At least in some cases, the first set of data comprises a monoclonal antibody therapy. At least in some cases, the first set of data comprises an angiogenesis inhibitor. At least in some cases, the first set of data comprises a PARP inhibitor.

[0048] In at least some cases, the first set of data comprises a targeted therapy. In at least some cases, the first set of data comprises an indication for use. In at least some cases, the first set of data comprises a clinical trial. In at least some cases, the first set of data comprises a distance to a location to conduct a clinical trial. In at least some cases, the first set of data comprises a variant of unknown significance. In at least some cases, the first set of data comprises a mutation effect.

[0049] In at least some cases, the first set of data comprises a variant allele fraction. In at least some cases, the first set of data comprises a low coverage region. In at least some cases, the first set of data comprises a medical history. In at least some cases, the first set of data comprises a biopsy result. In at least some cases, the first set of data comprises an imaging result. In at least some cases, the first set of data comprises an MRI result.

[0050] At least in some cases, the first set of data comprises CT results. At least in some cases, the first set of data comprises a treatment prescription. At least in some cases, the first set of data comprises a treatment administration. At least in some cases, the first set of data comprises a cancer subtype diagnosis. At least in some cases, the first set of data comprises a cancer subtype diagnosis by RNA class. At least in some cases, the first set of data comprises a result of a treatment applied to an organoid grown from the patient's cells. At least in some cases, the first set of data comprises a tumor quality measurement. At least in some cases, the first set of data comprises a tumor quality measurement selected from at least one of the set of PD-L1, MMR, tumor infiltrating lymphocyte count, and tumor ploidy. At least in some cases, the first set of data comprises a tumor quality measurement obtained from image analysis of a pathology slide of the patient's tumor. At least in some cases, the first set of data comprises a signaling pathway associated with the patient's tumor.

[0051] At least in some cases, the signaling pathway is the HER pathway. At least in some cases, the signaling pathway is the MAPK pathway. At least in some cases, the signaling pathway is the MDM2-TP53 pathway. At least in some cases, the signaling pathway is the PI3K pathway. At least in some cases, the signaling pathway is the mTOR pathway.

[0052] In at least some cases, the at least one data operation includes an operation for querying treatment options, the first set of data comprising a genomic variant, and the associating step comprises coordinating an operation for querying treatment options based on the genomic variant. In at least some cases, the at least one data operation includes an operation for querying medical history data, the first set of data comprising a treatment, and the associating step comprises coordinating an operation for querying medical history data elements based on the treatment. In at least some cases, the medical history data is a medication prescription, the treatment is pembrolizumab, and the associating step comprises coordinating an operation for querying a prescription for pembrolizumab.

[0053] In at least some cases, the second set of data comprises clinical health information. In at least some cases, the second set of data comprises genomic variant information. In at least some cases, the second set of data comprises DNA sequencing information. In at least some cases, the second set of data comprises RNA information. In at least some cases, the second set of data comprises DNA sequencing information from short read sequencing. In at least some cases, the second set of data comprises DNA sequencing information from long read sequencing. In at least some cases, the second set of data comprises RNA transcriptome information. In at least some cases, the second set of data comprises RNA full transcriptome information. In at least some cases, the second set of data is stored in a single data repository. In at least some cases, the second set of data is stored in multiple data repositories.

[0054] In at least some cases, the second set of data comprises clinical health information and genomic variant information. In at least some cases, the second set of data comprises immunological marker information. In at least some cases, the second set of data comprises microsatellite instability immunological marker information. In at least some cases, the second set of data comprises tumor mutation burden immunological marker information. In at least some cases, the second set of data comprises clinical health information comprising one or more of demographic information, diagnostic information, evaluation results, test results, prescribed or administered treatments, and outcome information.

[0055] In at least some cases, the second set of data comprises demographic information comprising one or more of patient age, patient date of birth, sex, race, ethnicity, system of care, comorbidities, and smoking history. In at least some cases, the second set of data comprises diagnosis information comprising one or more of tissue of origin, date of initial diagnosis, histology, histology grade, metastatic diagnosis, date of metastatic diagnosis, one or more sites of metastasis, and stage information. In at least some cases, the second set of data comprises stage information comprising one or more of TNM, ISS, DSS, FAB, RAI, and Binet. In at least some cases, the second set of data comprises assessment information comprising one or more of performance status (including ECOG or Karnofsky status), performance status score, and performance status date.

[0056] In at least some cases, the second set of data comprises laboratory information comprising one or more of lab type (e.g., CBS, CMP, PSA, CEA), lab result, lab unit, date of lab service, date of molecular pathology test, assay type, assay result (e.g., positive, negative, equivocal, mutation, wild type), molecular pathology method (e.g., IHC, FISH, NGS), and molecular pathology provider. In at least some cases, the second set of data comprises treatment information comprising one or more of drug name, drug start date, drug end date, drug dosage, drug units, number of drug cycles, type of surgical procedure, date of surgical procedure, radiation site, radiation modality, radiation start date, radiation end date, total dose of radiation administered, and total percentage of radiation administered.

[0057] In at least some cases, the second set of data comprises outcome information comprising one or more of: response to treatment (e.g., CR, PR, SD, PD), RECIST score, date of outcome, date of observation, date of progression, date of recurrence, adverse event to treatment, date of presentation of adverse event, grade of adverse event, date of death, date of last follow-up, and condition at last follow-up. In at least some cases, the second set of data comprises information that has been de-identified in accordance with de-identification methods permitted by HIPAA.

[0058] In at least some cases, the second set of data comprises information that has been de-identified in accordance with a Safe Harbor de-identification method permitted by HIPAA.In at least some cases, the second set of data comprises information that has been de-identified in accordance with a statistical de-identification method permitted by HIPAA.In at least some cases, the second set of data comprises clinical health information of patients diagnosed with a cancer condition.

[0059] In at least some cases, the second set of data comprises clinical health information of patients diagnosed with a cardiovascular disease. In at least some cases, the second set of data comprises clinical health information of patients diagnosed with a diabetic condition. In at least some cases, the second set of data comprises clinical health information of patients diagnosed with an autoimmune condition. In at least some cases, the second set of data comprises clinical health information of patients diagnosed with a lupus condition.

[0060] In at least some cases, the second set of data comprises clinical health information of patients diagnosed with a psoriasis condition. In at least some cases, the second set of data comprises clinical health information of patients diagnosed with a depression condition. In at least some cases, the second set of data comprises clinical health information of patients diagnosed with a rare disease.

[0061] In at least some embodiments, the present disclosure provides a method for audibly broadcasting a response to a user based on a user query for a molecular report of a particular patient. The method can be used in a collaboration device including a processor, a microphone, and a speaker linked to the processor. The method can include storing a plurality of molecular reports of patients in a system database, receiving an audible query from a user via the microphone, identifying at least one intent associated with the audible query, identifying at least one data operation associated with the at least one intent, accessing the molecular report of the particular patient, performing at least one of the identified at least one data operation on the set of first response data included in the molecular report of the particular patient to generate a set of first response data, using the set of first response data to generate an audible response file, and broadcasting the audible response file via the speaker.

[0062] In at least some cases, the method further includes identifying eligible parameters in the audible query, and identifying the at least one data action includes identifying the at least one data action based on both the identified intent and the eligible parameters.

[0063] In at least some cases, at least one of the eligibility parameters may include patient identity.

[0064] In at least some cases, at least one of the qualification parameters may include a medical condition of the patient.

[0065] In at least some cases, at least one of the qualifying parameters can include a genetic variation.

[0066] In at least some cases, at least one of the eligibility parameters may include a type of treatment.

[0067] In at least some cases, the method further includes identifying qualifying parameters in the molecular report of the particular patient, and identifying the at least one data action includes identifying the at least one data action based on both the identified intent and the qualifying parameters.

[0068] In at least some cases, the method may further include storing a general knowledge database containing non-patient-specific data on the particular topic, and wherein identifying at least one data action associated with the at least one intent includes identifying at least first and second data actions associated with the at least one intent, wherein the first data action is associated with a molecular report of a particular patient and the second data action is associated with the general knowledge database.

[0069] In at least some cases, a second data operation associated with a general knowledge database can be first performed to generate a second data operation result, the second data operation result can be used to define the first data operation, and a first data operation associated with a molecular report of a particular patient can be second performed to generate a first set of response data.

[0070] In at least some cases, a first data operation associated with a molecular report of a particular patient can be first performed to generate a first data operation result, the first data operation result can be used to define a second data operation, and a second data operation associated with a general knowledge database can be second performed to generate a first set of response data.

[0071] In at least some cases, identifying the at least one intent may include determining that the audible query is associated with a particular patient, accessing a molecular report for the particular patient, determining a cancer status for the particular patient from the molecular report, and then selecting an intent from a pool of intents related to the cancer status.

[0072] In at least some cases, the method may further include storing a general knowledge database containing non-patient-specific data on a particular topic, and the method further includes selecting an intent associated with the general knowledge database upon determining that the audible query is not associated with a particular patient.

[0073] In at least some cases, the collaboration device may include a portable wireless device that includes a wireless transceiver.

[0074] In at least some cases, the collaboration device may be a handheld device.

[0075] In at least some cases, the collaboration device may include at least one visual indicator, and the processor is linked to the visual indicator and controllable to change at least some aspects of the appearance of the visual indicator to indicate different states of the collaboration device.

[0076] In at least some cases, the processor can be programmed to monitor the microphone input to identify a "wake up" phrase, and the processor monitors for an audible query after the wake up phrase is detected.

[0077] In at least some cases, the series of audible queries may be received via a microphone, and at least one of the identified data operations may include identifying a subset of data that may be used in a subsequent query to identify an intent associated with the subsequent audio query.

[0078] In at least some cases, the method may further include identifying at least one activity that the collaboration device user may want to perform based on at least one audible query received via the microphone and associated data in the system database, and initiating the at least one activity.

[0079] In at least some cases, initiating the at least one activity may include generating a second audible response file, broadcasting the second audible response file to a user seeking verification that the at least one activity should be performed, monitoring a microphone for a positive response, and initiating the at least one activity upon receiving the positive response.

[0080] In at least some cases, the at least one activity may include periodically capturing health information from electronic health records contained in a system database.

[0081] In at least some cases, at least one activity may include checking the status of an existing clinical or test order.

[0082] In at least some cases, at least one activity may include ordering a new clinical or laboratory order.

[0083] In at least some cases, the collaboration device may be either a smartphone, a tablet computer, a laptop computer, a desktop computer, or an Amazon Echo.

[0084] In at least some cases, initiating the at least one activity may include automatically initiating the at least one activity without initiating input from a user.

[0085] At least in some cases, the method may further include storing and maintaining a general cancer knowledge database; persistently updating the molecular report of the particular patient; automatically identifying at least one intent and associated data operation related to the general cancer knowledge database based on the molecular report data of the particular patient; persistently performing the associated data operation on the general cancer knowledge database to generate a new set of response data not previously generated; upon generating the new set of response data, using the new set of response data to generate another audible response file; and broadcasting the other audible response file via a speaker.

[0086] In at least some cases, the method may be used in conjunction with an electronic health record system that maintains health records associated with multiple patients, including the particular patient, and the method further includes identifying at least another data action associated with at least one intent, and performing the other data action on the particular patient's health record to generate additional response data.

[0087] In at least some cases, using the first set of response data to generate the audible response file may include using the response data and the additional response data to generate the audible response file.

[0088] In at least some embodiments, the present disclosure provides a method for use in a collaboration device that includes a processor, a microphone, and a speaker linked to the processor to audibly broadcast a response to a user based on a user query about a molecular report of a particular patient. The method includes storing an individual molecular report for each of a plurality of patients in a system database; storing a general cancer knowledge database including non-patient-specific data related to the topic of cancer; receiving an audible query from a user via a microphone; identifying at least one intent associated with the audible query; identifying at least a first data operation associated with the at least one intent and the molecular report of the particular patient; identifying at least a second data operation associated with the at least one intent and the general cancer knowledge database; accessing the molecular report of the particular patient and the general cancer knowledge database; performing at least a first data operation on the first data set included in the molecular report of the particular patient to generate a first response data set; performing at least a second data operation of the general cancer knowledge database to generate a second set of response data; using at least one of the first and second sets of response data to generate an audible response file; and broadcasting the audible response file via a speaker.

[0089] In at least some embodiments, the present disclosure provides a method for use in a collaboration device including a processor, a microphone, and a speaker linked to the processor for audibly broadcasting a response to a user based on a user query for a molecular report of a particular patient, the method including storing a plurality of patients' molecular reports in a system database, receiving an audible query from a user via the microphone, determining that the audible query is associated with a particular patient, accessing the molecular report of the particular patient, determining a cancer status of the particular patient from the molecular report, identifying at least one intent from a pool of intents related to the cancer status of the particular patient and the audible query, identifying at least one data operation associated with the at least one intent, performing at least one of the identified at least one data operation on a first set of data included in the molecular report of the particular patient to generate a first set of response data, using the first set of response data to generate an audible response file, and broadcasting the audible response file via the speaker.

[0090] In at least some embodiments, the present disclosure provides a method for use in a collaboration device including a processor, a microphone, and a speaker linked to the processor to broadcast a response audibly to a user based on a user query about a patient, the method including storing health records of a plurality of patients in a system database and storing a general cancer knowledge database, receiving an audible query from a user via the microphone, identifying a particular patient associated with the audible query, accessing the health record of the particular patient, identifying cancer-related data in the health record of the particular patient, identifying at least one intent associated with the identified cancer-related data, identifying at least one data action associated with the at least one intent, performing at least one data action against the general cancer knowledge database to generate a first set of response data, using the first set of response data to generate an audible response file, and broadcasting the audible response file via the speaker. [Brief description of the drawings]

[0091] [Figure 1] 1 is a schematic diagram illustrating a collaboration system consistent with at least some aspects of the present disclosure including a portable wireless collaboration device. [Diagram 2] 2 is a schematic diagram illustrating components of the exemplary collaboration device shown in FIG. 1. [Diagram 3] FIG. 13 is a schematic diagram of a second exemplary collaboration device. [Figure 4] 4 is a schematic diagram illustrating components of the second exemplary collaboration device shown in FIG. 3. [Diagram 5] 1 is a flowchart illustrating a collaboration process consistent with at least some aspects of the present disclosure. [Figure 6]1 is a schematic diagram illustrating a collaboration device user interacting with the system of claim 1. [Figure 7] FIG. 2 is a schematic diagram illustrating a workstation that can be used to access stored collaboration session data. [Figure 8] FIG. 8 is a screenshot similar to FIG. 7, but showing another screenshot. [Figure 9] FIG. 1 is a schematic diagram showing a portable audio collaboration device being used in combination with a workstation including a display. [Figure 10] FIG. 9 is a schematic diagram showing another screenshot similar to the diagram of FIG. 8. [Figure 11] FIG. 11 is a schematic diagram illustrating a second collaboration system consistent with at least some aspects of the present disclosure, in which a portable collaboration device is running an AI application to generate seed data for data operations and converts data responses into audio response files that are broadcast via the collaboration device. [Figure 12] FIG. 13 is a schematic diagram illustrating a third collaboration system consistent with at least some aspects of the present disclosure. [Figure 13] 1 is a schematic diagram illustrating several collaborating devices that can communicate with each other and / or with at least one of a first transceiver and a second transceiver using mesh networking. [Figure 14] FIG. 1 illustrates two additional collaboration device configurations, including a cube-shaped configuration and a tablet-type configuration. [Figure 15] FIG. 1 is a schematic diagram showing a workstation including various types of input / output collaboration devices. [Figure 16] FIG. 1 illustrates a headset that may operate as yet another type of input / output audio interface consistent with at least some aspects of the present disclosure. [Figure 17A]FIG. 1 is a schematic diagram showing page 1 of the Pancreas Clinical Report, which can be printed in hard copy or accessed electronically via a workstation, pad, smartphone device, etc. [Figure 17B] FIG. 1 is a schematic diagram showing page 2 of the Pancreas Clinical Report, which can be printed in hard copy or accessed electronically via a workstation, pad, smartphone device, etc. [Figure 17C] FIG. 1 is a schematic diagram showing page 3 of the Pancreas Clinical Report, which can be printed in hard copy or accessed electronically via a workstation, pad, smartphone device, etc. [Figure 18] 6 is a flow chart similar to the chart shown in FIG. 5, but in which condition-specific clinical records and associated intents are used to drive the query process. [Figure 19] FIG. 1 illustrates an audio response process, consistent with at least some aspects of the present disclosure. [Figure 20] FIG. 2 illustrates a system database, consistent with at least some aspects of the present disclosure. [Figure 21] 1 illustrates a screenshot used by a system administrator to specify system intents, intent parameters, and answer formats for a provider panel type, consistent with at least some aspects of the present disclosure. [Figure 22] FIG. 22 is similar to FIG. 21 but includes a screenshot for specifying gene-specific system information. [Diagram 23] FIG. 23 is similar to FIG. 22 but includes a screenshot for specifying provider methods. [Figure 24] FIG. 11 is a schematic diagram of an example fourth exemplary system including a mobile device. [Diagram 25] FIG. 1 shows a screenshot of a mobile application. [Figure 26] FIG. 26 shows a second screenshot of the mobile application in FIG. 25. [Figure 27] FIG. 26 shows a third screenshot of the mobile application in FIG. 25. [Figure 28] FIG. 13 is a schematic diagram of a fifth exemplary collaboration system. [Figure 29] 1 is a flow chart of a process for generating supplemental content for a physician based on molecular reports associated with a particular patient. [Diagram 30] 1 is a flowchart of a process for generating non-patient-specific supplemental content for a physician. [Diagram 31] 1 is a flow chart of a process that may be used for onboarding oncologists. [Diagram 32] 1 is a screenshot for use by a system administrator to visually specify system intents, intent parameters, and answer formats for a provider panel type, consistent with at least some aspects of the present disclosure. [Diagram 33] FIG. 1 is a schematic diagram of an intent extraction architecture. [Diagram 34] FIG. 1 is a schematic diagram of a question and answer workflow. [Diagram 35] FIG. 2 is a schematic diagram of an exemplary conversation workflow. [Diagram 36] 1 is a flowchart of a process for providing an audible response to an oncologist using at least one microservice and / or engine, consistent with at least some aspects of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0092] Various aspects of the disclosure will now be described with reference to the drawings, in which like reference numerals correspond to like elements throughout the several views. It should be understood, however, that the following drawings and the related detailed description are not intended to limit the claimed subject matter to the particular forms disclosed. Rather, the intent is to cover all modifications, equivalents, and alternatives falling within the spirit and scope of the claimed subject matter.

[0093] In the following detailed description, reference is made to the accompanying drawings, which form a part of this specification and are shown by way of example, with reference to specific embodiments in which the present disclosure may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the present disclosure. However, it should be understood that the detailed description and specific examples, while showing examples of embodiments of the present disclosure, are given by way of illustration only and not by way of limitation. From this disclosure, it will be apparent to those skilled in the art that various substitutions, modifications, additional rearrangements, or combinations thereof may be made within the scope of the present disclosure.

[0094] According to common practice, various features shown in the drawings may not be drawn to scale. The figures presented herein are not meant to be actual drawings of any particular method, device, or system, but merely idealized representations adopted to describe various embodiments of the present disclosure. Thus, dimensions of various features may be arbitrarily enlarged or reduced for clarity. Furthermore, some of the drawings may be simplified for clarity. Thus, the drawings may not depict all components of a given apparatus (e.g., device) or method. Moreover, similar reference numbers may be used to denote similar features throughout this specification and the drawings.

[0095] The information and signals described herein may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be referenced throughout the above description may be represented by voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof. Some figures may show signals as single signals for clarity of presentation and explanation. It will be understood by those skilled in the art that a signal may represent a bus of signals, and that the bus may have various bit widths, and that the present disclosure may be implemented with any number of data signals, including a single data signal.

[0096] The various exemplary logic blocks, modules, circuits, and algorithmic acts described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and acts have been described generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the disclosed embodiments described herein.

[0097] Furthermore, it should be noted that the embodiments may be described in terms of a process that is depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operational acts as a sequential process, many of these acts may be performed in a different order, in parallel, or substantially simultaneously. Additionally, the order of the acts may be rearranged. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, or the like. Additionally, the methods disclosed herein may be implemented in hardware, software, or both. If implemented in software, the functions may be stored on or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media includes both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another.

[0098] Reference to elements herein using designations such as "first," "second," etc., should be understood not to limit the quantity or order of those elements unless such limitation is expressly stated. Rather, these designations may be used herein as a convenient method of distinguishing between two or more elements or instances of an element. Thus, reference to a first and a second element does not imply that only two elements may be used therein or that the first element must precede the second element in any way. Also, unless otherwise stated, a set of elements may comprise one or more elements.

[0099] As used herein, terms like "component," "system," and the like are intended to refer to a computer-related entity that is either hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a processor, an object, an executable, a thread of execution, a program, and / or a process running on a computer. By way of illustration, both an application running on a computer and the computer can be a component. One or more components may reside within a process and / or thread of execution, and a component may be localized on one computer and / or distributed between two or more computers or processors.

[0100] The word "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs.

[0101] Furthermore, the disclosed subject matter may be implemented as a system, method, apparatus, or article of manufacture using standard programming and / or engineering techniques to generate software, firmware, hardware, or any combination thereof for controlling a computer or processor-based device to implement the aspects detailed herein. The term "article of manufacture" (or alternatively "computer program product") as used herein is intended to encompass a computer program accessible from any computer-readable device, carrier, or medium. For example, computer-readable media may include, but are not limited to, magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips...), optical disks (e.g., compact disks (CDs), digital versatile disks (DVDs)...), smart cards, and flash memory devices (e.g., cards, sticks). Furthermore, it should be understood that a carrier wave may be used to carry computer-readable electronic data such as those used in sending and receiving e-mail or accessing a network such as the Internet or a local area network (LAN). Of course, those skilled in the art will recognize that many modifications may be made to this configuration without departing from the scope or spirit of the claimed subject matter.

[0102] The term "genetic analyzer" is used herein to mean a device, system, and / or method for determining the characteristics (including sequence) of nucleic acid molecules (including DNA, RNA, etc.) present in a biological specimen (including a tumor, biopsy, tumor organoid, blood sample, saliva sample, or other tissue or bodily fluid).

[0103] The term "genetic profile" is used herein to mean the combination of one or more variants, RNA transcriptome, or other informative genetic features determined for a patient from next generation sequencing, which may also be commonly referred to as "massively parallel sequencing."

[0104] The term "gene sequence" is used herein to mean a record of the series of nucleotides present in a patient's RNA or DNA, as determined from sequencing the patient's tissue or bodily fluids.

[0105] The term "mutant" is used herein to mean a difference in a gene sequence or gene profile as compared to a reference gene sequence or expected gene profile.

[0106] The term "expression level" is used herein to mean the number of copies of an RNA or protein molecule produced by a gene or other genetic locus, which may be defined by chromosomal location or other genetic mapping indicator.

[0107] The term "gene product" is used herein to mean a molecule (including a protein or an RNA molecule) produced by the manipulation (including transcription) of a gene or other genetic locus, which may be defined by a chromosomal location or other genetic mapping indicator.

[0108] Referring now to the drawings, in which like reference numerals correspond to like elements throughout the several views, and more particularly to FIG. 1 , the present disclosure will be described in the context of an exemplary collaboration system 10 consistent with at least some aspects of the present disclosure. The system 10 includes a collaboration server 12, an artificial intelligence (AI) server 14, a user interface collaboration device 20, and a service provider database 18. Referring again to FIG. 1 , in the illustrated embodiment, the AI ​​server 14 is shown as separate from the collaboration server 12. Nevertheless, it should be understood that in at least some embodiments, the functions of the two servers may be performed via a single server. Similarly, while the exemplary system 10 is described herein as having certain process steps or functions performed by the server 12 and others performed by the server 14, in other cases, the division of functions and steps between the two servers 12 and 14 may differ. Additionally, in at least some embodiments, some of the processes performed by the servers 12 and 14 may be performed by a processor located within the collaboration device 20. For example, at least in some cases, some or most of the processes related to speech recognition, intent matching, parameter extraction, and audio response generation performed by AI server 14 may be performed by collaboration device 20. By executing at least some of the processes performed by servers 12 and 14 on collaboration device 20, the latency of outputting generated responses can be reduced by up to two seconds, as described in further detail below.

[0109] The collaboration server 12 is linked to a wireless transceiver (e.g., a transmitter and receiver) 16 to enable wireless two-way communication between the collaboration devices 20 and the collaboration server 12. The transceiver 16 may be any type of wireless transceiver including, for example, a cellular telephone transceiver, a Wi-Fi transceiver, a Bluetooth transceiver, a combination of different types of transceivers (e.g., including Bluetooth and cellular), etc. The server 12 executes software applications or modules to perform the various processes and functions described throughout this specification. In particular, the server 12 executes a collaboration application 60 that includes, among other things, a visual response module 62 and a data behavior module 64. Server 12 receives user voice query (hereinafter "voice message") 59 captured by device 20, collaborates with AI server 14 to identify the meaning of the voice message (e.g., intent and key parameters), performs data operations on data in database 18 that match the voice message to generate a data response, collaborates with AI server 14 to generate an audio response file based on the data response and, at least in some cases, a visual response file, and transmits response files 73, 77 back to collaboration device 20. Device 20 then broadcasts (66) the audio response to the user and, if there is a visual response suitable for presentation via device 20, generates a visual response in some manner (e.g., presents content on display 48 of device 20, illuminates signal light 50, etc.). Display 48 and / or signal light 50 may be considered visual indicators.

[0110] 2, collaboration device 20 includes an external housing 22, a device processor 30, a battery 32 or other power source, a device memory 34, a wireless transceiver 36, one or more microphones 38, and one or more speakers 44 or audio output devices, as well as several components or processes that can be used to activate device 20 to initiate user collaboration activities. External housing 22 includes an exterior surface that forms a sphere in the illustrated example, where the diameter of the sphere is selected so that device 20 can be easily held in the hand by an oncologist. For example, the diameter of device 20 will most often be between three-quarters and five inches, and in a particularly advantageous embodiment, the diameter is between one and a quarter inch and two inches.

[0111] In other cases, the outer housing includes an exterior surface that forms a cube or other three-dimensional rectangular prism. In such cases, in particularly advantageous embodiments, the maximum dimensions of the three-dimensional shape (height, width, depth) are between 1 and a quarter inch and 2 inches.

[0112] System 10 may be implemented in other ways. For example, collaboration device 20 may be a smartphone, tablet, laptop, desktop, or other computing device, such as an Apple iPhone®, a smartphone running the Android® operating system, or an Amazon Echo. Some of the processes performed by servers 12 and 14 may be performed through the use of an app or another program running on a processor located within collaboration device 20.

[0113] The exterior surface may be formed by several different components from several different materials, including opaque materials for some of the surfaces and transparent or translucent materials for other parts that need to pass light from indicator lights mounted within the housing. The exterior surface of the housing may form speaker and microphone apertures, a charging port opening (not shown), and other apertures or openings for different purposes. The housing forms an internal cavity in which most of the other device components are mounted. The device 20 may include a single speaker and a single microphone, but in an optimized assembly, the device 20 includes multiple speakers and microphones arranged around the housing assembly so that the oncologist's audio signals can be captured from all directions.

[0114] There are many different hardware configurations that may be used to provide the collaboration device processor 30. One particularly useful processor for the purposes of the present device 30 is the Qualcomm QCS405 SoC (system on chip), which supports many different types of connectivity, including Bluetooth, ZigBee, 802.11ac, 802.11 ax-ready, USBC2.0 / 3.0, etc. This solution includes an on-device AI engine that allows for the execution of AI algorithms on the device, so that, at least in some cases, the AI ​​functions described herein with respect to the server 14 may be performed by the processor 30. This SoC supports up to four microphones and supports high performance keyword detection. The processor 30 is linked to each of the battery 32, memory 34, transceiver 36, microphone 38, and speaker 44. In some embodiments, the battery 32 can be charged using a charging dock (not shown).

[0115] While device 20 is activated and remains active, microphone 38 captures user voice messages 57 which are provided to processor 30. Processor 30 transmits the voice messages (59) via transceiver 36 to collaboration server 12 (see again FIG. 1 ). Audio response files are received (81) by transceiver 36 of device 20, and processor 30 broadcasts them via speaker 44. Although not shown, it is contemplated that device 20 may also include some type of haptic signal component (e.g., a vibrator, etc.) to indicate one or more device states.

[0116] The device 20, and more specifically, the memory 34, may include acoustic processes, light control processes, security processes, connectivity processes, and other suitable purposes. These processes may be stored as firmware in a portion of the memory 34 that is non-volatile and, in some embodiments, read-only. The firmware may include an acoustic process that is capable of recognizing a library of wake words or phrases spoken by the oncologist. For example, the library may include the phrases "Tempus ONE" or "Hey, One." By storing key phrases that are repeatedly used by the oncologist directly in the memory 34, the latency in processing commands spoken by the oncologist may be reduced. The acoustic processes may also include a silence detection process, a fallback audio response playback process that audibly notifies the oncologist of errors or timeouts that occur during data transmission (such as TCP packets, or HTTP messages), a speaker protection algorithm, a digital signal processing (DSP) algorithm, and / or other suitable processes related to acoustics.

[0117] The firmware may include a conversation flow process for determining whether a follow-up question requires the use of a wake word phrase such as "Tempus ONE." For example, an initial question, "What were Dwayne Holder's results?" may be followed by "And how old was he?" without the need to use the "Tempus ONE" wake word phrase or to specify the patient's name again. The question, "And how old was he?" may not be relevant unless the person in question has already been identified, in which case a follow-up question may be asked. The firmware may include a battery status algorithm for determining the charge level and / or state of charge of the battery 32. The firmware may include connection and security processes for storing and / or maintaining secure element cryptographic keys, storing device identifiers, storing valid networks (e.g., Wi-Fi networks), and other suitable processes. In some embodiments, the firmware may be updated over the air.

[0118] The firmware may also include a lighting control process for controlling the indicator light 50. The lighting control process may change the color and / or brightness of the indicator light 50 and pulse the indicator light 50 on and off.

[0119] Firmware processes can be used to control indicator lights 50 and / or speaker 44 based on the state of collaboration device 20. Some states are initiated by the oncologist. The oncologist can actuate one or more of input buttons 52, pronounce commands, and / or move collaboration device 20 (e.g., by placing collaboration device 20 in a charging dock). Exemplary lighting and speaker controls based on collaboration device 20 states are included in Table 1 below. Some oncologist interactions include "app" functionality, which is described in more detail below.

[0120] [Table 1A]

[0121] [Table 1B]

[0122] In at least some cases, device 20 may be activated by a specially voiced voice command. To this end, processor 30 may be always on and monitoring for a special trigger activation command, such as "Hey, query." Once an activation command is received, processor 30 may be activated to participate in a user collaboration session. Here, processor 30 may confirm the activation command by transmitting a response, such as "Hello, how can I help you?" or a tone or other audio indication, and then enter a "listening" state to capture a subsequent user voice message. Once the subsequent voice message is captured, the collaboration session may proceed as described above.

[0123] In addition to or instead of being activated by a spoken activation command, device 20 may be activated by selection of a device activation button or touch sensor, such as when device 20 is picked up or otherwise moved. To this end, see optional input button 52 and motion and orientation sensors 40 and 42 in Figure 2. The motion sensor may include an accelerometer, a gyroscope, both an accelerometer and a gyroscope, or some other type of motion sensor device.

[0124] In addition to being able to provide audio responses to the user's queries, at least in some cases, the device 20 is equipped to provide some sort of visual response. For example, in the simple case, the device 20 may include one or more indicator lights 50, which may be LEDs or other light sources that can be activated or controlled to change color to complicate different device 20 states. For example, at least in some cases, when the device 20 is inactive and waiting to be activated, the indicator light 50 may be off or dim green. Now, while the device 20 is activated and waiting or listening to an audio message, the light 50 may be activated bright green to indicate "go". When the user is speaking and an audio message is being captured by the device 20, the light 50 may be activated teal to indicate an audio message capture state. When the query audio signal ends, the light 50 may be illuminated yellow to indicate a "thinking" or query processing state. When the audio response is being broadcast to the user, light 50 may illuminate orange to indicate an output status, and once the audio response is completed, light 50 may illuminate bright green again to indicate that the device is again waiting or listening for the next voice message to be spoken by the user.

[0125] In at least some cases, whenever device 20 is activated and waiting for a new or next voice message, device 20 may be programmed to wait in the active state for a threshold period (e.g., 30 seconds) and then assume an inactive state waiting to be reactivated via another activation utterance or other user input. In other cases, once device 20 is activated, device 20 may remain activated for a longer period (e.g., 10 minutes) and enter into the deactivated listening state before the longer period expires only if the user utters a deactivation phrase (e.g., "end session," "end query," or "Hey query" followed by "end session," etc.) or otherwise actively deactivates device 20 (e.g., by selecting deactivation input button 52).

[0126] Still referring to FIG. 2, in some cases, device 20 may include one or more flat or curved or other contoured display screens 48 for presenting visual responses to user queries when the visual responses are suitable for consumption via a relatively small display screen. Here, for example, a short answer to a user query may be presented as text via display 48. As another example, a summary phrase associated with a data response that includes data that cannot be easily presented via a small display screen may be generated and presented via display 48. Other text phrases or graphics are contemplated for other purposes. For example, if the visual response is presented via some other display device (e.g., a display device paired or otherwise associated with collaboration device 20), a text message may be presented via display 48 indicating that additional information or visual response is being presented via the associated display. As another example, display 48 may be controlled to light a particular color to indicate a state as described above with respect to light device 50, or may present only the answer to the query in text format. Referring again to FIG. 1, AI server 14 executes software application programs and modules that perform various functions consistent with at least some aspects of the present disclosure. In at least some cases, the AI ​​server 14 includes an automatic speech recognition (ASR) module 70, an intent matching module 72, a parameter extraction module 74, and an audio response module 76.

[0127] The ASR module 70 receives user voice messages from the collaboration application 60 (61) and automatically converts the voice signal, essentially in real time, into text corresponding to the user-uttered voice message. Thus, if the oncologist's voice signal message is "How many of your male patients over 45 years old have pancreatic cancer?" or "What type of treatment should I prescribe for this patient?", the ASR module 70 generates matching text using speech recognition software. Speech recognition applications are well known in the art and include Dragon software by Nuance, Google Voice by Google, Watson by IBM, and others. In some cases, the recognition application supports an industry-specific term / phrase dictionary in which specific terms and phrases used within an industry are defined and can be recognized. In some cases, a user-specific dictionary is also supported for terms or phrases routinely used by a particular oncologist. In either case, new terms and phrases can be added to the industry and user dictionaries. The text file is provided to the intent matching module 72.

[0128] The intent matching module 72 includes a natural language processor (NLP) programmed to determine the intent of the user's voice signal message. Here, for example, the intent may be to identify a data subset in the database 18. As another example, the intent associated with the phrase "How many male patients over 45 years old have pancreatic cancer?" may be to identify the number of patients. As another example, the intent associated with the phrase "What type of treatment should be prescribed for patient John Doe?" may be to identify the treatment that the system has determined will maximize the quality of life for patient John Doe. Literally thousands of other intents may be recognized by the matching module 72. Intents are described in more detail below.

[0129] 1, the parameter extraction module 74 extracts important parameters from the user-uttered voice message. For example, parameters extracted from the phrase "How many male patients over 45 years old have pancreatic cancer?" may include "pancreas," "male," and "45 years old." For each user voice message, the AI ​​server 14 returns (63) (i) the associated text file, (ii) the matching intent, and (iii) the extracted parameters to the collaboration server 12, and more specifically to the data behavior module 64.

[0130] The data actions module 64 accesses the database 18 and creates a collaboration record on the database to memorialize the collaboration session (65). The text file received from the server 14 is stored in the database 18 with date and time, oncologist identification information, etc. The data actions module 64 converts the intent and extracted parameters into data actions and then performs the actions 65 on the data in the database 18. For example, for the voice message "How many male patients over 45 years old have pancreatic cancer?", the actions module 64 constructs a database query to retrieve (e.g., intent) the number of male patients over 45 years old who have pancreatic cancer (e.g., extracted parameters). The data actions result in a data response including the number of male patients over 45 years old who have pancreatic cancer.

[0131] As another example, for the voice message "What type of medication should I prescribe for John Doe?", the operation module 64 constructs a database query to search for medications (e.g., intents) for a cohort of patients who are clinically similar to patient John Doe and where such medications have produced optimal outcomes for the cohort. Determining whether a cohort of patients is clinically similar may be accomplished by querying the database 18 for patients who have certain factors identical and / or similar to those of John Doe, such as age, stage of cancer, previous treatments, mutations, RNA expression, etc. As a simple example, if John Doe has a PTEN genomic mutation, the database 18 may select for inclusion in the cohort all patients who also have a PTEN genomic mutation. As another example, if John Doe has metastatic prostate cancer but has become unresponsive to first-line androgen suppression therapy, the database 18 may select for inclusion in the cohort all metastatic prostate cancer patients who have become unresponsive to first-line androgen suppression therapy.

[0132] As another example, for the voice message "What is Jane Smith's expected progression-free survival if she is prescribed Keytruda?", the operation module 64 constructs a database query to search for patients clinically similar to Jane Smith, selects a cohort of those patients who were prescribed Keytruda, analyzes the progression-free survival of the selected cohort of patients, and returns the mean progression-free survival from the selected cohort.

[0133] As indicated above, the physician's voice message may be relevant to a question regarding a particular individual. The operation module 64 may be further configured to access a patient data repository to identify the patient's clinical, genomic, or other health information. The patient data repository may take many forms and may include an electronic health record, a health information exchange platform, a patient data warehouse, a research database, and the like. The patient data repository may include data stored in a structured format, such as a relational database, a JSON file, or other data storage configurations known in the art. The operation module 64 may communicate with the patient data repository in a variety of ways, such as through data integration, may use a variety of technologies, and may rely on a variety of frameworks, such as Fast Healthcare Interoperability Resources (FHIR). The patient data repository may be owned, operated, and / or managed by the physician, the physician's employer, a hospital, the physician's clinic, a clinical laboratory, a contract research organization, or another entity associated with the delivery of health care. The patient data repository may include all of the patient's health information, or a subset of the patient's health information. For example, a patient data repository might include structured data including patient demographic information (e.g., age, sex), a clinical description of the patient's cancer (e.g., stage such as "stage 4" and subtype such as "pancreatic cancer"), a genomic description of the patient and / or the patient's cancer (e.g., a list of nucleotides in specific introns or exons, somatic variants such as "BRAF mutations", immunological markers such as microsatellite instability and tumor mutation burden, RNA over- or under-expression, a list of pathways affected by the variants found, etc.), an imaging description of the patient's cancer (e.g., features derived from radiology or pathology images), an organoid-derived description of the patient's cancer (e.g., a list of treatments that have been effective in reducing or destroying organoid cells derived from the patient's tumor), and a list of previous and current pharmaceutical therapies, treatments, surgeries, procedures, or other treatments.

[0134] The operational module 64 may use various methods to identify how the particular patient being queried is clinically similar to other individuals whose data are stored in the database 18. An example of determining clinical similarity is described in U.S. patent application Ser. No. 16 / 671,165, filed Oct. 31, 2019, the contents of which are incorporated by reference in their entirety for all purposes. Another example of determining clinical similarity is described in U.S. patent application Ser. No. 16 / 732,168, filed Dec. 31, 2019, the contents of which are incorporated by reference in their entirety for all purposes.

[0135] A determination of which medication provided the optimal outcome for a cohort of identified individuals may be determined by comparing outcome information stored in database 18 for those individuals with the medications prescribed or administered, dividing the cohort into sub-cohorts, analyzing outcome measures such as progression-free survival, overall survival, survival, etc., for each sub-cohort, and returning one or more measures indicative of an optimal outcome.

[0136] In another example, the data actions module 64 may select a first treatment from the list of treatments, look up information from all patients in the database 18 who were provided with the first treatment, divide the patient group into a first cohort of patients with a positive outcome and a second cohort of patients without a positive outcome, compare health characteristics (e.g., clinical, genomic, and / or imaging) of the queried patient to the health characteristics of the first cohort, compare the health characteristics of the queried patient to the health characteristics of the second cohort, and determine whether the characteristics of the queried patient are closer to the characteristics of the first cohort or the second cohort. If the characteristics of the queried patient are more clinically similar to the first cohort, the data actions module 64 may prepare a data response indicative of the first treatment. If the characteristics of the queried patient are more clinically similar to the second cohort, the data actions module 64 may not prepare a data response indicative of the first treatment. The data operations module 64 may then select a second, third, fourth, etc. treatment from the list of treatments and repeat the above process for each selected treatment, or continue until all treatments in the list of treatments have been explored. Various algorithmic approaches using mathematical or statistical methods known in the art may be used on the associated health characteristics to determine whether the queried patient characteristics are clinically similar to the first or second cohort, including mean, median, principal component analysis, etc.

[0137] In another example, the data behavior module 64 may select all or a subset of records from a patient in the database 18. From these records, the module 64 may then select records from a first cohort of patients with similar genomic biomarkers as the queried patient. The module 64 may then filter the first cohort of patients prescribed a first treatment from the list of treatments. The module 64 may then examine the outcomes of the patients in the first cohort, subdivide the first cohort into two or more sub-cohorts based on the outcomes, and divide patients with similar outcomes into the same sub-cohort. Each sub-cohort may be further divided into additional sub-cohorts as the first cohort, and so on until there are no significant outcome differences within each sub-cohort. At this point in the method, there may be dozens or more sub-cohorts. The data behavior module 64 may then compare the health characteristics of the queried patient to those in each sub-cohort to identify the sub-cohort that is most clinically similar to the health characteristics of the patient. The data operations module 64 may then select a second, third, fourth, etc. treatment from the list of treatments and repeat the above process for each selected treatment, continuing until all treatments in the list of treatments have been explored.

[0138] The data behavior module 64 returns the data response to the AI ​​server 14 (67), and more specifically to the audio response module 76, which uses the data to generate an audio response file. For example, if 576 male patients over the age of 45 had pancreatic cancer in the retrieved dataset, the response module 76 may generate the phrase "576 male patients over the age of 45 had pancreatic cancer." The audio response file is transmitted (71) to the collaboration application 60, which stores the response file as well as a textual representation thereof in a collaboration record in the database 18 for later access. The collaboration application 60 also transmits (73) the audio response file via the transceiver 16 to the collaboration device 20, which then broadcasts the audio file to users.

[0139] The AI ​​server 14 may be provided through many different software application programs. One particularly useful suite of software modules that may provide AI capabilities is the Qualcomm Smart Audio 400 Platform development kit that can be used with the Qualcomm SoC processors mentioned above. Another useful suite is the Dialogflow program developed and maintained by Google. Dialogflow is an end-to-end, build-once deploy- everywhere suite for creating conversational interfaces for websites and mobile applications. System administrators can use the Dialogflow interface to define a set of intents, training phrases, parameters, and responses to intents. An intent is a general intent by a user (e.g., what the user wants) to access or manipulate database data in a certain way. For example, one intent may be to generate a database data subset (e.g., patients that meet eligible query parameters). As another example, another intent may be to return a number (e.g., the number of patients that meet eligible parameters). Other intents may be welcome intents (e.g., when a user first activates device 20), adverse sequence intents (e.g., returning a list or at least an indication of adverse sequences in a treatment regimen), medication intents (e.g., returning a list or indication of previous medications), schedule event intents (e.g., scheduling an appointment, test, procedure, etc.), etc. A typical system is expected to include hundreds, and possibly thousands, of intents.

[0140] For each intent, the administrator provides a relatively small set of seeds or training phrases that are used to train the intent matching module to recognize the intent associated with the received voice message. The training phrases include phrases that a user might say if the goal or purpose associated with the utterance matches the associated intent. For example, if there is an intent to return a large number of patients who meet the qualifying parameters (e.g., age, disease, condition, cancer gene, mutation, residence, stage, treatment, medical YYY side effects, outcome, etc.), some typical training phrases might be "How many patients have pancreatic cancer?", "How many stage 3 breast cancer patients in Chicago are HER2 positive?", "How many patients showed side effects while taking XXX medication?", "How many ovarian cancer patients have p85 PIK3CA mutations in the past 48 months?", "What percentage of basal cell carcinoma patients have had cryosurgery in the past 18 months?", and "How many smokers also have lung cancer?". Dialogflow also supports follow-up intents that may be sequentially associated with other intents, more specifically, second or subsequent intents that are recognized in a series of questions after a first intent has been identified. For example, a first phrase, "How many ovarian cancer patients have had p85 PIK3CA mutations in the past 48 months?" may be followed by a second phrase, "How many of these patients have been seen in the past 12 months?". As another example, for the purpose of returning suggested treatments for a particular patient, some exemplary training phrases may be "What type of medication should we prescribe for John Doe?", "What type of immunotherapy should this patient receive?", "What is the expected progression-free survival for Jane Smith if we prescribe Keytruda?".

[0141] Once a small set of training or seed phrases is provided by the administrator, the machine learning module (e.g., an AI engine) uses those phrases to automatically train and generate many other similar phrases that may be associated with an intent. This automated training process in which many similar queries are generated and associated with a particular intent is called "fanning," and the newly generated queries are called "fan queries." The machine learning module stores the complete set of training phrases and derived phrases (hereinafter "intent phrases") for use during collaboration sessions. When a user then uses the system to utter a phrase that is similar to, but not exactly a match with, one of the intent phrases, the intent matching module recognizes the user's intent despite the inexact match and responds accordingly. Additionally, if an utterance is similar to, but not exactly the same as, one of the intent phrases, the system may automatically save the utterance as an additional intent phrase associated with the intent and train additional other intent phrases based on it so that the intent matching module becomes more intelligent over time.

[0142] In most cases, the system user's intent alone is not detailed enough to identify the specific information the user is looking for or how to respond, and the user must utter or provide additional query parameters. Dialogflow allows the administrator to specify a set of parameter types to extract from the received voice message. For example, some parameters may include date, time, age, illness, condition, medication, treatment, procedure, physical condition, mental condition, etc. For each parameter type, the administrator specifies example parameter phrases or combinations of data (hereinafter "parameter phrases") that the system user may utter to indicate the parameter, and the machine learning module uses the administrator-specified parameter phrases to train a larger set of parameter phrases that can be used to recognize instances of the parameter. During a collaboration session in which a user query is received, after module 72 identifies the intent, extraction module 74 uses the parameter phrases to extract parameter values ​​from the user's voice message, and the intent and extracted parameters together provide the raw material needed by data actions module 64 to formulate data actions to perform on data in database 18 (see again FIG. 1).

[0143] Dialogflow allows administrators to tag some parameters as needed and define feedback prompts that will be presented to users if a received voice message does not contain the required parameters. So, for example, if a particular intent requires a date and the query associated with that intent does not contain a date parameter, the system may automatically present the user with a feedback prompt requesting the date (e.g., "What date range are you interested in?").

[0144] Dialogflow also guides administrators to define intent responses. Intent responses typically include a text response that specifies one or more phrases, a data response, or a formatted combination of text and data that can be used to respond to a user's query. For example, if the intent is to return the number of patients who meet eligible parameters, the response phrase might be "The number of patients who have _", with blanks representing data fields to be filled with parameters from the voice message, data from a database, data derived from a database, or options specified in combination with the response phrase.

[0145] Hereinafter, the intent and all information associated with a particular intent specified by the system (e.g., parameters, fun queries, data actions, and answer phrases) may be referred to as the intent and supporting information to simplify this description.

[0146] At least in some cases, module 72 may identify at least one intent in the query. At least in some cases, the query may be an audible query. At least in some cases, the at least one intent may be an intent related to a clinical trial. At least in some cases, the at least one intent may be related to a pharmaceutical. At least in some cases, if the intent is related to a pharmaceutical, the intent may be referred to as a pharmaceutical intent. At least in some cases, the pharmaceutical intent may be related to a pharmaceutical, such as chemotherapy. At least in some cases, the pharmaceutical intent may be an intent related to a PARP inhibitor intent. At least in some cases, the at least one intent may be related to a gene. At least in some cases, the at least one intent may be related to immunology. At least in some cases, the at least one intent may be related to a knowledge database. At least in some cases, the at least one intent may be related to a testing methodology. At least in some cases, the at least one intent may be related to a gene panel. At least in some cases, the at least one intent may be related to a reporting. At least in some cases, the at least one intent may be related to an organoid process. At least in some cases, at least one intent may be related to imaging. At least in some cases, at least one intent may be related to pathogens. At least in some cases, at least one intent may be related to vaccines.

[0147] 1, the response module 76 uses the response phrases to generate responses, and more specifically, to generate audio response files that are sent back to the collaboration server 12. Again, it is contemplated that a typical system may include hundreds or thousands of response phrases, at least one response phrase format or structure for each intent supported by the system.

[0148] In the illustrated exemplary system 10, the AI ​​server 14 does not control the database 18 and therefore transmits the intent and extracted parameters back to the collaboration server 12, which executes the data action module 64. In this case, it is contemplated that many data responses may not be able to be presented to the user in an easily digestible audio response file. For example, in some cases, the data response may include a graphical display of comparative cancer data that cannot be easily described audibly in an auditorily comprehensible manner. In these cases, after the data action module 64 receives the data response from the database 18, the module 64 may pass the data to the visual response module 62, which generates an appropriate visual response to the user's query, which transmits the visual response to the device 20 via the transceiver 16 for presentation.

[0149] In at least some cases, a summary audio response may be formulated by system 10 and broadcast via device 20 as needed. For example, in some cases, the data response may simply comprise a list-type subset of the database data to form the basis for additional searching and data manipulation. For example, a sub-dataset may include data for all male cancer patients who had adverse reactions to taking their medication since 1998. This sub-dataset may serve as data for a subsequent query that restricts the cancer type to pancreatic, or the treatment to treatment XXX, or any other more detailed parameter combination. In these cases where the database subset is limited, an appropriate audio response file may include a summary response such as, for example, "A subset of the data for all male cancer patients who had adverse reactions to taking their medication has been identified" (see 66 in FIG. 1). This response phrase is specified via Dialogflow or other conversation definition software application.

[0150] It is believed that, at least in some cases, the system may not be able to associate the oncologist's voice query (i.e., an audible query) with an intent or a system-supported parameter with a high level of confidence. In some cases, it is believed that the AI ​​server 14 may be assigned a confidence factor to each intent and extracted parameter and may be programmed to return one or more probing queries to the oncologist if the confidence factor of the intent or parameter value falls below some threshold level. In some cases, to help direct the oncologist to the system-supported queries, the probing feedback queries may be tailored or customized to known structures or data content in the database 18 or to the intents and parameters supported by the AI ​​server 14.

[0151] If the intent and / or extracted parameters are not supported by the AI ​​server or other system process, the system 10 may generate a record of the unsupported query for review by an administrator and subsequent access by the oncologist. In such a case, the system may present the unsupported query and related information to an administrator during a system maintenance session to allow the administrator to determine whether a new intent and / or parameters should be specified in Dialogflow or via some other query flow application. If the administrator specifies new intent and / or parameters, the system may provide a data response to the query and update the collaboration record with the unsupported query to indicate that the query is now supported, and the oncologist may be notified via email, text, or other method that the query is supported during a subsequent collaboration session.

[0152] In some cases, database 18 may include an electronic health record database from a hospital or hospital system. In other cases, database 18 may include an electronic data warehouse with data extracted from EHRs, transformed, and loaded into a multidimensional data format. In other cases, database 18 may include data collected from multiple hospitals, clinics, health systems, and other providers across the United States and / or internationally. The data in database 18 may include clinical data elements reflecting the health status of multiple patients over time. Clinical data elements may include, but are not limited to, demographics, age / DOB, sex, race / ethnicity, facility, relevant comorbidities, smoking history, diagnosis, site (tissue of origin), date of initial diagnosis, histology, histology grade, metastatic diagnosis, date of metastatic diagnosis, site of metastasis, stage (e.g., TNM, ISS, DSS, FAB, RAI, Binet), evaluation, lab and molecular pathology, lab type (e.g., CBS, CMP, PSA, CEA), lab result and unit, lab date, performance status (e.g., ECOG, Karnofsky), performance status score, performance status date, molecular pathology test date, gene / biomarker / assay, gene / biomarker / assay result (e.g., positive, negative), data elements (e.g., PSA for prostate), treatment, drug name, drug start date, drug end date, drug dose and units, number of drug cycles, type of surgical procedure, date of surgical procedure, radiation site, radiation modality, radiation start date, radiation end date, total dose of radiation administered, and total percentage of radiation administered, outcome, response to treatment (e.g., CR, PR, SD, PD), RECIST, outcome / observation date, date of progression, date of recurrence, adverse event to treatment, date of presentation of adverse event, grade of adverse event, date of death, date of last follow-up, and disease status at last follow-up.The information in database 18 may have the data in a structured format, for example, through the use of a data dictionary or metadata repository, which is a repository of information about information such as meaning, relationships to other data, origin, use, and format. The information in database 18 may be in the form of original medical records, such as pathology reports, progression records, DICOM images, medication lists, etc.

[0153] The database 18 may further include other health data associated with each patient, such as next generation sequencing (NGS) information generated from the patient's blood, saliva, or other normal specimens, NGS information generated from the patient's tumor specimen, imaging information, such as radiological images, pathology images, or extracted features thereof, other omics information, such as metabolic information, epigenetic analysis, proteomic information, and the like. Examples of NGS information may include DNA sequencing information and RNA sequencing information. Examples of imaging information may include radiation therapy images, such as planning CT, contours (rtstruct), radiation plans, dose distributions, cone beam CT, radiology, CT, PET, and the like. The information in the database 18 may include disease state information at the time of diagnosis (e.g., cancer diagnosis), longitudinal information of the patient, such as 6 months post diagnosis, 1 year post diagnosis, 18 months post diagnosis, 2 years post diagnosis, 30 months post diagnosis, 3 years post diagnosis, 42 months post diagnosis, 4 years post diagnosis, and the like. The information in the database 18 may include protected health information. The information in the database 18 may include de-identified information.For example, information in Database 18 may include (1) patient name, (2) address, city, county, district, zip code, and their equivalent geographic codes, but not (a) all geographic subdivisions smaller than a state where the geographic unit formed by combining all zip codes with the same first three digits contains 20,000 people or more, and (b) all such geographic units containing 20,000 people or less have the first three digits of the zip code changed to 000, according to data currently publicly available from the Census Bureau, (3) all elements (except year) of dates directly related to the individual, including date of birth, date of admission, date of discharge, date of death, and all ages 89 years or older, and where such ages and elements are aggregated into a single category 90 years or older. The database 18 may be in a structured format that does not include all elements of such age-indicative date (including year), (4) telephone numbers, (5) vehicle ID and serial numbers, including license plate numbers, (6) fax numbers, (7) device identifiers and serial numbers, (8) email addresses, (9) web universal resource locators (URLs), (10) social security numbers, (11) internet protocol (IP) addresses, (12) medical record numbers, (13) biometric identifiers, including fingerprints and voice prints, (14) health insurance beneficiary numbers, (15) full face photographs and any equivalent images, (16) account numbers, (17) certificate / license numbers, and (18) any other unique identifying numbers, features, or codes, except that the number of records of information in the database 18 may reflect information from 10, 100, 1,000, 10,000, 100,000, 1,000,000, 10,000,000, or more patients. Other examples of the type of information in database 18 are described in U.S. patent application Ser. No. 16 / 657,804, filed October 18, 2019, the contents of which are incorporated by reference in their entirety into this specification for all purposes.

[0154] The collaboration device 20 can reduce the amount of personal health information that may currently be included in emails sent to the oncologist. The collaboration device 20 can delete the generated response (visual or audio) from memory 34 after the response is output through speaker 44. Thus, the personal health information may be eliminated from memory external to database 18. The collaboration device 20 can be configured to recognize a “can you repeat that” command that causes the collaboration device 20 to re-query the last question asked by the oncologist (e.g., from collaboration server 12), play the response again, and delete it from memory 34.

[0155] 2 and 3, an embodiment of an exemplary second collaboration device 20a is shown. The collaboration device 20a can include a second external housing 22a including seven substantially flat sides. The collaboration device can include a second light device 50a, one or more second microphones 38a that can be positioned near a small circular opening in the second external housing 22a, and a second speaker 44a that can be positioned near a substantially oval opening in the second external housing 22a. In some embodiments, the second external housing 22a can be made of metal and may include stainless steel and / or anodized steel. The second collaboration device 20a can be approximately 1.5 inches wide, 1.5 inches high, and 1.5 inches deep. In some embodiments, the second speaker 44a can be a headphone coupled to the second collaboration device wirelessly (e.g., via Bluetooth) or via a wired connection (e.g., a 3.5 mm jack audio cable).

[0156] 2 and 3 and 4, the second collaboration device 20a, similar to the collaboration device 20 described above, may include a processor 30 linked to a battery 32, a memory 34, a transceiver 36, input buttons 52, a display screen 48, and each of the sensors 40, 42. The processor 30 may also be linked to a second light device 50a, a second microphone 38a, and a second speaker 44a. The transceiver 36 may be configured to communicate using a 5G cellular network protocol.

[0157] The second collaboration device 20a can include a touch interface 23 that can receive input from an oncologist. The touch interface 23 can be linked to a processor 30. The touch interface 23 can include a second outer housing 22a and a sensor (not shown), such as a force sensor, coupled to the second outer housing 22a. The sensor can sense deflection of the second outer housing 22a and output a corresponding signal to the processor 30. In some embodiments, the sensor can include a new technology force sensor film. The second outer housing 22a can be marked (e.g., engraved) at appropriate locations to identify different sensing areas that may correspond to different virtual buttons (e.g., "power", "OK", "mute", etc.) for the oncologist.

[0158] In some embodiments, the touch interface 23 may include one or more touch sensors disposed to receive input from the oncologist. The one or more touch sensors may include one or more capacitive touch sensors configured to output a signal to the processor 30. In some embodiments, the touch sensors may be disposed beneath the one or more display screens 48. The processor 30 may provide input (e.g., by displaying instructions and / or prompts on the one or more display screens 48 and / or by issuing audible instructions at the speaker 44a) to prompt the user to receive a signal associated with the oncologist from the one or more touch sensors. The processor 30 may authenticate the oncologist based on the signal by determining whether the signal matches a predefined fingerprint profile associated with the oncologist. The processor 30 may determine a selection (e.g., a menu option, a power on / off command, a mute command, etc.) based on the signal.

[0159] The second collaboration device 20a may also include a power interface module 33 coupled to the battery 32 to provide power to the battery 32. The power interface module 33 may include any suitable hardware for conditioning the power provided to the battery. The power interface module 33 may include a hardwired interface (not shown) for connecting to a complementary interface coupled to an external power source. The hardwired interface may include copper or gold contact pins and may be magnetic. Alternatively, the power interface module 33 may include one or more transformers (not shown) for wirelessly receiving power. The one or more transformers may include a pot core transformer configured to receive power transmitted at approximately 300 MHz, which is higher than other wireless charging systems that use standards such as the Qi wireless power transmission standard. Certain wireless charging systems may use coils to implement wireless charging. These wireless charging systems that may be compatible with the Qi standard may not be usable with the second collaboration device 20a due to the small size of the collaboration device 20a. In embodiments in which the second external housing 22a is metallic, care must be taken to ensure that wireless charging does not result in heating of the second external housing 22a. A pot core transformer may funnel the transferred energy into the battery 32 to prevent energy from being dissipated in the second external housing 22a rather than in the coil transformer.

[0160] The second collaboration device 20a may include a mesh networking transceiver 37 linked to the processor 30. The mesh networking transceiver 37 may be a Wi-Fi transceiver, a Z-Wave transceiver, a Zigbee transceiver, or a combination of different types of transceivers (e.g., including Z-Wave and Zigbee). In particular, the mesh networking transceiver 37 may communicate at frequencies other than one or more of the frequencies used by the transceiver 36 to communicate with the transceiver 16, as shown in FIG.

[0161] Although the transceiver 36 may be used to communicate with transceivers included in other collaboration devices, the mesh networking transceiver 37 may reduce potential transmission traffic on communication frequencies used by the transceiver 36 to communicate with the transceiver 16. For example, the transceiver 36 may be used to communicate with the transceiver 16 at a 2.4 GHz frequency (e.g., using a Wi-Fi or Bluetooth protocol), and the mesh networking transceiver 37 may be used to communicate with another mesh networking transceiver 37 in another collaboration device 20a at a 900 MHz frequency (e.g., using a Z-Wave protocol).

[0162] The transceiver 36 and / or the mesh networking transceiver 37 may be configured to transmit and receive information using an ultra-wideband (UWB) protocol. UWB may be useful for detecting the real-time location of the second collaboration device 20a and / or for tracking how an oncologist is operating the second collaboration device 20a.

[0163] The second collaboration device 20a may include a secure element 35 linked to the processor 30. The secure element 35 may perform authentication and encryption tasks such as key storage. The secure element may include an ATECC608A microchip from Microchip Technology Inc.

[0164] It is understood that at least some of the components included in the second collaboration device 20a, such as the touch interface 23, the second optical device 50a, the second microphone 38a and the second speaker 44a, the mesh networking transceiver 37, the secure element 35, and the power interface module 33, may be included in the collaboration device 20 and may be linked to the processor 30 and / or the battery 32. In some embodiments, the processor 30 may be a SoC, such as the Qualcomm QCS405 SoC mentioned above. The SoC may be used to implement edge computing as well as machine learning processes locally in the second collaboration device.

[0165] Referring now to Figure 5, a process 100 for facilitating a collaboration session that may be implemented via the systems of Figures 1 and 2 is shown, consistent with at least some aspects of the present disclosure. Process 100 will first be described in the context of a system in which the only interface device used by the oncologist is collaboration device 20 (e.g., the system does not include a supplemental or additional large display screen or other light emitting surface for presenting additional visual data response representations to the user). In this type of system, the portions of process 100 enclosed in dashed lines are not present.

[0166] 1, 2, and 5, at process block 102, an industry-specific data set is stored and maintained in database 18. At block 104, intent matching module 72, parameter extraction module 74, and audio response module 76 are each trained using Dialog flow or some other conversation definition application as described above. Further, at block 104, visual response module 62 is programmed to receive data responses from module 64, where the responses provide seed data for constructing a graphic or other visual representation of the response information.

[0167] 1 and 5, in a system including only an interface device 20, control passes from block 104 to block 106, where the collaboration device 20 monitors for activation (e.g., voice activation, movement, selection of an activation button, etc.). When the collaboration device 20 is activated in block 108, control passes to block 112, where a voice signal is captured by the device 20 and the voice signal is transmitted to the collaboration server 12 (57). In block 114, the captured voice signal is transmitted to the AI ​​server 14 (61), where the ASR module 70 transcribes the voice signal into text, the intent matching module 72 examines the text file to determine the oncologist's intent, and the parameter extraction module 74 extracts key parameter values ​​from the transcribed text. The text file, intent, and extracted parameters are returned to the collaboration server 12, and more specifically, to the data operation module 64 (63).

[0168] In block 116, the data action module 64 instantiates a new collaboration record on the database 18 and stores the text file in the collaboration record (65). The action module 64 also uses the intent and the extracted parameters and associated values ​​to construct a data action in block 118, which is executed in block 120 resulting in a data response. In process block 124, the action module 64 provides the data response to the AI ​​audio response module 76 (69), which generates an audio response file. The audio response file is sent back to the collaboration application (71) and sent to the collaboration device 20 in process block 126 (73, 81). The audio response file and associated text are stored in block 126 as part of the collaboration record. The audio response file is broadcast (66) over the speaker 44 of the device 20 for the oncologist to hear in block 128, after which control returns to block 106 and the process continues to cycle indefinitely.

[0169] If the collaboration session persists over multiple rounds of oncologist queries and system responses, the oncologist's voice messages and associated text and response files, and each of the associated text, are stored in a collaboration record, and a series of preceding and following voice and response messages are captured for subsequent access and review.

[0170] In at least some embodiments, the system also supports audio file broadcasting capabilities, as well as visual output capabilities that provide process status or state information, as well as at least some level of response data in response to user queries. For example, in FIG. 1, once the oncologist's speech signal is captured by device 20 and AI server 14 generates the transcribed text, server 12 may transmit that text file to device 20 to be presented in real time via display 48 as a feedback mechanism to allow the oncologist to confirm that the query was accurately perceived. Here, in some cases, the feedback text may persist (e.g., lasting for a few seconds in most scenarios) or may persist for a set period of time (e.g., 5-7 seconds) before being replaced by a visual data response if necessary. In other cases, the feedback text may only be replaced via the next feedback text phrase, allowing the oncologist more time to evaluate the accuracy of the perceived speech.

[0171] As another example, still referring to FIG. 1 where the data response is suitable for visual representation or even best presented visually via the device display 48, the data response or a portion thereof may be provided to a visual response module 62, as shown at 63. In these cases, the module 62 uses the data response to create a visual response file (see 77 and 81) that is transmitted to the device 20 to drive the display 48. In some cases, the presented visual response may include a textual representation of an audio response file. In other cases, the visual response may include a reminder, an alert, a notification, or any other type of user instruction. When a visual file is generated and presented to the user, the collaboration server 12 may store all visual representations as part of the ongoing collaboration record for subsequent access.

[0172] 6, an exemplary collaboration conversation between an oncologist 150 and a collaboration device 20 is shown, with the oncologist's voice message shown in the left column 160 and the interleaved audio response broadcast by the device 20 shown in the right column 162. When the device 20 is activated, it responds with the phrase "How can I help you?" to prompt the oncologist 150 to utter a first substantive query of the database 18. The oncologist 150 responds with the first query, "Select patients with pancreatic cancer." Here, consistent with the above description, the AI ​​server 14 (FIG. 1) identifies the intent and query parameters used to construct a data operation that generates a data response and ultimately an audio response, "Patients with pancreatic cancer cohort identified." The oncologist 150 then utters a second query, "Limit cohort to males," causing the system to construct and execute another data operation to generate another audible response. This back and forth "conversation" continues until Oncologist 150 ends the session.

[0173] When the collaboration application 60 stores collaboration records in the database 18, the system allows the oncologist to access those records to update the memory or to initiate more detailed lines of queries supported by additional output affordances such as a large workstation display screen. For this purpose, see FIG. 7, which shows the input and output devices at the workstation invoking a large flat panel display screen 170, a keyboard 172, and a mouse input device 174. The mouse 174 controls on-screen pointing icons 176 for selecting virtual icons and tools on the screen, as is well known in interface technology. The screenshot on the display 170 shows a collaborator window 180 containing a list of oncologist-system collaborations for a particular oncologist that can be selected to access the complete collaboration record. The list includes two columns, including a date column 182 showing the date of the corresponding collaboration session, and a collaboration column 184 containing the first query corresponding to each collaboration represented in the list. The first entry in column 184 corresponds to the collaboration session shown in FIG. 6 and is shown selected via icon 176, highlighted to indicate selection.

[0174] When the first entry in column 184 is selected, screenshot 190 shown in Figure 8 may be presented, including the complete collaboration record in text with the oncologist query in a first column 192 and the audio system response represented as text in a second column 194. The example in Figure 8 corresponds to the conversation in Figure 6. Here, although the conversation is presented as text, it is contemplated that the oncologist may play an audio recording of the conversation as a memory aid, and to that end a selectable "play" icon 196 is provided to play the collaboration audio.

[0175] Although collaboration device 20 is advantageous due to its relatively small size and portability, in at least some cases, data response presentation is more appropriate via visual representation than audio, or audio representation is best supplemented via visual representation at a larger scale than provided by device display 20. To this end, it is contemplated that where a larger visual representation of the response data is optimal, portable collaboration device 20 may be supplemented as an output device via a proximate large flat panel display screen. Referring now to FIG. 9, an input / output configuration 200 is shown that may take the place of collaboration device 20 in FIG. 1. In FIG. 9, the input / output configuration includes portable collaboration device 20, a proximate large flat panel display screen 202, and an input keyboard 204 and mouse device 206, respectively.

[0176] Still referring to FIG. 9 , in at least some cases, device 20 may be programmed to wirelessly “pair” with any Bluetooth or other wireless protocol enabled display screen in the general vicinity of device 20 when some pairing event occurs. Here, the pairing event may simply include any time device 20 is in proximity to a pairable display 202, regardless of whether device 20 is activated to listen for a user's audio signals. In other cases, device 20 may only pair with a display when device 20 becomes active (e.g., a pairing event would be the activation of device 20). In still other cases, pairing may occur only when device 20 receives a video response file that requires large display 202 for content presentation (e.g., a pairing event would be the receipt of a video file containing data optimally presented on a large display screen).

[0177] Regardless of the pairing event, the pairing may occur automatically upon the occurrence of the event or may require some affirmative action by the user to pair. For example, the affirmative action may include device 20 broadcasting a voice query to the user requesting permission to pair with display 202, and the user vocalizing a “yes” response in return.

[0178] When device 20 is paired with display 202, application programs executed by the display processor may take over the entire display desktop image and present a large-scale collaboration interface across the entire display screen. Alternatively, the application may open a collaborator window 210, as shown in Figure 9, to present a visual response file. In Figure 9, an exemplary visual response representation is shown at 212.

[0179] In at least some cases, the collaborator window 210 or desktop image may be automatically presented via the display 202 when a pairing event occurs. In other cases, even if the device 20 is paired with the display 202, the collaborator window 210 may not be provided until some secondary triggering event occurs, such as, for example, the device 20 is activated or a visual response file is received that is displayed on the display 202. In still other cases, the window 210 may be presented only after a user takes an affirmative action to pair the device 20 with the display 202.

[0180] In at least some embodiments, even if the device 20 is paired with the display 202, the response file may sometimes be presented to the user only via the device 20. For example, in many cases the collaboration server 12 generates only an audio response file, in which case the audio file is broadcast only via the device 20, without a visual representation on the display 202. Here, some user queries may result in a response only via the device 20, other queries may result in a response only via the display 202, and still other queries may result in a combined response via each of the device 20 and the display 202.

[0181] As noted above, in at least some embodiments, all collaboration system communications with the display 202 may be through the device 20, such that the server 12 does not communicate directly with the display 202. In other cases, it is contemplated that the display 202 will have its own Internet of Things (IoT) address, and thus the server 12 may be able to communicate visual response files directly to the display 202. In this case, pairing requires a location-based association of the device 20 and the display 202, with that association information being stored in a database by the server 12 so that audio and visual response file transmissions to the device 20 and the display 202 can be coordinated.

[0182] It is contemplated that, at least in some cases, if a visual response file is presented on the paired large display 202, the tailored visual response may be presented via the collaboration device display 48 directing the oncologist to the larger display 202. Similarly, the audio broadcast by the device 20 may direct the oncologist to the larger display 202 and may include some sort of summary message related to the visual representation of the large display 202. In FIG. 7, the illustrated audio broadcast 220 summarizes the visual content on the large display 202, with the device display 48 instructing the oncologist to refer to the larger paired display 202 for more detailed information.

[0183] In yet other cases, when the portable collaboration device 20 is far from the large display and cannot pair, and the response file is best presented via the large display, the system may notify the oncologist that a better response can be obtained by a pairing device 20 with an auxiliary large display, where the notification may be presented via the device display 48 or audibly via the speaker 44. The notification may be in addition to broadcasting an audio response file that includes the omitted response data.

[0184] When the system 10 presents visual data via the display screen 202 during a collaboration session, in at least some embodiments, all visual files presented are stored in a collaboration record for subsequent access. To this end, see, for example, FIG. 8, in which a third record column 196 includes visual response data 198 corresponding to each of the audio responses in column 194, where each visual response is accessible to view information visually presented during the associated collaboration session. FIG. 10 shows one of the visual response icons selected, which opens a sub-window 230 to present visual content presented during the previous session.

[0185] It is contemplated that, at least in some cases, system 10 will generate data responses suitable for generating both audio and visual response files that are stored in a collaboration record without presenting any visual information during collaboration, where all communication during a collaboration session occurs through device 20, despite the generation of a useful visual response file. The visual information may then be subsequently accessed through interfaces similar to those shown in Figures 8 and 10.

[0186] 11, a second exemplary system 300 consistent with at least some aspects of the present disclosure is shown. Here, unlike the system of FIG. 1, where the AI ​​processes are executed by a separate AI server 14, the AI ​​processes are executed by a portable collaboration device 20 that passes information to the collaboration server 12 for performance or execution of data operations. As shown, an ASR module 70, an intent matching module 72, and a parameter extraction module 74 are all respectively included in the device 20. An oncologist's voice signal captured by the device 20 is provided to the ASR module 70, which generates a test that is provided to the intent matching module 72 (310). Module 72 identifies the oncologist's intent, and then module 74 extracts parameters from the voice signal, and each of the text, intent, and extracted parameters are wirelessly transmitted to the collaboration server 12 via the transceiver 16 (302). The server 12 operates in the same manner as described above to create and build a collaboration record based on the oncologist's voice message and the system responses, and to use the intents and parameters to formulate data operations to be performed on the database 18 to generate the data necessary to respond to the oncologist's query. The data responses are sent (304) back to the device 20 where the voice response module 76 generates an audio file to drive the speaker 44 and present the audio response.

[0187] 12, a third exemplary system 320 is shown consistent with at least some aspects of the present disclosure. Similar to the second exemplary system 300 shown in FIG. 11, the AI ​​processes performed by the independent AI server 14 are executed by the portable collaboration device 20. Unlike the second exemplary system 300, the processes performed by the independent collaboration server 12 are executed by the portable collaboration device 20. The collaboration device 20 is linked to the database 18 to send data operations 322 to the database 18 and receive data responses 324 from the database 18. The collaboration device 20, and more specifically the audio response module 76, then generates an audio file to drive the speaker 44 and present the audio response as described above.

[0188] 9 and 12, by implementing at least a portion of the processes performed by the AI ​​provider server 14 and the collaboration server 12 locally on the collaboration device 20, the latency of generating an audio response can be reduced by up to 2 seconds. In some embodiments, at least a portion of the modules 62, 64, 70, 72, 74, 76 and / or collaboration application 60 can be stored on the collaboration server 12 or the AI ​​provider server 14, periodically updated, and pushed to the collaboration device 20. In other words, the collaboration server 12 or the AI ​​provider server 14 can store the latest versions of the modules 62, 64, 70, 72, 74, 76 and / or collaboration application 60, and periodically (e.g., once a day or once a week) update the processes stored on the collaboration device 20 to include the latest modules 62, 64, 70, 72, 74, 76 and / or collaboration application 60 processes stored on the collaboration server 12 or the AI ​​provider server 14. In this manner, the process performed by collaboration device 20 can be continually updated, reducing the latency of generating audio responses based on input from the oncologist.

[0189] Referring to FIG. 13, several collaboration devices 20b-e can communicate with each other and / or with at least one of the first transceiver 16a and the second transceiver 16b using mesh networking techniques. The plurality of collaboration devices 20b-e can include a third exemplary collaboration device 20b, a fourth exemplary collaboration device 20c, a fifth exemplary collaboration device 20d, and a sixth exemplary collaboration device 20e. Each of the collaboration devices 20b-e can include at least some of the components of the collaboration device 20 or the second collaboration device 20a described above. In some embodiments, each of the collaboration devices 20b-e can be the collaboration device 20 or the second collaboration device 20a. Although four collaboration devices 20b-e are shown, it is understood that more than four collaboration devices can be used. Each of the first transceiver 16a and the second transceiver 16b can be the transceiver 16 described above.

[0190] Each of the multiple collaboration devices 20b-e may include a corresponding transceiver 36b-e, each of which may be substantially similar to the transceiver 36 described above. Each of the multiple collaboration devices 20b-e may be linked to the first transceiver 16a and / or the second transceiver 16b to transmit voice and message signals to the first transceiver 16a and / or the second transceiver 16b using the corresponding transceiver 36b-e included in one of the collaboration devices 20b-e. For example, the fourth collaboration device 20c may be linked to the second transceiver 16b, the fifth collaboration device 20d may be linked to the first transceiver 16a and the second transceiver 16b, and the sixth collaboration device 20e may be linked to the second transceiver 16b.

[0191] The first transceiver 16a and the second transceiver 16b can be linked to the collaboration server 12 to transmit voice signal messages to the collaboration server 12 and to receive visual and audio response files transmitted from the collaboration server 12 as described above. The collaboration server 12 can be linked to the AI ​​provider server 14 to transmit voice signal messages and data responses to the AI ​​provider server 14 and to receive text files associated with the voice signal messages, matching intents, and extracted parameters, as well as audio response files associated with the transmitted data responses from the AI ​​provider server 14. The collaboration server 12 can be linked to the database 18 to create collaboration records and perform data operations in the database 18, as well as to receive transmitted data responses from the database 18.

[0192] Each of the multiple collaboration devices 20b-e can directly communicate with at least one other collaboration device 20b-e to form a mesh network. The multiple collaboration devices 20b-e can communicate with each other using a communication protocol supported by a corresponding transceiver 36b-e, for example, a Wi-Fi protocol, a UWB protocol, and / or a Zigbee protocol. Each of the multiple collaboration devices 20b-e can also include a corresponding mesh networking transceiver 37b-e, each of which can be substantially similar to the mesh networking transceiver 37 described above.

[0193] The multiple collaboration devices 20b-e can directly communicate with each other using the corresponding mesh networking transceivers 37b-e, thereby reducing transmission traffic on the communication frequencies used by the corresponding transceivers 36b-e. Direct connections between the collaboration devices 20b-e can be useful when one of the multiple collaboration devices 20b-e cannot communicate with either the first transceiver 16a or the second transceiver 16b. For example, if the third collaboration device 20b cannot communicate with the transceivers 16a-b, the third collaboration device 20b can route communication through the fifth collaboration device 20d linked to the first transceiver 16a to transmit voice signal messages and receive audio and / or visual response files transmitted as described herein. Thus, all of the collaboration devices 20b-e can be linked to the collaboration server 12.

[0194] The location of each of the multiple collaboration devices 20b-e can be determined to potentially prevent the loss or theft of the multiple collaboration devices 20b-e. A monitoring process, which may be included in a server in communication with the first transceiver 16a and the second transceiver 16b (e.g., the collaboration server 12), can monitor the locations of the collaboration devices 20b-e. The monitoring process can cause heartbeat messages to be transmitted from the transceivers 16a-b to the collaboration devices 20b-e and can receive heartbeat messages transmitted from the collaboration devices 20b-e to the transceivers 16a-b. The monitoring process can then determine the location of each of the collaboration devices 20b-e based on the heartbeat messages.

[0195] In some embodiments, each of the multiple collaboration devices 20b-e may transmit a heartbeat message to the first transceiver 16a and the second transceiver 16b at a predetermined interval (e.g., every 10 minutes), which may then retransmit the heartbeat message to another device, such as the collaboration server 12. In this manner, other devices and / or processes, such as the collaboration server 12, may track and / or triangulate the location of a given collaboration device (e.g., the fifth collaboration device 20d). In some embodiments, the first transceiver 16a and the second transceiver 16b may be associated with the MAC addresses of wireless access points, which may be associated with GPS coordinates. A monitoring process may then estimate the locations of the collaboration devices 20b-e based on the GPS coordinates (indirectly) associated with the transceivers 16a-b. The monitoring process can determine that a given collaboration device (e.g., the third collaboration device 20b) has transmitted a heartbeat message to both transceivers 16a-b and estimate the location of the given collaboration device based on the GPS locations associated with the transceivers 16a-b. The GPS coordinates and / or MAC addresses can be stored in the collaboration server 12.

[0196] If a heartbeat message transmitted by one of the plurality of collaboration devices 20b-e is not received by both the first transceiver 16a and the second transceiver 16b, the monitoring process may determine that the device is lost and notify a system administrator and / or the monitoring process that the device is lost. The first transceiver 16a and the second transceiver 16b may also transmit heartbeat messages to the plurality of collaboration devices 20b-e to ensure that the device is not potentially lost or stolen. In some embodiments, if one of the plurality of collaboration devices 20b-e, such as the fifth collaboration device 20d, does not receive a heartbeat message from the transceivers 16a-b within a predetermined interval, the fifth collaboration device 20d may enter a restricted mode that limits the processes that can be executed by the fifth collaboration device 20d and / or lock the fifth collaboration device 20d to help prevent potential tampering with sensitive data.

[0197] Alternatively, or in addition to using the transceivers 16a-b to track the locations of the multiple collaboration devices 20b-e, the collaboration devices 20b-e themselves can be used to track each other. More specifically, one or more of the collaboration devices 20b-e can track another one of the collaboration devices 20b-e using one or more of the direct connections between the collaboration devices 20b-e. One of the collaboration devices 20b-e may communicate directly with another one of the collaboration devices 20b-e using a UWB protocol. For example, the third collaboration device 20b can be linked to the fifth collaboration device 20d and the sixth collaboration device 20e. The third collaboration device 20b can send heartbeat messages to the fifth collaboration device 20d and the sixth collaboration device 20e, and in response, the fifth collaboration device 20d and the sixth collaboration device 20e can send heartbeat messages back to the third collaboration device 20b. If the third collaboration device 20b does not receive heartbeat messages returned from the fifth collaboration device 20d and the sixth collaboration device 20e, the third collaboration device 20b may enter a restricted mode that limits the processes that can be executed by the third collaboration device 20b and / or may lock the third collaboration device 20b to help prevent potential tampering of sensitive data. Furthermore, the fifth collaboration device 20d and / or the sixth collaboration device 20e may send a notification to at least one of the transceivers 16a-b that the third collaboration device 20b has potentially been lost or stolen. The transceivers 16a-b may transmit the notification to the collaboration server 12 for further processing.

[0198] In some embodiments, at least some of the processes stored and executed on AI server 14 and / or collaboration server 12 may be stored locally on collaboration devices 20b-e. In these embodiments, the processes stored and executed on AI server 14 and collaboration server 12 may be updated, for example, continuously by an external program or internally by collaboration server 12 and / or AI server 14. As processes are updated, collaboration server 12 and / or AI server 14 may update the corresponding processes stored on collaboration devices 20b-e. Processes that may include speech recognition, intent prediction, analysis, and / or routing processes may be updated based on data generated by collaboration devices 20b-e.

[0199] For example, the third collaboration device 20b and the fourth collaboration device 20c may receive voice signal messages with different phrases (e.g., phrases with different word choices) corresponding to the same intent. The AI ​​server 14 can then learn that the different phrases match the same intent and update the associated module (e.g., the intent matching module 72 shown in FIG. 1) accordingly. Some of the collaboration devices 20b-e can be located within the same institution (e.g., the third collaboration device 20b, the fifth collaboration device 20d, and the sixth collaboration device 20e), and others can be located at another institution (e.g., the fourth collaboration device 20c). In this manner, the AI ​​server 14 and / or the collaboration server 12 can be updated based on feedback from multiple oncologists from multiple institutions.

[0200] Additionally or alternatively, processes stored and executed on AI server 14 and / or collaboration server 12 can be updated based on external processes. For example, an administrator can add intents to AI provider server 14. The AI ​​serving server can then upload the updated processes to collaboration devices 20b-e.

[0201] After an audible collaboration session, it is often difficult to return to the same dialog flow at a later time because it is difficult to remember the communications that preceded and followed the dialog. For this reason, at least in some cases, the system allows the user to reinsert themselves into the flow using a display screen such as that shown in FIG. 8. Thus, in FIG. 8, a "Continue" button 197 is presented that is selectable to place the entire system 10 in the state that existed at the end of the session. Here, "state" means that all the context associated with the lines of question (e.g., subsets of data, qualifying parameters, etc.) is restored at the end of the session, allowing the oncologist to pick up where they left off, if desired.

[0202] One problem oncologists and physicians in general have is the need to enter notes into a patient's record every time they see and treat a patient. At least some studies have shown that a typical oncologist spends more than 1.5 hours each day memorializing events and thoughts in their patient notes. Some oncologists record or document notes during the patient visit, while others wait until they have a break or are "off work" to write notes. When oncologists write notes with patients, the physician's attention is divided between the notes and the patient, which is not ideal. When oncologists write notes following a patient visit, thoughts, observations, and findings are often misremembered or captured with fewer details.

[0203] To address this issue, at least in some cases, the portable collaboration device 20 may be programmed to "listen" to an oncologist-patient care episode and record at least a portion of the oncologist and patient interactions essentially in real time as a "raw transcription." Additionally, the system processor may be programmed to process the raw transcription data through OCR and NLP algorithms to identify words, phrases, and other content in the captured raw speech signal. It is contemplated that, at least in some cases, the processor may be trained using Dialogflow or other AI software programs to recognize the oncologist's intent from the captured words and phrases and various parameters necessary to instantiate different types of structured notes, records, or other documents that match one or more of the oncologist's intents. Additionally, it is contemplated that the processor may take into account other patient visit context in identifying the oncologist's intent, as well as in identifying key parameters of a particular structured note, record, or document.

[0204] For example, while speaking with a patient with pancreatic cancer, the processor may use the oncologist's appointment schedule to automatically identify the patient and access the patient's medical records to be used as context for the voice messages captured during the patient's visit. When the oncologist and patient speak, the processor may be programmed to recognize the oncologist's voice and the patient's voice, where over time the processor trains on the oncologist's voice and becomes able to recognize the oncologist's voice based on tone, pitch, voice quality, etc., and is programmed to assume that other voice signals that do not match the oncologist's are those of the patient.

[0205] At least in some cases, the oncologist may intent-indicate the type of structured note the system will generate. For example, in a simple case, the system may be programmed to generate five different structured note types, each type including a different subset of 15 different parameters. Here, during Dialogflow training, the administrator may provide five different phrases for each of the five different note types, each phrase being associated with an intent to generate the associated note type. The processor trains five phrases for each note type and comes up with many other phrases to associate with the note type intent. Further, during training, the 15 parameter subsets for each note type are specified. Further, structured note types are created and stored in a structured note database for use in instantiating a particular instance of the note type for a particular patient visit. Further, feedback queries of at least the required parameters may be created and stored, as in the case of the Dialogflow system described above.

[0206] During a patient visit with an oncologist, if the oncologist wants the system to generate a particular note type, the oncologist can simply activate device 20 by uttering a phrase such as "go, one" and then "create an instance of the first note type." The processor recognizes the intent to create an instance of the first note type and then listens during the dialogue to select the parameters needed to instantiate the instance of the note type. In at least some cases, if the system cannot identify some parameters needed for the note instance, device 20 may be programmed to query the oncologist for the missing parameters. Feedback queries may be generated during the patient visit, immediately after the visit when facts and information about the visit are fresh in the oncologist's mind, or at other scheduled times such as breaks, scheduled office hours, etc.

[0207] In other cases, instead of having the physician vocalize the specific note type to create, the system may listen to the oncologist-patient dialogue and identify the oncologist's intent from the ongoing dialogue without a specific request.

[0208] Raw transcriptions, notes, records, or other documents generated by the system during or in association with a patient visit may be stored in the patient's EMR or any other suitable database. The AI ​​can learn over time from the oncologist's utterances and become smarter as described above. Additionally, structured notes may be presented to the oncologist for review before or after storage so that the oncologist can review the information in the structured record. If the oncologist modifies the information captured by the system, any changes may be passed back to the system processor and used to further train the processor AI to more effectively capture future intents and / or parameters.

[0209] Another document type that the system may generate automatically, at least in some cases, is a billing document. Again, the system processor may "listen" to what the oncologist is saying during the patient visit and may identify intent that affects billing. At that point, the processor may start listening for other parameters to instantiate a complete billing record or document. In some cases, the billing record may be automatically sent to the billing system or may be presented in some way to the oncologist to verify the accuracy of the billing record before transfer.

[0210] In yet other cases, another document type that the system may automatically generate while listening to an oncologist is a schedule appointment. Again, the processor may recognize the oncologist's intent to schedule an appointment from many different utterances and simply listen for other parameters necessary to instantiate a complete event scheduling action.

[0211] In a particularly advantageous system, the processor may be programmed to listen to the oncologist and automatically identify several simultaneous intents to generate several different types of notes, records, or documents, and may monitor the oncologist's speech to identify all the parameters required for each of the simultaneous intents. For example, if the processor determines that a billable activity or event has occurred and that the oncologist wants to simultaneously generate a structured patient visit note, each of the structured bill and the structured note requires a separate subset of 15 different parameters, the processor may listen to the oncologist's speech for all the parameters to instantiate each of the bill record and the patient visit note. Again, if the system is unable to capture the required parameters, the processor may generate and broadcast or present (e.g., visually on a display) a query to the oncologist to input the required information at the appropriate time.

[0212] In some cases, an oncologist may indicate automatic document preferences for each patient visit, and the system may then automatically assume the intent associated with each preferred document type and simply listen to the oncologist-patient dialogue to identify the parameters required to instantiate an instance of each of the preferred document types for each patient visit. Thus, for example, an oncologist may want the system to generate structured patient visit notes and structured billing records, and tee up next visit schedule options for each patient the oncologist attends. Now, at the start of each scheduled patient visit session, the system immediately identifies three intents: a patient visit note intent, a billing record intent, and a scheduling activity intent. The system accesses the structured records for each of the intents and proceeds to capture all the parameters required for the intent. In the case of the scheduling activity intent, the system may identify the specific activity to be scheduled based on the captured parameters, and may present the oncologist and patient with one or more scheduling options for the specific activity at the appropriate time (e.g., the last five minutes of the scheduled patient visit). Here, the oncologist and patient may accept to decline scheduling any proposed activity or proposed time for the activity.

[0213] In still other cases, after the system processor identifies an intent based on the oncologist-patient interaction, the processor may be programmed to broadcast a query confirming the intent. For example, if the system identifies an intent to generate a patient visit note, the processor may be programmed to broadcast the query, "Do you want to have a patient visit note generated for this visit?", where an affirmative response causes the processor to identify a structured note format and proceed to collect note format parameters to instantiate the note.

[0214] In at least some embodiments, the collaboration device 20 may listen to all utterances by oncologists, and many oncologists may use the device 20 to capture their utterances and live voice messages. For example, the system may capture all utterances of oncologists during patient visits, tumor board participation, office hours, and other situations in which oncologists are discussing all aspects of cancer treatment. Here, the system processor or server may be programmed to recognize all utterances by the associated oncologist and distinguish them from the utterances of others (e.g., patients, other medical personnel, other researchers, etc.). The processor may store all or at least a subset of the oncologist's live voice messages / utterances and may process those utterances to identify text, words, phrases, context, and ultimately the oncologist's impressions. For example, one impression may be that for a pancreatic cancer patient who initially responded well to drug AAA, but for whom the drug is no longer effective, the next line of attack should be drug BBB.

[0215] In some cases, the system will identify and automatically use the perceived impressions, while in other cases the system will be programmed to immediately present the perceived impressions to the oncologist and allow the oncologist to confirm or reject the impressions. Rejected impressions may be discarded or may be recorded to memorialize the rejection, the rejection itself being an indicator of a typical oncologist's impressions and thus useful for future analysis. Confirmed impressions are stored in the system database for later use. In other cases, impressions may only be periodically presented to the oncologist for confirmation or rejection.

[0216] The oncologist's impressions may be used as seed data for the AI ​​machine learning algorithm, so that over time, the algorithm learns from the impressions and populates the database with new data representing the oncologist's thoughts. The system may be programmed to associate different intents with different thoughts, and then when an oncologist's voice utterance is received, it associates the utterance with the intent, identifies parameters associated with the intent, and then retrieves the oncologist's previous impressions or thoughts and provides a response that is consistent with the previous thoughts or impressions.

[0217] In at least some cases, where the system collects impressions from many different oncologists, the system may combine impressions and thoughts from multiple oncologists so that all oncologists using the system have access to responses informed by at least a subset of the impressions and thoughts from the entire group. Now, as the database of impressions evolves, when an oncologist poses a question to her collaboration device 20, the system may again identify the intent as well as the parameters necessary to search the database for an answer and identify one or more impressions of interest to answer the question.

[0218] At least in some cases, the system could automatically track the effectiveness of cancer or other treatments to be used as a quality metric associated with oncology impressions, where effective treatments are assigned a high confidence or other type of factor, while less effective treatments are assigned based on the relative effectiveness of other treatments for comparable cancer conditions. An oncologist then queries the system, where the system identifies the intent and necessary parameters to generate a structured data query, returning information associated with only the most effective impressions.

[0219] In still other cases, the system may rank certain oncologists based on one or more factors and then present a query response based on or representative of the impressions of only the "top" oncologists. For example, oncologists may be ranked based on peer reputation, based on patient treatment effectiveness on a risk-adjusted basis, or using other methods (e.g., a combination of factors with different weightings). Here, the response is limited to data relevant to only the top oncologists.

[0220] In still other cases, it is contemplated that a query may be limited to data and impressions of only a particular oncologist. For example, a first oncologist may desire the impressions of a second particular oncologist regarding a particular cancer condition. Here, the first oncologist may limit the query to the second oncologist by a particular name. For example, when the first oncologist is collaborating with device 20 to access information related to a first patient, the first oncologist may simply utter, "What would Sue White say?" In this case, the processor capturing the query recognizes the intent of another oncologist's impressions, identifies Sue White as a defining parameter, and then accesses impressions associated with Sue White, as well as other contextual parameters (e.g., patient name, cancer condition factors, etc.) previously captured and recognized by the system during a previous dialogue. The response broadcast or presented to the first oncologist is limited to data and information associated with Sue White.

[0221] In many cases, especially when the system is learning during use, the system may make mistakes and return information that is not what was requested. In some cases, it is clear from the response that the query identified by the system is not what the oncologist intended, but in other cases, the incorrect response may not be facially recognizable from the response. If a response is recognized as incorrect reflecting an incorrectly identified query, one problem is that the oncologist must repeat the query with a better pronunciation. It is conceivable that, at least in some cases, if the oncologist rejects the response, the system may automatically attempt to identify a different query that the oncologist intended and a different appropriate response. For example, if, upon hearing a response, the oncologist utters "no" or some other rejection phrase, the system may recognize that response, create a different query based on the intent and parameters, and then issue a different response.

[0222] In some cases, in addition to recognizing an incorrect response, the response can be used to figure out the error in the query identified by the system that led to the incorrect response. For example, if an oncologist asks for a certain cancer condition characteristic of Tom Green and the system responds with "Tom Brown's characteristics are XXX," the answer can be used to identify that the perceived question was incorrect. In this case, to avoid the oncologist having to re-speak the entire query, the system may be programmed to allow for partial queries where the intent and parameters associated with the previous incorrectly perceived query are used along with additional information in the partial query to recognize a different data action to be performed. Thus, in the above example, the oncologist may respond "No, I meant Tom Green." Now, to access Tom Green's characteristics, the system uses the previous query information including the intent (e.g., the characteristic sought), as well as the new parameter "Tom Green." The idea here is that since the system preserves the context during the dialogue, the oncologist does not have to continually re-speak complex queries that were incorrectly perceived by the system, but instead simply provides a subset of information in the next query selected to clear up the misunderstanding.

[0223] At least in some cases, as indicated above, the answer to the query may not include a clear indication that the query was perceived incorrectly by the system. In some cases, the system may be programmed to provide a confirmation broadcast or other message to the oncologist for each uttered query, or at least a subset thereof, thus allowing the oncologist to confirm or reject the perceived query. Confirmation results in a data action, while rejection causes the system to identify a different query or request a rephrasing of the query. In still other cases, the oncologist may be able to ask the system to broadcast the question (e.g., data action) that the system has perceived for confirmation.

[0224] While the invention may be susceptible to various modifications and alternative forms, specific embodiments are shown by way of example in the drawings and described in detail herein. It should be understood, however, that the invention is not intended to be limited to the specific forms disclosed. For example, while a spherical collaboration device is described above, the portable device can take many different forms. For example, with reference to FIG. 14, the second exemplary collaboration device 20a may include a cube-shaped device including one or more emissive exterior surfaces for providing visual content. As another example, the third collaboration device may include a tablet-type device 20b, or any other portable device with components suitable for performing the above functions.

[0225] In still other cases, the portable collaboration device may be one interface device in a larger interface ecosystem that includes other interface devices with the ability for oncologists to seamlessly move between system interface devices during a collaboration session. For example, the ecosystem may include other interface devices, particularly one or more stationary interface devices with better interface affordances, such as better microphones, larger speaker components, etc. In this regard, for example, refer to FIG. 15, which illustrates another exemplary interface device 20c that is substantially larger than interface device 20 and is provided for stationary use in a workstation 350. The exemplary interface device 20c includes a larger housing structure that forms a cavity for receiving various components as described above with respect to FIG. 2. Here, the speaker will be larger and perhaps of higher quality than the speaker in device 20. In this case, device 20c is intended to be used in locations such as counters, work surfaces, and conference tables in a conference room.

[0226] In at least some exemplary contemplated systems, devices 20 and 20c may work in conjunction with each other, where a collaboration session can be handed off from one of devices 20 to the other 20c to optimize for a given situation. For example, if an oncologist is roaming while collaborating via device 20 and enters a space (e.g., arrives at a workstation) that includes a more affordable fixed device 20c, devices 20 and 20c may recognize each other and communicate wirelessly to coordinate the transfer of the collaboration session from device 20 to device 20c. Here, the collaboration session continues even though the fixed device 20c is being used. Similarly, if an oncologist uses device 20c to collaborate and gets up and leaves the station, the collaboration session may be switched to device 20 automatically or with a user request or confirmation so that collaboration can continue.

[0227] In still other cases, headphones, smart glasses with speakers and microphones, etc. may be used as collaboration devices in the disclosed system. In this regard, see the exemplary headphone assembly 370 in FIG. 16, which includes an ear speaker 372 and a built-in microphone 374.

[0228] Although described in the context of a dedicated collaboration device, aspects of the invention may also be implemented using any type of computer interface device, whether dedicated or not, that has a microphone and speaker to enable conversation between the user and the system. For example, a user's laptop computer may be used as a collaboration device running a collaboration program, an existing voice-activated smart speaker may be used as a collaboration device, etc.

[0229] Technology, or new technology-based tools, are great when they work well for their intended purpose, but when the technology or tool does not work as expected by the user, the user quickly becomes frustrated and often simply dismisses the technology or tool going back to resources to complete various tasks. This tendency to quickly dismiss imperfect new technology is exacerbated when users are very busy and therefore time-constrained. Because oncologists tend to be very busy people, they typically have little tolerance for ineffective or inefficient technologies and tools.

[0230] One problem with dialog systems such as those described herein is that systems that support only a subset of the queries that oncologists may pose are often unable to identify the precise intent of a received query. Then, in response, the system generates an answer for the wrong intent, or simply indicates that the system does not currently have an answer to the posed query. These types of incomplete answers cause frustration and often ultimately cause oncologists to dismiss these types of collaboration systems altogether.

[0231] In at least some embodiments, for a given dataset or record type, it is contemplated that an essentially rich set of intents / parameters, associated database queries and responses are defined using Dialogflow or some other dialog specification software so that the system can effectively answer almost any query posed in relation to the dataset. As new datasets, databases, and record types are linked to the system, additional intents and associated information may be specified for those datasets, databases, and record types. For example, at least in some cases, the system may be programmed to support hundreds of thousands of different intents, including literally predictable intents that may be intended by an oncologist. A team of system administrators / programmers work behind the scenes to identify additional possible intents and supplement the system with new intents / parameters, associated database queries and responses. Additional intents may be based on existing datasets and record types and / or may be developed in response to new data types, new information, and / or new oncological insights that evolve over time.

[0232] When a system supports a huge number of different intents (e.g., tens or hundreds of thousands), distinguishing one intent from another is complicated because, naturally, the larger the number of intents supported, the harder it becomes to recognize the difference between any given intent and a set of similar but distinct intents. The task of accurately identifying intents is exacerbated in Dialogflow-type systems, where the AI ​​engine uses a process of query "fanning" to generate literally hundreds or thousands of similar queries during system training and associate them with a particular intent, thus increasing the likelihood of overlapping fan queries for two or more different intents.

[0233] At least some embodiments of the disclosed system use one or any combination of several techniques to recognize intended intents from intents supported by other systems. The first technique is based on the system operating during a collaboration session to distinguish different "dialog paths" occurring during the session, and information related to a particular dialog path is used to inform subsequent intents during the same dialog path. For example, if a physician asks device 20, "Can you tell me the results of my patient, Dwayne Holder's, sequencing report?" and then subsequently asks the question, "What is the best clinical trial option?", the system determines that these questions are on a dialog path and answers the clinical trial question based on the clinical trial recommendations provided in Dwayne Holder's clinical reports (e.g., the system recommends a clinical trial on the sequencing reports, and the system has access to all the data in each of those reports). In at least some embodiments, only one dialog path is actively followed at a time. Nonetheless, in some cases, the system maintains a memory cache of past dialog paths for the oncologist to inform future questions and answers.

[0234] A second technique for recognizing intended intent in a system that supports a vast number of intents has the system create "entities" around key concepts relevant to the oncologist's query and the associated system response. For example, drugs, drug regimens, clinical trials, patient names, pharmaceutical companies, mutations, variants, adverse events, drug alerts, biomarkers, cancer types, etc. are all examples of entities supported by the exemplary system. While a small number of entities are identified herein, it should be understood that a typical system may support hundreds of different entities.

[0235] At least in some cases, the system may be programmed to connect entities in a query or entities identified in a query path to form entity sets that can be used to narrow down a list of answers that may be the best answers to a particular query. For example, if a query path is associated with a patient, Dwayne Holder, and a medication, XXX, those patient and medication entities may form a set that limits the intent most likely to be associated with a subsequent query. The system may also be programmed to leverage the entities to evaluate whether the physician's question is still part of the same dialog path or whether the new question is related to a new topic that will be associated with a new dialog path.

[0236] A third technique for recognizing intended intent in a system that supports a vast number of intents is commonly referred to as "personalization." The idea here is that many particular oncologists routinely follow similar dialog paths and utter similar queries with persistent syntax and word choices, so once the system has identified persistent query characteristics for a particular oncologist and accurately associated them with a particular intent, subsequent queries with similar characteristics can be associated with the same intent, even if they are qualified by a different set of query parameters.

[0237] In at least some cases, the system builds a real-time profile of each oncologist or other system user based on the oncologist's past query characteristics (e.g., word choice, syntax, etc.), the query paths taken, previous system-provided responses to those queries, the oncologist's responses to the responses (e.g., did the oncologist's response indicate that the system was responsive and therefore the perceived intent was correct), and overall system usage. For example, when an oncologist logs into the system, the system may automatically link to a list of patients the oncologist has sent to a sequencing service provider, the results present in those patients' sequencing reports, and the leading therapies and clinical trials that have been recommended for those particular patients. These linked lists support the decision-making process that the system utilizes to determine the questions the oncologist intends to ask (e.g., the oncologist's intent). For example, if an oncologist logs in and recently saw a patient named Dwayne Holder, even if the system receives distorted audio and converts it to text that sounds like, "How are the results for my patient, Lane Bolda?", the system may be programmed to recognize that the oncologist recently met with Dwayne Holder, who has a name similar to Lane Bolda, and proceed to generate an answer based on that recognition.

[0238] In a particularly advantageous system, all three of the above techniques are used in series, in parallel, or in combination to recognize the intent of an oncologist's query. Thus, for example, the system may use entities to narrow down the oncologist's intent when uttering a particular query, may further narrow down possible intents based on the current query path, and may then select the most likely intent based on personalization features associated with the speaking oncologist.

[0239] It is contemplated that, at least in some cases, the system may provide tools during a system training session to avoid confusion of subsequent intents. For example, assume that the system is already programmed to support 100,000 different intents when an administrator specifies the 100,001st intent and three associated seed or training queries to drive the fanning process of AI engine queries. Now, during the fanning process, the system processor may be programmed to compare the fan query of the 100,001st intent with other queries associated with other intents to identify duplicate or substantially identical queries. At least in some cases, the system may be programmed to automatically avoid cases where fan queries for two or more intents are identical or substantially identical.

[0240] In other cases, when the system recognizes that the first and second queries associated with the first and second intents are substantially identical, the system presents a warning to the administrator, allowing the administrator to evaluate the situation and how to handle the confused situation. In some cases, the substantially identical fan queries may mean that the system already supports the newly specified intent, in which case the administrator may simply stop enabling the new intent. In other cases, the administrator may select one of the previous and new intents to be associated with the query in question, and in other cases the administrator may allow the fan query to be associated with the two intents. In still other cases, the administrator considering the two intents may determine that additional information is needed to identify one or the other or both of the previous and new intents, and may further specify factors to consider in identifying one or the other or both of those intents.

[0241] In operation, when an oncologist voices a query, if the query is associated with two intents, the system identifies both intents and generates a response query that is broadcast to the oncologist to allow the oncologist to consider which intent is meant. In other cases, both intents may match the oncologist's voice query, and thus answers to both queries may be generated and broadcast sequentially to the oncologist for consideration.

[0242] Although the goal of the collaboration system is to handle any question that can be answered using data in the system dataset or database, at least in some cases, despite the intent recognition techniques described above, the system may simply not be able to clearly identify one intent and / or required parameters associated with the intent among many intents supported by the system. For example, it is conceivable that in some cases, the system may not be able to identify the intent associated with the query, or may identify two or more intents associated with the query. In these cases, the system may be programmed to facilitate a triage process to focus on the particular intent of the query. In this regard, at least in some cases, the system may be programmed to generate and broadcast a response query to the oncologist, indicating that the system was unable to determine the user's intent, and requesting the oncologist to restate the query.

[0243] In other cases where the system identifies two or more intents that may be associated with a query, the system may broadcast a query to the oncologist, such as "Did you mean _?", with the blanks filled in with parameters possibly associated with the first intent collected from the first query. The system may ask about a second or other intent if the oncologist indicates that the first intent was not what was intended.

[0244] If the system cannot recognize a particular intent from a query or follow-up answer from an oncologist, the system may automatically broadcast a message to the oncologist indicating that the system was unable to understand the query and that a system administrator may review the query and intent so that the system can be trained to handle the oncologist's query. Queries that cannot be associated with a particular intent are then presented to an administrator, who may review the query in context (e.g., in a dialog path) and may either associate the query with an intent supported by the particular system or specify a new intent and associated (e.g., required and optional) parameters to be associated with the query. Now, if a new intent is specified, the administrator may specify a small set of additional seed queries for the intent, and the system AI engine may facilitate a fanning process that again generates hundreds of additional queries to associate with the new intent. The administrator then specifies one or more data operations for the new intent, as well as an audible response file for generating an audible response for the intent. As new intents, parameters, data behaviors, and response files are published to the system for use, emails or other notifications are automatically generated and sent to oncologists who have raised initially unrecognized queries and, in some cases, appropriate answers to those queries.

[0245] When the system associates a perceived query with an intent supported by a single system and then executes a data operation to access the data necessary to create an audible answer, in at least some cases, the databases and / or records searched will not produce results from which to derive an answer. For example, if an oncologist utters a query about a particular patient by name and no information for that patient exists in the system database, the data operation will not return data to respond to the query. In this case, the system may be programmed to broadcast a message indicating that "there is no data in the system for the patient you identified."

[0246] In other cases, in addition to generating data directly responsive to a query, the system may generate additional data (hereinafter "supplemental data") to supplement the response data. The supplemental data may take essentially any type of form that can be supported by data in the system databases, and may include, for example, qualifying statements or phrases that apply to the associated directly applicable response phrases, additional data of interest, clinical trials that may be relevant to the query, conclusions based on the data, and data that supports the answer statement.

[0247] Here, it is contemplated that the supplemental data can be driven by conditional or supplemental data actions, or actions triggered by the results of primary data actions, and associated answer phrases and sentences. For example, a primary data action that generates data directly responsive to the first query intent may be associated with the first intent, and data from that action may be used to create a direct response answer phrase that directly responds to the oncologist's query paired with the first intent. Additionally, a second or supplemental data action may also be associated with the first intent, and may generate data results that are used to formulate a type of supplemental answer phrase (e.g., qualifying statements, additional data of interest in addition to data directly associated with the initial query, clinical trials of interest, conclusions, and supporting data, etc.), which do not directly respond to the first query, but add additional information of interest to the directly responsive answer phrase. Here, when the primary data action generates results, those results may be used to generate a direct response phrase that responds to the query. Similarly, when a supplemental data action associated with the first intent generates results, those results may be used to generate a second or supplemental answer phrase. In this case, the direct response supplemental phrases may be broadcast in sequence for the oncologist to hear.

[0248] In the above cases, if only the primary data operation generates a result and an associated direct response answer phrase (e.g., if the supplemental data operation cannot generate data that can be used to generate a supplemental response phrase), the system will only generate the direct response phrase. Thus, in these cases, the system response to the query may include either only the direct response phrase, or a sequence that includes a direct response phrase followed by a supplemental phrase.

[0249] In some cases, three, four, five, or more supplemental data actions and answer phrases may be associated with a single intent in the system. Here, once an intent is identified, all data actions (e.g., primary and each supplemental) may be executed in an attempt to generate a result that can be used to generate and broadcast a sufficient system response. If only a subset of the supplemental data actions generate a result, then only the phrases associated with those results are generated and broadcast sequentially. Thus, for example, if a primary and a first through a fifth supplemental data action are associated with an intent, then if the data action generates a result for the primary, second, and fifth supplemental actions, the answer will include three consecutive answer phrases, the first for the primary action result, and the second and third for the second and fifth supplemental action results.

[0250] A supplemental qualifying statement may be based on the inability to effectively provide a complete answer to a query. For example, if a primary data action returns 50 different effective drugs for a particular cancer condition, instead of broadcasting all 50 drugs by voice, the system may identify the three most effective drugs and broadcast them as options along with a qualifying statement such as "There are 47 other effective drugs, if you say email me the complete list of drugs, I can send you the complete list now."

[0251] Another type of supplemental qualified statement may be generated by a supplemental data action that evaluates the weight of evidence supporting the outcome of the primary data action. For example, a direct response query answer may indicate that "There is evidence that at least some patients with the cancer condition will respond positively to YYY treatment" whereas only two previous patients with a particular cancer condition responded positively to YYY treatment, and a supplemental response could be "However, please note that only two patients responded positively to YYY treatment." In this case, the supplemental data action would identify the number of patients responding positively and compare it to a statistically significant number associated with a higher level of confidence, and if that number is less than the statistically significant number, the action would generate the supplemental response as a qualified statement. As another example, if the primary data action response is "Chemotherapy is recommended for pancreatic cancer in the adjuvant setting," a qualified supplemental phrase could be "However, the role of radiation is still being explored in clinical studies." This supplemental phrase would be generated based on the outcome of a supplemental data action associated with the query intent.

[0252] Other types of well-formed statements are possible.

[0253] The additional data of interest can be any data, a subset of data, a compilation of data, or a derivative of the system data. For example, if an oncologist asks about the symptom status of a particular patient, the additional data may include additional typical symptom statuses given the current cancer status of the particular patient.

[0254] The supplemental response may include more information related to the clinical trials identified in response to the primary data action. For example, here, the phrase directly responding to the query may be, "There are two clinical trials that may be of interest to Dwayne Holder," and the supplemental response could be, "The first clinical trial is 23 miles from your office, and the second is 35 miles from your office." Many other supplemental data actions related to clinical trials are considered.

[0255] At least some databases include specialized clinical or other report types developed for a specific purpose in which data is collected from the EMR and other system databases and used to instantiate a particular instance of the report for a particular patient and cancer condition, where at least some of the instantiated reports are generated and stored in a persistent format (e.g., dated and immutable) and in other cases the instantiated reports are stored but dynamic such that the system periodically updates the reports as the patient's cancer condition progresses over time. When reports are stored in a persistent format, multiple instances of the report are persistently stored such that a historical record of the report can be created over time. When reports are dynamically stored, historical values ​​of report fields can be stored to enable later generation of time-based instances of the report that reflect the report information at any point during the patient's treatment.

[0256] One advantage of using a particular type of fully formatted clinical report (e.g., pancreatic cancer, breast cancer, melanoma, etc.) is that an oncologist who routinely uses instantiated instances of a particular report type will immediately understand the type of information available in the report, as well as where in the report the information resides. As the familiarity of the report matures, when specific information related to a particular patient's cancer status is sought, the oncologist will know whether that information is in the patient's clinical report and, upon accessing the report, where the specific information is located.

[0257] Another advantage associated with clinical reports is that the reports serve as summaries of the EMR data and can include additional results of complex data operations on the EMR data, so that the oncologist does not have to manually recreate or process these operations. Thus, the reports can include not only clinically significant EMR data, but also data and other information derived from the raw EMR data. The collaboration device 20 may provide the oncologist with information not available in the clinical report, such as Tempus Insights, actionable mutations, etc.

[0258] 17A-17C, three pages of an exemplary clinical report related to patient Dwayne Holder suffering from pancreatic cancer are shown. The report contains all the important clinical information related to the patient's cancer condition, including report sections clearly marked as: genomic variants, immunotherapy markers, FDA approved therapies and current diagnoses, FDA approved therapies and other indications, current clinical trials, variants of unknown significance, low coverage regions, -clinically actionable somatic variant details, germline variant details, medical history and oncologist notes (see bottom left field of FIG. 17A). Here, the report format is simple and clearly defined, allowing oncologists to quickly find the specific information of interest.

[0259] In view of the present disclosure, the use of formatted clinical reports as the primary data source to drive a voice-based collaboration system facilitates the task of developing a rich set of intents and associated supporting information for those records. In this regard, please refer back to FIGS. 17A-17C. Although a large amount of clinically significant patient information is presented in the reports, the amount of information is limited in order for oncologists to quickly become familiar with the report format and the available data. Because oncologists know the patient's general cancer status (e.g., pancreatic, breast, etc.), as well as the report format and report data type for that status, they will naturally tend to limit system queries to those that are calculated to be answerable via the report type information. Because the report data is limited to a set of the patient's specific medical record data (although it includes all clinically significant data), the number of intents required to support anticipated queries is quite limited. For example, the number of intents required to fully support anticipated queries of the FIG. 17A-17C reports may be on the order of a few thousand, rather than the 100,000 or more of a full EMR.

[0260] Another benefit associated with using formatted clinical reports as the primary data source to drive a voice-based collaboration system is that the limited number of intents required to fully support anticipated queries makes it much easier for the collaboration system to uniquely distinguish the intended intent from all other supported intents. Thus, for example, if only 5,000 intents are required to fully handle all anticipated queries regarding pancreatic clinical record information, accurate intent recognition is more likely to occur than if 100,000 intents had to be supported.

[0261] Another benefit associated with using formatted clinical reports as the primary data source to drive a voice-based collaboration system is that the system can leverage complex data calculations already supported by the entire EMR system that generate the important information in the clinical report. Thus, in the context of pancreatic cancer, the example report in Figures 17A-17C already contains all the clinically important data, including the results of complex data operations, so the collaboration system does not need to independently derive the necessary data and other information.

[0262] In some cases, near the beginning of a collaboration session, when the collaboration system identifies a particular patient, the system identifies the patient's cancer condition and condition-specific clinical medical record and automatically loads a subset of intents associated with the patient's cancer condition for consideration (e.g., "condition-related intents"). In some cases, the condition-related intents may be the only intents considered by the system unless the oncologist indicates otherwise. In other cases, the condition-related intents may be prioritized (e.g., considered the first option or a more weighted option) over other, more general EMR-related intents, such that if the first and second intents in the pool of condition-related intents and more general intents are identified as possible intended intents, the system automatically selects the condition-related intents over the more general intents.

[0263] In at least some embodiments, data actions associated with condition-related intents are limited to associated clinical records. Thus, for example, referring again to Figures 17A-17C, if Dwayne Holder is identified as a pancreatic cancer patient and a query intent is identified, in these cases, data actions are limited to the data and information presented in the Figures 17A-17C records.

[0264] In other cases, data operations associated with condition-related intents may include any operations related to any EMR or other database data accessible by the system processor, in addition to direct operations on the data and information in the clinical reports FIGS. 17A-17C.

[0265] In still other cases, the cancer condition-specific intent may be treated as the preferred intent, and other more general dataset intents may be considered only if the system is unable to identify a condition-specific intent that matches the received query. Here, at least in some cases, even if a condition-specific intent is identified, the system may generate a confidence factor associated with the intent, and if the confidence factor is below a certain threshold level, other more general system intents may be considered as candidates that match the particular query.

[0266] Referring now to Figure 18, a process 400 similar to that described above with respect to Figure 5 is shown, except that the collaboration system automatically limits the intent to a particular cancer condition if a clinical report of the particular condition is available for the particular patient. Although process 400 is similar to the process of Figure 5, to simplify this description, some of the process steps of Figure 5 have been removed from process 400. For example, Figure 18 does not include, among other things, a step for providing a visual response to an oncology question. Nevertheless, it should be understood that in at least some embodiments of the present disclosure, any of the additional steps shown in Figure 5 may be added to process 400 of Figure 18.

[0267] 18, in an initial process step 402, an EMR or other system stores and maintains clinical reports for a particular patient and a particular cancer condition (e.g., pancreatic, breast, etc.). In block 404, an administrator uses the example cancer condition-specific clinical reports for each cancer condition to train an essentially complete set of condition-specific intents and other supporting information (e.g., parameters, data actions and response files or phrases).

[0268] After system training, the system monitors for collaboration device activation in block 406. Once the collaboration device is activated, the system monitors voice signals and collects voice signal queries uttered by the oncologist in decision block 408. Any received utterances are transcribed into text and stored in a text file in process block 412.

[0269] Still referring to FIG. 18, in decision block 414, the system processor monitors the utterance for information identifying a particular patient. If the oncologist does not identify a particular patient, the system may pass control to a process more similar to that shown in FIG. 5 to identify a more general query intent based on a larger data set. If a patient is identified by the oncologist in block 414, control passes to process block 416 where the patient's cancer status is identified in the system database. In block 418, the system determines whether there is a condition-specific clinical record stored in the system database for the user. If there is no condition-specific clinical record for the patient, control may also pass to the process shown in FIG. 5 to identify a more general query intent based on a larger data set.

[0270] In FIG. 18 , if the patient's condition-specific clinical record exists, control passes to block 420, where the system restricts the pool of intents to match the query to condition-related intents (e.g., intents that are specifically associated with the patient's condition-specific clinical record type). Again, in some cases, the restriction simply means that a weighting factor is applied to the intents, which makes it more likely that the system will select a condition-specific intent over a more general system intent. In other cases, the restriction means that the system will only consider general intents until the oncologist performs some activity that causes the system to identify a condition-specific intent.

[0271] In a particularly advantageous case, once a patient's general cancer condition (e.g., pancreatic, breast, etc.) is determined, the system tightly restricts the intent pool (e.g., not considering other intents during the query path or collaboration session) to match queries to the condition-specific clinical report set.

[0272] Subsequently, in block 422, the processor compares the received query to the limited intent set to identify the intent, and then extracts intent-related parameters from the query. In process block 424, the system uses the intent and the extracted parameters to define one or more data operations (e.g., primary, or primary and supplemental as described above) to be performed on the clinical report data, and at least in some cases, other accessible data sets. In block 426, the data operations are performed to generate information that can be used to respond to the query. In block 428, the response file associated with the intent and the data operation is used to create an audio response file, which is transmitted to the collaboration device and broadcast to the oncologist in block 430.

[0273] At least in some cases, it is contemplated that the system will support an email feature whereby an oncologist can request email copies of different clinical record datasets or other system datasets during a collaboration session. For example, after the system broadcasts information related to a clinical trial that may not be of interest to a particular patient, the oncologist may utter, "Please send me the information related to the trial." Here, the system recognizes the oncologist's intent to obtain an email containing the trial information for the trial in question, performs a data operation to access the trial information, and then transmits the information to the oncologist's email address. Additionally, once the trial information is transmitted via email, the system may generate and broadcast to the oncologist a response indicating that the trial information was sent via email. In other cases, it is contemplated that data and information may be transmitted to the oncologist via other communication systems (e.g., as a text link, via hard copy in regular mail, etc.). A more complex email-related dialogue path may include the following query: "Therapy Company" represents the name of one or more companies providing the therapy, and "Therapy" represents the name of one or more therapies. Dwayne Holder sequencing results. Does my patient have high TMB? Are they good candidates for immunotherapy? What immunotherapy drugs are currently approved? Who manufactures Therapy? What are the main adverse events of treatment? Please email me the Therapy medication label. Who manufactures Therapy? What is Therapy Company's patient financial assistance phone number? Please email me the Therapy Company Compassionate Use Agreement. Please email me the Tempus insurance reimbursement letter that my patient Dwayne Holder has data justifying their off-label use of Therapy.

[0274] In this example, the oncologist may issue several email requests, each of which will deliver a different set of information to the oncologist's email account.

[0275] In at least some cases, when the system receives a query via the collaboration device, data operations are performed on data from two or more different types of datasets. The first type may include a specific patient's genomic dataset with the molecular report details of the specific patient. The second data type includes data present in a general knowledge database (KDB) that includes non-patient specific information on a specific topic based on accepted industry standards (e.g., efficacy of a specific drug in treating a specific cancer condition, clinical trial information, drug class-mutation interactions, genes, etc.) or empirical information derived by the service provider, as well as information about the service provider's system capabilities (e.g., information about specific tests and activities performed by the provider, test requirements, etc.). To this end, see the exemplary system database 500 shown in FIG. 20, which includes a molecular report genomic dataset, and a clinical dataset 502, as well as a non-patient specific knowledge database (KDB) 504. By arranging the data operations in this manner, the universe of possible intents and data operations that can be associated with any query is proscribed as described above, providing the advantage of being associated with such an arrangement result.

[0276] Still referring to FIG. 20, the dataset 502 includes, among other data, genomic, transcriptomic, epigenomic, microbiome, clinical, stored alterations proteomic, proteomic, organoid, image and cohort, as well as trend datasets described in detail in other patent applications. The KDB, as illustrated, has separate sub-databases related to specific information types including provider panels 506 (e.g., information related to gene panels supported by the service provider operating the system), drug classes (e.g., drug class specific information (e.g., whether a particular class of drug works for pancreatic cancer or drugs considered to be in a particular drug class)), specific genes 508, immune results (e.g., information related to treatment based on the results of a particular immune biomarker), specific drugs, drug class-mutation interactions, mutation-drug interactions, provider methodology (e.g., questions related to the process performed by the service provider), clinical Subdatabases include: clinical conditions such as tests, immunology general, and clinical diseases; terminology sheets (e.g., definitions of industry-specific terms); provider coverage (e.g., information about the provider's tests and results); provider samples (e.g., information about the types of samples the provider can process); knowledge (e.g., scripted questions and answers for various frequently asked questions not categorized in other subdatabases); radiation (e.g., information about appropriate radiation treatments given a particular cancer condition); NCCN guidelines (e.g., national guidelines related to classification of cancer conditions, accepted treatments, etc.); and clinical trial questions-answers (e.g., information related to the location and administrators of clinical trials). Organizing the KDB into subdatabases makes it easier to manage those databases as the information in the KDB evolves over time, and also allows for the addition of new subdatabases related to other predefined information types.

[0277] To identify genomic datasets associated with a particular patient's molecular report, the system identifies data operations associated with the query and then associates at least one of those operations with the patient's genomic dataset displayed in the molecular report before performing at least one data operation on the set.

[0278] In at least some cases, the results of a data operation on the patient's molecular report data inform other data operations to perform on the KDB or inform other operations of the results of operations on the KDB to perform on the patient's molecular report data. For example, if an oncologist queries, "What is the impact of CDKN2A mutations on treatment for Dwayne Holder?" the system may associate that query with an intent. The intent may be associated with two data operations, including a first operation to search a general KDB for appropriate treatments for CDKN2A mutations and a second operation to determine whether the patient has already been treated with one or more of the appropriate treatments. In this case, the results of the KDB data operation inform the molecular report data operation. As another example, if an oncologist queries, "Did Dwayne Holder lose heterozygosity for his BRCA2 mutation?", the system again identifies two data operations, this time including a first operation against the genomic dataset associated with Dwayne Holder's molecular report to return the patient's loss of heterozygosity (LOH) value, and a second operation to perform against the KDB to determine whether the patient's mutation and LOH value pairing are known to be tumor drivers. In this case, the molecular report data operation results inform the KDB data operation.

[0279] Below, first and second exemplary processes are described, respectively, associated with processing the queries "What is the therapeutic impact of Dwayne Holder's CDKN2A mutation?" and "Did Dwayne Holder lose heterozygosity for his BRCA2 mutation?" To simplify this description, the first and second processes will be referred to as the first and second examples, respectively, unless otherwise noted.

[0280] 19, a process 450 is shown for associating data operations with a genomic dataset represented in a patient's molecular report prior to performing those operations on the dataset, consistent with at least some aspects of the present disclosure. At process block 452, collaboration device 20 (see again FIG. 1) receives an audible query from an oncologist via a device microphone related to information displayed in a particular patient's molecular report, which may be stored in a system database. In some embodiments, process 450 may store the particular patient's molecular report and / or other patients' molecular reports in a system database. In this manner, process 450 may store multiple patient's molecular reports. In some embodiments, process 450 may identify a particular patient, as described in connection with FIG. 18. At least in some cases, the audible query may include a question regarding a nucleotide profile associated with the patient. The nucleotide profile associated with the patient may be a cancer profile of the patient. The nucleotide profile associated with the patient may be a germline profile of the patient. The nucleotide profile associated with the patient may be a DNA profile. The nucleotide profile associated with the patient may be an RNA expression profile. The nucleotide profile associated with the patient may be a mutation biomarker. The nucleotide profile associated with the patient can be a BRCA biomarker. At least in some cases, the audible query can include a question regarding treatment. At least in some cases, the audible query can include a question regarding genetics. At least in some cases, the audible query can include a question regarding clinical data. The clinical data may include at least one of the clinical data elements described above. At least in some cases, the audible query can include a question regarding next generation sequencing panels. At least in some cases, the audible query can include a question regarding biomarkers. At least in some cases, the audible query can include a question regarding immune biomarkers.At least in some cases, the audible query may include a question regarding an antibody-based test. At least in some cases, the antibody-based test may be a blood sample-based antibody test. At least in some cases, the audible query may include a question regarding a clinical trial. At least in some cases, the audible query may include a question regarding an organoid assay. At least in some cases, the audible query may include a question regarding a pathology image. The pathology image may be a slide image, e.g., an image generated using whole slide imaging (WSI). At least in some cases, the audible query may include a question regarding a type of disease.

[0281] In some embodiments, at block 452, process 450 may identify at least one qualifying parameter in the audible query. In some cases, the at least one qualifying parameter may include a patient identity, a patient condition, a genetic mutation, and / or a type of treatment. In some embodiments, process 450 may identify qualifying parameters in a molecular report of the first patient.

[0282] At block 454, the system identifies at least one intent associated with the audible query. Here, block 454 involves identifying general intents as well as contextual parameters in the query so that a specific intent can be formulated. For example, for a first exemplary query, "What is the impact of CDKN2A mutations in Dwayne Holder on treatment?", the identified general intent may be "What is the impact of treatment based on the patient's genetic mutation?", the specific query parameters may include "CDKN2A" and "Dwayne Holder", and the underlined gene and patient fields in the general query are populated with "CDKN2A" and "Dwayne Holder" to generate the specific query intent.

[0283] For a second example query, “Did Dwayne Holder lose heterozygosity for his BRCA2 mutation?”, the identified general intent may be “Did the patient experience a genetic trait with a gene mutation?”, where the underlined patient, gene mutation, and gene fields in the general query are populated with “Dwayne Holder,” “heterozygosity,” and “BRCA2,” respectively, to generate the specific query intent.

[0284] At least in some cases, at least one intent can be associated with the audible query. At least in some cases, the at least one intent can be an intent related to a clinical trial. At least in some cases, the at least one intent can be related to a pharmaceutical. At least in some cases, if the intent is related to a pharmaceutical, the intent can be referred to as a pharmaceutical intent. At least in some cases, the pharmaceutical intent can be related to a pharmaceutical that is chemotherapy. At least in some cases, the pharmaceutical intent can be an intent related to a PARP inhibitor intent. At least in some cases, the at least one intent can be related to a gene. At least in some cases, the at least one intent can be related to immunology. At least in some cases, the at least one intent can be related to a knowledge database. At least in some cases, the at least one intent can be related to a testing methodology. At least in some cases, the at least one intent can be related to a gene panel. At least in some cases, the at least one intent can be related to a reporting. At least in some cases, the at least one intent can be related to an organoid process. At least in some cases, the at least one intent can be related to imaging. At least in some cases, the at least one intent may be associated with a pathogen. In some embodiments, the pathogen may be a pathogenic mutation. At least in some cases, the at least one intent may be associated with a vaccine.

[0285] The at least one intent can be associated with at least one activity. The at least one activity can include periodically capturing health information from an electronic health record for inclusion in the knowledge database. The at least one activity can include checking the status of an existing clinical or test order. The at least one activity can include ordering a new clinical or test. The at least one activity can include automatically initiating the at least one activity without initiating input from an oncologist. The at least one activity can include uploading a patient's EHR to the knowledge database.

[0286] Still referring to FIG. 19, once a particular intent is identified, in block 456 the system identifies at least one data action associated with the particular intent, where a database associates data actions with the intent. For example, in some cases, one or more data actions may be correlated with each particular intent. In other cases, at least some data actions may depend on the results of other data actions (e.g., a second action is executed only if the result of a first action is within a particular range of values).

[0287] In some embodiments, at block 456, process 450 may identify at least one data action based on both the identified intent and the at least one qualifying parameter.

[0288] For the first example, for the specific intent, "What is the impact of treatment based on Dwayne Holder's CDKN2A mutation?", the exemplary data operations may include: (1) for the CDKN2A mutation, searching for an appropriate treatment in the treatment KDB; and (2) for an appropriate treatment, searching the treatment history portion of the patient's molecular report genomic dataset to identify whether the patient has already received the appropriate treatment. Similarly, for the second example, for the specific intent, "Did Dwayne Holder experience loss of heterozygosity due to a BRCA2 mutation?", the exemplary data operations may include: (1) searching for an LOH value in the patient's molecular report genomic dataset and searching whether the mutation is germline or somatic; and (2) optionally searching a KDB (e.g., KDB 504) to determine whether the LOH value and the mutation are known to be tumor drivers based on the LOH value.

[0289] At least in some cases, the at least one data operation can include an operation for identifying at least one treatment option. At least in some cases, the at least one data operation can include an operation for identifying knowledge about a treatment. At least in some cases, the at least one data operation can include an operation for identifying knowledge related to at least one medicine. For example, the knowledge can be knowing what medicine, if any, is associated with high CD40 expression. At least in some cases, the at least one data operation can include an operation for identifying knowledge related to mutation testing. For example, the knowledge can be whether Dwayne Holder's sample was tested for a KMT2D mutation. At least in some cases, the at least one data operation can include an operation for identifying knowledge related to the presence of a mutation. For example, the knowledge can be whether Dwayne Holder has a KMT2C mutation. At least in some cases, the at least one data operation can include an operation for identifying knowledge related to a tumor characteristic. For example, the knowledge can be whether Dwayne Holder's tumor is a BRCA2 driven tumor. At least in some cases, the at least one data operation can include an operation for identifying knowledge related to a testing requirement. For example, the knowledge can be the percentage of tumors that Tempus requires for a TMB result. At least in some cases, the at least one data operation can include an operation for querying definition information. For example, the definition information can be a definition of PDL1 expression. At least in some cases, the at least one data operation can include an operation for querying expert information. For example, the expert information can include the clinical relevance of PDL1 expression or what are the general risks associated with Whipple surgery. At least in some cases, the at least one data operation can include an operation for identifying information related to a recommended treatment.For example, the information may be, "If Dwayne Holder is at the 88th percentile for PDL1 expression, is he a candidate for immunotherapy?" In at least some cases, the at least one data operation may include an operation for querying information related to the patient. In at least some cases, the at least one data operation may include an operation for querying information related to patients having one or more clinical characteristics similar to the patient. For example, the information may be, "What are the most common adverse events for patients similar to Dwayne Holder?" In at least some cases, the at least one data operation may include an operation for querying information related to a patient cohort. For example, the information may be, "What are the most common adverse events for patients with pancreatic cancer?" At least in some cases, the at least one data operation can include an operation to query for information related to clinical trials. For example, the information can be which clinical trial is best for Dwayne Holder. At least in some cases, the at least one data operation can include an operation to query for a feature related to a genomic mutation. At least in some cases, the feature can be loss of heterozygosity. At least in some cases, the feature can reflect a cause of the mutation. At least in some cases, the cause can be germline. At least in some cases, the cause can be somatic. At least in some cases, the feature can include whether the mutation is a tumor driver.

[0290] Referring again to FIG. 19, in block 458, the system associates each of the at least one data operation with a first dataset (i.e., a first set of data) presented in the molecular report for the particular patient. In a first example, the system associates each of the data operations with CDKN2A, which is displayed in the molecular report, as shown in FIG. 17A. In a second example, the system associates the first data operation with BRCA2 and Dwayne Holder in the molecular report genomic dataset. In some embodiments, the system can access the molecular report for the particular patient in block 458.

[0291] In at least some cases, the first set of data may be a gene editing therapy previously studied and / or documented by a reliable information source. In at least some cases, the gene editing therapy may be a clustered regularly interspaced short palindromic repeats (CRISPR) therapy. In at least some cases, the first set of data may include a patient name. In at least some cases, the first set of data may include a patient age. In at least some cases, the first set of data may include a next generation sequencing panel. In at least some cases, the first set of data may include a genomic variant. In at least some cases, the first set of data may include a somatic genomic variant. In at least some cases, the first set of data may include a germline genomic variant. In at least some cases, the first set of data may include a clinically actionable genomic variant. In at least some cases, the first set of data may include a loss of function variant. In at least some cases, the first set of data may include a gain of function variant. In at least some cases, the first set of data may include an immunological marker. In at least some cases, the first set of data may include a tumor mutation burden. In at least some cases, the first set of data may include a microsatellite instability status. At least in some cases, the first set of data can include a diagnosis. At least in some cases, the first set of data can include a treatment. At least in some cases, the first set of data can include a treatment approved by the U.S. Food and Drug Administration. At least in some cases, the first set of data can include a pharmaceutical therapy. At least in some cases, the first set of data can include a radiation therapy. At least in some cases, the first set of data can include a chemotherapy. At least in some cases, the first set of data can include a cancer vaccine therapy. At least in some cases, the first set of data can include an oncolytic virus therapy.In at least some cases, the first set of data can include immunotherapy. In at least some cases, the first set of data can include pembrolizumab therapy. In at least some cases, the first set of data can include CAR-T therapy. In at least some cases, the first set of data can include proton therapy. In at least some cases, the first set of data can include ultrasound therapy. In at least some cases, the first set of data can include surgery. In at least some cases, the first set of data can include hormone therapy. In at least some cases, the first set of data can include off-label use. In some aspects, the off-label use can include pharmaceutical therapy. In at least some cases, the first set of data can include on-label use. In at least some cases, the first set of data can include bone marrow transplant events. In at least some cases, the first set of data can include cryoablation events. In at least some cases, the first set of data can include radiofrequency ablation. In at least some cases, the first set of data can include monoclonal antibody therapy. In at least some cases, the first set of data can include angiogenesis inhibitors. In at least some cases, the first set of data can include PARP inhibitors. In at least some cases, the first set of data can include targeted therapy. In some embodiments, the targeted therapy may be a molecular targeted therapy. At least in some cases, the first set of data may include an indication of use. In some embodiments, the indication of use may be an indication of use of a pharmaceutical agent in treating a condition, such as a disease. At least in some cases, the first set of data may include a clinical trial. At least in some cases, the first set of data may include a distance to a location where a clinical trial is conducted. At least in some cases, the first set of data may include variants of unknown significance.In some embodiments, the variants may be classified as pathogenic, likely pathogenic, variants of unknown significance, likely benign, or benign. At least in some cases, the first set of data may include a mutation effect. In some embodiments, the mutation effect may be positive (e.g., associated with a decreased risk of heart disease), negative (e.g., associated with an increased risk of heart disease), or neutral (e.g., associated with no significant change in risk of heart disease). At least in some cases, the first set of data may include a mutant allele fraction. In some embodiments, the mutant allele fraction may be a ratio of mutant reads for a given mutation. At least in some cases, the first set of data may include low coverage regions. At least in some cases, the first set of data may include a medical history. At least in some cases, the first set of data may include a biopsy result. In some embodiments, the biopsy result may include a grade of aggressiveness of the cancer. For example, the grade may range from 1 to 4, with 1 indicating the least aggressive cancer and 4 indicating the most aggressive cancer. At least in some cases, the first set of data may include an imaging result. At least in some cases, the first set of data can include MRI results. At least in some cases, the first set of data can include CT results. At least in some cases, the first set of data can include a treatment prescription. At least in some cases, the first set of data can include a treatment administration. At least in some cases, the first set of data can include a cancer subtype diagnosis. At least in some cases, the first set of data can include a cancer subtype diagnosis by RNA class. At least in some cases, the first set of data can include the results of a treatment applied to the organoids grown from the patient's cells. At least in some cases, the first set of data can include a tumor quality measurement.In at least some cases, the first set of data can include a tumor quality measure selected from at least one of the following set: PD-L1, MMR, tumor infiltrating lymphocyte count, and tumor ploidy. In at least some cases, the first set of data can include a tumor quality measure obtained from image analysis of pathology slides of the patient's tumor. In at least some cases, the first set of data can include a signaling pathway associated with the patient's tumor. In at least some cases, the signaling pathway can be the HER pathway. In at least some cases, the signaling pathway can be the MAPK pathway. In at least some cases, the signaling pathway can be the MDM2-TP53 pathway. In at least some cases, the signaling pathway can be the PI3K pathway. In at least some cases, the signaling pathway can be the mTOR pathway.

[0292] In at least some cases, the at least one data operation can include an operation for querying treatment options, the first set of data can include a genomic variant, and the associating step (i.e., block 458) can include coordinating an operation for querying treatment options based on the genomic variant. In at least some cases, the at least one data operation can include an operation for querying medical history data, the first set of data can include a treatment, and the associating step (i.e., block 458) can include coordinating an operation for querying medical history data elements based on the treatment. In at least some cases, the medical history data can be a medication prescription, the treatment can be pembrolizumab, and the associating step can include coordinating an operation for querying a prescription for pembrolizumab.

[0293] Subsequently, in block 460, the system performs each of the data operations on the second dataset to generate response data. For the first example, a first data operation on the KDB (e.g., the second dataset) generates palbociclib as an appropriate treatment for the patient's CDKN2A mutation, and a second data operation on the molecular report genomic dataset (e.g., another second data operation) indicates that Dwayne Holder has already been treated with palbociclib. For the second example, the response data from the first data operation on Dwayne Holder's molecular report genomic dataset (e.g., the second dataset) indicates that he does not have a pathogenic somatic BRCA2 mutation, but also indicates that he has a pathogenic germline BRCA2 mutation and associated LOH loss (see the BRCA2 section of the molecular report shown at the bottom of FIG. 17B, which shows the LOH). In the second example, the first data operation results (e.g., the presence of a germline BRCA2 mutation and somatic LOH) are used to drive the second data operation, and the response data indicates that the tumor is a BRCA2-driven tumor.

[0294] In at least some cases, the second set of data can include clinical health information. In at least some cases, the second set of data can include genomic variant information. In at least some cases, the second set of data can include DNA sequencing information. In at least some cases, the second set of data can include RNA information. In at least some cases, the second set of data can include DNA sequencing information from short read sequencing. In at least some cases, the second set of data can include DNA sequencing information from long read sequencing. In at least some cases, the second set of data can include RNA transcriptome information. In at least some cases, the second set of data can include RNA full transcriptome information. In at least some cases, the second set of data can be stored in a single data repository. In at least some cases, the second set of data can be stored in multiple data repositories. In at least some cases, the second set of data can include clinical health information and genomic variant information. In at least some cases, the second set of data can include immunological marker information. In at least some cases, the second set of data can include microsatellite instability immunological marker information. In at least some cases, the second set of data may include tumor mutation burden immunological marker information. In at least some cases, the second set of data may include clinical health information including one or more of demographic information, diagnostic information, evaluation results, test results, prescribed or administered treatments, and outcome information. In at least some cases, the second set of data may include demographic information comprising one or more of patient age, patient date of birth, sex, race, ethnicity, system of care, comorbidities, and smoking history. In at least some cases, the second set of data may include diagnostic information including one or more of tissue of origin, date of initial diagnosis, histology, histology grade, metastatic diagnosis, date of metastatic diagnosis, one or more sites of metastasis, and stage information.At least in some cases, the second set of data can include stage information including one or more of TNM, ISS, DSS, FAB, RAI, and Binet. In some embodiments, the stage information may be referred to as "cancer stage information." At least in some cases, the second set of data can include assessment information including one or more of a performance status including at least an ECOG status or a Karnofsky status, a performance status score, and a performance status date. At least in some cases, the second set of data can include laboratory information comprising one or more of a lab type (e.g., CBS, CMP, PSA, CEA), a lab result, a lab unit, a date of lab service, a date of molecular pathology test, an assay type, an assay result (e.g., positive, negative, equivocal, mutation, wild type), a molecular pathology method (e.g., IHC, FISH, NGS), and a molecular pathology provider. In at least some cases, the second set of data can include treatment information including one or more of: drug name, drug start date, drug end date, drug dose, drug unit, number of drug cycles, type of surgical procedure, date of surgical procedure, radiation site, radiation modality, radiation start date, radiation end date, total dose of radiation administered, and total percentage of radiation administered. In at least some cases, the second set of data can include outcome information including one or more of: response to treatment (e.g., CR, PR, SD, PD), RECIST score, date of outcome, date of observation, date of progression, date of recurrence, adverse event to treatment, date of presentation of adverse event, grade of adverse event, date of death, date of last follow-up, and disease state at last follow-up. In at least some cases, the second set of data can include information that has been de-identified according to a de-identification method permitted by HIPAA. In at least some cases, the second set of data can include information that has been de-identified according to a Safe Harbor de-identification method permitted by HIPAA. In at least some cases, the second set of data may include information that has been de-identified in accordance with statistical de-identification methods permitted by HIPAA.In at least some cases, the second set of data may include clinical health information of patients diagnosed with a cancer condition. In at least some cases, the second set of data may include clinical health information of patients diagnosed with a cardiovascular disease. In at least some cases, the second set of data may include clinical health information of patients diagnosed with a diabetic condition. In at least some cases, the second set of data may include clinical health information of patients diagnosed with an autoimmune condition. In at least some cases, the second set of data may include clinical health information of patients diagnosed with a lupus condition. In at least some cases, the second set of data may include clinical health information of patients diagnosed with a psoriasis condition. In at least some cases, the second set of data may include clinical health information of patients diagnosed with a depression condition. In at least some cases, the second set of data may include clinical health information of patients diagnosed with a rare disease.

[0295] 19, at block 462, the system creates an appropriate audio response file, and at block 464, the response file is used to broadcast an audible response to the oncologist. In a first example, the system may generate a response that reads, "The provider has recommended Palbociclib, a CDK4 / 6 inhibitor based on Dwayne Holder's CDKN2A mutation. However, since he has already received this medication from September 20, 2017 to January 6, 2018, we recommend that you consider targeting one of his other clinically actionable mutations." In a second example, the system may generate a response that reads, "Dwayne Holder's results indicate a pathogenic germline BRCA2 mutation combined with somatic loss of heterozygosity, indicating that this may be a BRCA2-driven tumor."

[0296] It is recognized that many different query intents may take a similar format where the differences between the particular intents are defined by particular parameters. Similarly, many system responses to different queries may have a similar format where the differences between the particular responses are defined by the results produced by particular parameters in the query and / or data operations. For these reasons, in at least some embodiments, a dedicated user interface was developed to ease the burden on system administrators associated with specifying all possible system intents, contextual query parameters, data operations, and audio response files, as well as to manage that information as knowledge evolves over time. The interface generates sub-databases (see sub-databases in FIG. 20) that form the KDB shown in FIG. 20.

[0297] See FIG. 21, which illustrates generally an exemplary user interface screenshot 520 corresponding to the provider panel sub-database 506 shown in FIG. 20. In addition to presenting the provider panel data set, the screenshot includes individually selectable icons for each sub-database type in FIG. 20 so that an administrator can access any of those sub-databases via a screenshot similar to that shown in FIG. 21. Screenshot 520 includes a spreadsheet-type arrangement of row and column information cells used by the system to process queries and generate responses, as well as interface tools for scrolling up and down and left and right to access additional sub-database information. Although not shown, the exemplary interface also includes a keyboard, mouse device, and / or other input device for interacting with the interface (e.g., scrolling, modifying information, adding or deleting information, etc.).

[0298] Still referring to FIG. 21 , screenshot 520 includes query intents 522A-ZZZ arranged in a first row of cells, a separate intent in the top cell of each column in the first row. Intents often take the form of predefined queries to which a received query can be associated. An exemplary intent A shown is “Does provider $panel come with clinical data structuring?” where the “$panel” representation is a parameter collected from a query received from an oncologist. While only a few intents are shown, it should be understood that hundreds or more intents may be expressed and accessed through the interface. The $panel representations are referred to as parameter fields, and the system supports many parameter types with different parameter fields, and any intent may include two or more different parameter fields.

[0299] 21, the parameters that may be entered into the $panel parameter field in an intent are listed in cells located in the left column 524 of screenshot 520 and include xT, xE, and xF, and may include many other panel types. Thus, depending on the query received (e.g., does the query refer to an xT panel?), any of xT, xE, xF, etc. may be entered into the $panel field in intent A to define a panel-specific intent.

[0300] In the answers section 526 of the screenshot, an answer is provided for each intent and parameter combination. In general, the answers section includes a separate cell for each parameter row and intent column, and for each intent and parameter combination, a separate scripted answer may be provided in each of the answer cells. For example, for intent C and an xT panel, the answer in the associated answer cell 530 is "Yes, the xT panel contains matching regular sequencing."

[0301] If a general answer format is applicable to each parameter in column 524, an answer format may be provided where a particular parameter is to be used to fill in the parameter field in the answer format. For this purpose, see the answer formats in field 532 that require panel parameters in field $panel, where, during operation, the system retrieves the appropriate panel parameters from column 524 and populates field $panel as needed. Although not shown in FIG. 21, negative answer row 536 is also provided that may include negative answer formats for one or each of the intents listed in row 522.

[0302] Still referring to FIG. 21, an administrator can modify any intent, add intents, delete intents, modify parameters in column 524, add parameters, delete parameters, and / or modify answers by simply selecting an instance of the information to be modified and then entering the various information into the associated cells. In this manner, intents and answers of similar formats with various parameters can be quickly specified and managed with little overall effort. For example, in FIG. 21, assume that an interface specifies 200 different intents and that an administrator wants to add a new panel to the parameter options. Now, the administrator can associate all of the intents in row 522 with the new panel name simply by selecting another cell in the parameters column and naming the new panel. Furthermore, when a new panel is added to the panel column, for each answer format (e.g., see 532 again) that remains valid for the new panel, that answer format is automatically applied to the new panel.

[0303] 22, a second administrator interface screenshot 550 is shown having a similar format to the provider panel screenshot of FIG. 21 and therefore including an intent row 552, an answer section 554, and a parameters section 556. Each example intent includes a parameter field $Gene that is filled with one of the parameters from the parameters column 556 that form part of the received query.

[0304] In FIG. 22, answer section 554 differs from FIG. 21 because an "answer value" is provided for each answer cell (e.g., a cell corresponding to a particular intent column and parameter row combination) that is used in at least one, and possibly two, different ways. First, the answer in the answer cell corresponding to a particular intent and parameter combination can be used to select either answer format 551 or negative answer format 553. To this end, each of the answer formats and negative answer formats for each format includes a rule and a response format where the rule is applied based on the answer cell value. Thus, for example, for the answer format in cell 560, the rule is "IF TRUE" (e.g., if a TRUE value is in the answer cell), then apply the associated answer format. Similarly, for the negative answer format in cell 562, the rule is "IF FALSE" (e.g., if a FALSE value is in the answer cell), then apply the associated negative answer format. Thus, for example, answer cell 570 contains gene ABCB1 and intent A value TRUE, so the answer format in cell 560 is applied and the response file contains the phrase "Yes, the provider will sequence ABCB1." Similarly, answer cell 572 contains gene ABCB4 and intent A value false, so the negative answer format in cell 562 is applied and the response file contains the phrase "No, the provider will not sequence ABCB4."

[0305] Second, at least in some cases, the value of the answer cell may also be used to populate one or more fields in an answer format or a negative answer format. To this end, see, for example, the answer format in cell 576, which in addition to including a $Gene field also includes an $AV (e.g., answer value) field. Now, if an answer format rule is satisfied (e.g., IF AV, there is an answer value in the answer cell) such that answer format 576 is used to generate a response file, then in addition to populating the $Gene field with one of the genes in column 556, the $AV field is populated with a value from the associated answer cell below it. For example, for gene ABCB1, answer cell 578 contains the value 1%, so that when intent C is applied and qualified by gene parameter ABCB1, the answer format rule in cell 576 is satisfied and the response tile contains the phrase "The donor has a pathogenic mutation in ABCB1 in 1% of pancreatic cancer patients." In negative answer cell 580, there is a rule that if the answer cell below it is blank, then that cell format is used to generate a response file.

[0306] Although there are two answer format rows shown in each of Figures 21 and 10 (e.g., an answer format row and a negative answer format row), in other cases there may be three or more answer formats that change based on the value of the particular answer field below them to support more complex answer generation schemes.

[0307] Again, as with the data presented in Figure 21, the data in Figure 22 shows only a small subset of the gene data that can be accessed left-right and up-down by scrolling up and down through the parameters and intents. For example, the genes in parameter column 556 may include entire gene panels (e.g., hundreds of genes), and the intents in row 552 may include hundreds or thousands of intents.

[0308] FIG. 23 illustrates another administration screenshot 600 that corresponds to a provider method data set, but is similar to the screenshots of FIG. 21 and FIG. 22. The spreadsheet representation in FIG. 23 is similar to the representations in FIG. 21 and FIG. 22, including an intent row 602, an answer format section 610, and a parameter column 604. One difference in FIG. 23 is that the first intent A includes two parameter fields, and the parameter section includes first and second parameter rows, one for each of the parameter fields in intent A. More specifically, the parameter section includes a first column that lists tests and a second column that lists test methods for populating the associated $test and $testmethod fields in the intent statement. Additionally, at least in some cases, an answer format, such as the negative answer format shown in cell 606, includes more than one parameter or value field. The operation here is similar to that described above, although it uses two parameters to instantiate a particular intent and final response file.

[0309] Referring again to Figure 20, interface screenshots similar to those described in Figures 21-23 are included in a system for specifying intents, parameters, and answer formats for each information type associated with the sub-databases shown. Some screenshots include specific scripted answers for specific intents, while others rely on answer formats, one or all of the formatting rules, and entering intent parameters and / or database values ​​into answer fields that are displayed in answer cells as described above. Other combinations of screenshots and tools are possible.

[0310] At least in some cases, it is contemplated that the system will enable the oncologist to request visual access to query answers and / or related information (e.g., relevant documentation (e.g., clinical trial information, drug label warnings, etc.)). For example, the oncologist may say, "Make that answer available on the system web platform," and the system may make the latest broadcast answer available via a nearby or dedicated computer display screen for the oncologist. At least in some cases, it is contemplated that the system will enable the oncologist or other user to provide a query via a typed question in lieu of an audible query. For example, rather than speaking a question, the oncologist may type the query into a mobile phone or other computing device, and the query may be processed as described herein.

[0311] 24, a fourth exemplary system 650 is shown including a mobile device 652. The fourth exemplary system may include a collaboration device 20, a collaboration server 12, an AI provider server 14, and a database 18. The collaboration device 20, the collaboration server 12, the AI ​​provider server 14, and the database 18 may be linked together as described above in conjunction with FIG. 1. The mobile device 652 may be used in combination with the collaboration device 20 to authenticate and / or onboard user credentials and perform at least some of the functions of the collaboration device 20 (e.g., process queries regarding patients), among other suitable uses.

[0312] The mobile device 652 can be a smartphone, tablet, or another suitable mobile computing device. The mobile device 652 can include a camera 653, a speaker 654, a fingerprint sensor 656, and input buttons 658, as well as a touch screen 660. Similar to the collaboration device 20, the mobile device 652 can transmit (666) a voice signal message to the transceiver 16 for processing by the collaboration server 12 and / or the AI ​​provider server 14. The mobile device 652 can also receive (662) a visual response file and / or receive (664) an audio response file generated based on the voice message signal from the transceiver 16.

[0313] Further, the mobile device 652 may transmit authentication information to the transceiver 16 to unlock the collaboration device 20 (668). The collaboration device 20 may be configured to request authentication from the oncologist at predefined times (e.g., every 30 minutes, every hour, etc.) or when the oncologist moves the collaboration device 20. For example, the collaboration device 20 may detect that it has been moved if contact with the transceiver 16 is lost. The oncologist may move the collaboration device to another room in the same building (e.g., hospital) or to another building entirely (e.g., another hospital, home office, etc.). It is understood that the collaboration device 20 is mobile and can be moved and used in various locations with appropriate connectivity (e.g., wireless Internet).

[0314] The mobile device 652 may install a mobile device application (not shown) that can determine which authentication credentials are needed at a particular time and, if applicable, display a notification regarding the needed authentication credentials on the touch screen 660. For example, if the mobile device application determines that 30 minutes have passed since the last authentication, the mobile device application may output a notification that the oncologist needs to re-authenticate before using the full functionality of the collaboration device 20 (e.g., querying the collaboration device 20 for a particular patient). In some embodiments, the mobile device application may not output a notification and the collaboration device 20 may prompt the oncologist to re-authenticate when the oncologist attempts to query the collaboration device.

[0315] The mobile device 652 can provide multiple forms of authentication information to the transceiver (668). The authentication information can include a fingerprint scan generated using the fingerprint sensor 656, a photograph of the oncologist's face generated using the camera 653, and / or a text password. In some embodiments, the mobile device application can provide the raw fingerprint scan, image, and / or password to the transceiver 16, and another process (e.g., a process within the collaboration server 12) can determine whether the authentication information is sufficient (e.g., whether the fingerprint scan sufficiently matches a predefined fingerprint scan associated with the oncologist). In other embodiments, the mobile device application can determine whether the authentication information is sufficient and transmit authentication information indicating whether the authentication information is sufficient (e.g., a Boolean yes / no) (668). If the authentication information is sufficient, the collaboration device 20 can resume full operation.

[0316] The mobile device 652 may also transmit user requests (i.e., oncologist requests) to the transceiver for processing by the collaboration server 12 (669). A user request may be a note, a clinical report, a molecular report, a patient, a case, etc., a product suggestion (e.g., a feature of the collaboration device 20 that the oncologist would like to add), a product fulfillment request (e.g., an order for a test kit), the status of an ordered test kit (e.g., a liquid or tissue-based biopsy test kit to perform molecular testing), a recommendation for a tumor board session for the patient, or other requests that may be more easily disseminated by a human than a computer process, or a request for a human (e.g., an administrator, or possibly a medical practitioner) to review the request that is not necessarily related to the fulfillment of an intent. Some user requests may be generated and transmitted by the mobile device 652 and / or the collaboration device 20 based on voice signals captured from the oncologist. The AI ​​database 14 may determine the intent of the user request, for example, being an order for a test kit or a request for manual review of a particular case. The text form of the speech signal and any associated information (e.g., intent) may then be transmitted to collaboration server 12, which may transmit the text form of the speech signal and any associated information to an administrator and / or appropriate computer process. If the user's request includes a recommendation for a patient for a tumor board or clinical trial, the associated information about the patient may also be transmitted along with the request, greatly reducing the need to fill out application forms for clinical trials and / or tumor boards.

[0317] In some embodiments, processes performed by the mobile device 652 (eg, authentication processes, transmitting a user request (669), etc.) may be performed by the collaboration device 20.

[0318] 24 and 25, a mobile application screenshot 700 is displayed. The mobile application screenshot 700 can be part of a mobile device application included in the mobile device 652. The mobile application screenshot 700 can include a battery level indicator 702 indicating the battery level of the collaboration device 20, a username 703 corresponding to the current oncologist who is logged in, an authentication indicator 704 indicating whether the oncologist has been authenticated by the mobile device 652, a microphone button 706, a night mode button 708, and a mute button 710. The oncologist can select the microphone button 706 instead of speaking a wake-up word or phrase (e.g., "Tempus ONE") to prompt the collaboration device 20 and / or the mobile device 652 to record an audio signal. The night mode button 708 can control the darkness and / or color displayed by the mobile device application.

[0319] The mobile application screenshot 700 may include a slider 712. The oncologist may activate the slider 712 to control the volume of the collaboration device 20. The mobile application screenshot 700 may include a suggested questions section 714 that may display example queries and / or general queries for the oncologist to ask the collaboration device 20. For example, a first question 716 may show the oncologist how to ask a question about a particular patient, and a second question 718 may show the oncologist how to ask a medical question that is not patient-specific.

[0320] 24 and 25, and 26, a screenshot 720 of a second mobile application is shown. The screenshot 720 of the second mobile application can include the suggested questions section 714, the first question 716, and the second question 718 included in the screenshot 700 of the mobile application shown in FIG. 25. In FIG. 26, the suggested questions section 714 is shown to include additional suggested questions. The screenshot 720 of the second mobile application can include a suggest new feature button 722. An oncologist can, for example, select the suggest new feature button 722 and provide a suggestion for a new feature in a pop-up box. The mobile device 652 can then transmit the suggestion to the administrator.

[0321] The second mobile application screenshot 720 may include a frequently asked questions (FAQ) section 724 that includes common questions regarding the functionality of the collaboration device 20. The second mobile application screenshot 720, and more specifically, the FAQ section 724, may include a search button 726 that the oncologist can select to search a set of FAQs.

[0322] 25 as well as 27, a third mobile application screenshot 730 is shown. The third mobile application screenshot may include an answer 732 to the second question 718 shown in FIG 25. The answer 732 may include text and may be included in a pop-up box that is displayed when the oncologist selects the second question 718.

[0323] 28, a fifth exemplary collaboration system 750 is illustrated. The fifth exemplary system 750 can include an administrator device 752, such as a laptop or desktop computer. An administrator can use the administrator device 752 to analyze data aggregated from multiple oncologists, update firmware in the collaboration device 20, analyze requests from oncologists, update sets of intents (e.g., in Dialogflow), and other suitable tasks related to the operation 652 of the collaboration device and / or mobile device.

[0324] The fifth exemplary system 750 may include a cloud architecture 754 including several modules that may be located remotely (e.g., on one or more servers) relative to the administrator device 752, the collaboration device 20, and / or the mobile device 652. The collaboration device 20 and / or the mobile device 652 may be linked to an IoT core module 758 that may process authentication requests and other communications from the collaboration device 20. The IoT core module 758 may be linked to a pub / sub-module 760 that is linked to an authentication module 762 and a ping module 764. The pub / sub-module 760 may transmit updates (e.g., status updates) to the collaboration device 20 and / or the mobile device. The authentication module 762 may receive authentication requests from the collaboration device 20 and / or the mobile device 652. The pub / sub-module 760 may direct communications from the IoT core module 758 to either the authentication module 762 or the ping module 764, as appropriate. The pub / sub-module 760 , the IoT core module 758 , the authentication module 762 , and / or the ping module 764 may be stored on the first server 756 .

[0325] The collaboration device 20 and / or the mobile device 652 can be linked to a gateway module 768 that may be included in the second server 766. The gateway module 768 can include at least some of the processes included in the collaboration server 12. The collaboration device 20 and / or the mobile device 652 can transmit requests (e.g., user requests transmitted from the mobile device 652) and / or voice signal messages to the gateway module 768. The gateway module 768 can be linked to an AI module 774. The AI ​​module 774 can include at least some of the processes included in the AI ​​provider server 14 (e.g., a voice signal extraction process) and can receive voice signals, extract intents from the voice signals, and transmit data responses to the gateway module 768 for transmission to at least one of the collaboration device 20 and the mobile device 652. In some embodiments, the Dialogflow suite can be included in the AI ​​module 774. The gateway module 768 can also be linked to a debug bucket module 778 and a redis module 780 included in the third server 776. The gateway module 768 can be linked to an AI demo module 772 included in a fourth server 770 .

[0326] The third server 776 may be fully accessible only by the administrator device 752 (e.g., configured to allow full control and / or modification of the process) and not accessible by the collaboration device 20 and / or the mobile device 652. The third server 776 may include an administrator module 782 accessible by the administrator device 752 to update firmware of the collaborator device 20, define intents, update intent fulfillment processes, and perform other administrator functions. The administrator module 782 may also process and / or transmit user requests (e.g., orders for test kits) to the administrator device 752. The administrator and / or administrator device 752 may then analyze the user request and proceed accordingly. For example, the order for test kits may be transmitted to an order fulfillment center. As another example, an oncologist may request a manual review of a particular case, which may be transmitted to an administrator, who may assign the case to a practitioner for review within a predefined time frame, e.g., within 24 hours.

[0327] The third server 776 may include a console module 786 linked to the administrator device 752. The console module 786 may perform administrative tasks. An administrator database 784 may be linked to the administrator module 782. The console module 786 may be linked to a portal module 790 included in a fifth server 788. The portal module 790 may provide an interface for oncologists to review molecular test reports.

[0328] The fifth system 750, and more specifically, the administrator module 782, administrator device 752, console module 786, and / or administrator database 784, may track a number of collaboration devices 20. More specifically, the fifth system 750 may track relatively static information such as whether each collaboration device 20 is connected (e.g., in contact with the cloud architecture 754), active (e.g., processing queries), the version of firmware each collaboration device 20 is running, and the owner and / or institution associated with the device.

[0329] The administrator module 782 may include processes for analyzing queries from oncologists and generate usage data regarding how oncologists are using the collaboration device 20, how many test kits are ordered for different case types, how frequently questions regarding specific sections of generated clinical reports are asked regarding FDA on / off label drug questions, treatments associated with specific variants, actions taken by oncologists in different scenarios (e.g., what questions are being asked), and other suitable data.

[0330] It is understood that servers 756, 766, 770, 776, and 788 can each include multiple servers. Additionally, at least some of the modules and / or processes included in cloud architecture 754 can be implemented using infrastructure-as-code that can be migrated between clouds such as AWS, Google Cloud, Azure, etc.

[0331] The fifth system 750, and more specifically, the administrator module 782, administrator device 752, console module 786, and / or administrator database 784, can track intent being processed across a number of cases (e.g., thousands of cases) and / or other actions being taken by one or more collaboration devices 20, or developments occurring in the medical community (e.g., new research articles, studies, and / or treatment techniques) to provide "nudges" to the oncologist to potentially make the oncologist aware of information that the oncologist may not be aware of. The fifth system 750 can ask the oncologist for permission to analyze clinical data generated by the oncologist.

[0332] Other data that the administrator module 782, administrator device 752, console module 786, and / or administrator database 784 may track include the number of test kits ordered by an oncologist in a predefined time frame (which may indicate whether onboarding was successful), answers (e.g., statistics) or other information (e.g., Tempus) provided by the collaboration device 20 that is not included in a clinical report. may include the number of oncologists with XXX molecular mutations, XXX mutations, actionable mutations, etc.), regularly scheduled surveys of oncologists on the collaboration device 20 and / or mobile application (e.g., answers to "What information would be most helpful in making a clinical decision?"), which parts of the clinical report are most frequently asked about (either to an individual oncologist or to multiple oncologists), how oncologists behave on a macro scale (e.g., what tests have other oncologists performed for a given cancer type and / or molecular mutation), how similar patients perform on specific therapies (e.g., how many patients who presented with a XXX molecular mutation also had variant YYY and how many of those had a ZZZ response to therapy AAA over the course of the BBB), or other suitable data.

[0333] In some embodiments, the administrator device 752 may provide at least some of the functionality of the collaboration device 20, but tailored for the administrator. For example, the administrator device 752 may be appropriately equipped (e.g., with a microphone and speaker) and configured to answer questions such as "Where is sample[x] stored," "What is the SOP for scenario y," which may be relevant only to the administrator. In some embodiments, the administrator may use the collaboration device 20 with a set of administrator-specific intents that may not be available to oncologists (i.e., of limited use). The administrator may say, for example, "Tempus ONE, where is sample[x] stored?" and the collaboration device can determine that the intent of the query is to know the location (e.g., warehouse) of sample[x] and provide an appropriate visual and / or aural response.

[0334] Some data can then be used to customize the user experience for each oncologist. For example, data regarding which parts of the clinical report are most asked about can be used to customize the report layout and format suggestions that the oncologist can accept (i.e., update the report layout and / or format) or reject (i.e., keep the same report layout and / or format) after receiving the notification. The report displayed via the portal module 790 or mobile application (e.g., on the touch screen 660) can then follow the updated template and / or layout. Additionally or alternatively, the oncologist can provide suggestions regarding the report layout and / or format, and the report layout and / or format can be updated accordingly.

[0335] The fifth system 750 can use data collected by the fifth system 750 to provide nudges to the oncologist using the collaboration device 20 and / or the mobile device 652. The nudges can be provided to the oncologist without the oncologist having to ask. One nudge can include the fifth system 750 determining the most successful treatment for a patient similar to the one the oncologist is analyzing. The most successful treatment can be determined based on the patient's molecular data (e.g., molecular mutations and / or variants), age, gender, etc., as well as the success rates of various treatments in populations with the same molecular data, age, gender, etc. Another nudge can include informing the oncologist about cancer boards, clinical trials, and other programs within a predefined radius (e.g., 15 miles from the medical facility where the oncologist is located) for which the patient is eligible. Furthermore, the fifth system 750 can provide a nudge to the oncologist at a predefined time (e.g., within 24 hours) before the next patient visit, and can provide only the cancer boards, clinical trials, and other programs that were not available when the patient last visited and / or the last clinical report was generated for the patient. The oncologist can be notified by controlling the indicator light 50 with a predefined pattern and / or color, outputting a specific sound on the speaker 44, displaying a notification on the mobile device 652, vibrating the collaboration device 20 and / or the mobile device 652 using the haptic signaling component, etc. Furthermore, the findings of the tumor board and / or exam and / or action plan can be provided to the oncologist or a specific action plan following the tumor board can be cited. The oncologist can then easily retrieve the findings of the specific tumor board.

[0336] Further nudges may include controlling the indicator light 50 in a predetermined pattern and / or color to indicate that a new molecular or clinical report is available to the patient, outputting a particular sound on the speaker 44, displaying a notification on the mobile device 652, vibrating the collaboration device 20 and / or the mobile device 652 using a haptic signaling component, etc.

[0337] Yet another nudge can include informing the oncologist of newly available content (e.g., research papers, articles, journals, posters, etc.) relevant to the oncologist's practice area or patients. The oncologist can select notifications associated with multiple data sources, content types, cancer subtypes and / or diseases, molecular mutations / variants, treatments (on-label, off-label, investigational, etc. FDA), and clinical trials. A further nudge can include informing the oncologist that an ordered test may be completed more efficiently using an alternative test (e.g., using an xF liquid biopsy test instead of a tissue-specific xT panel). Although the oncologist may not be able to process the test kit due to insufficient tissue, the oncologist sees only the newly proposed test.

[0338] Further nudges can include informing the oncologist of various patient tests and orders that other oncologists have placed for similar patients and / or cases. For example, a nudge can include a notification that a peer oncologist (or x% of other oncologists) has placed a test order for a similar patient. In some embodiments, the fifth system 750 can determine if the oncologist has ordered PDL1 IHC for a particular test and inform the oncologist if that test is a good option for the patient. An oncologist may not want to use a new vendor initially and may want to consult with a colleague to better understand the type of information returned via the PDF report through the port AI module 790 or a mobile application on the mobile device 652. Knowing that other oncologist peers in the facility have ordered tests for x% of a particular patient cohort allows the oncologist to operate knowing how other oncologists are treating similar patients.

[0339] Still with respect to test kits, yet another nudge may include notifying the oncologist that test kit inventory is low (e.g., below a predefined threshold). Some nudges may include information regarding possible financial assistance available for the test kit. Once the oncologist is notified of the test kit options, the oncologist may order the test kit and / or apply for financial assistance by uttering appropriate commands on the collaboration device 20 and / or the mobile device 652. The fifth system 750 may then automatically populate the test kit order form and / or the financial assistance application.

[0340] Some nudges could inform oncologists of nearby continuing medical education (CME) courses and / or allow oncologists to enroll in CME credit courses or highlight local and / or online services within a particular specialty and / or area of ​​focus.

[0341] 19, 24, and 28, and 29, a process 1000 for generating supplemental content for a physician based on molecular reports related to a particular patient is shown. The process 1000 can be used to provide patient-specific nudges to an oncologist. The process 1000 can identify information that may be relevant to the patient's treatment and that the oncologist may not have considered when querying the collaboration device 20. In this manner, the collaboration device 20 may assist the oncologist in treating the patient with treatments, medicines, clinical tests, and / or other treatment techniques applicable to the particular patient that the oncologist may not have been aware of or may not have previously considered. The process 1000 may also provide the oncologist with information about how other oncologists have treated similar patients (e.g., diagnosed with similar genomic and / or similar cancer types). The process 1000 may be performed by a suitable system, such as the fifth exemplary system 750.

[0342] At 1002, process 1000 may determine a particular patient. At least in some cases, process 1000 may be performed in parallel and / or after process 450 has been performed and / or after process 450 has finished executing. Process 1000 may determine that the particular patient is the same particular patient identified by process 450. At least in some cases, process 1000 may be performed together with process 1000 to effectively form a single process. Process 1000 may then proceed to 1004.

[0343] At 1004, the process 1000 can store and maintain a general cancer knowledge database. The general cancer knowledge database can include raw and / or processed data on a large number of patients, including molecular reports, the presence of conditions such as diabetes, heart disease, etc., information about treatment history such as medications and / or treatments taken by each patient, and response to medications and / or treatments (e.g., the patient was successfully treated using the drug FFF), and / or other suitable data about the patient. The data associated with each patient can be permanently updated as additional information becomes available. The general cancer knowledge database can include non-patient specific information on a particular topic based on accepted industry standards (e.g., efficacy of a particular drug in treating a particular cancer condition, clinical trial information, drug class-mutation interactions, genes, etc.) or empirical information derived by the service provider, as well as information about the system capabilities of the service provider (e.g., information about specific tests and activities performed by the provider, testing requirements, etc.). The general cancer knowledge database can include the KDB 504 described above. The general cancer knowledge database can include information about available clinical trials, treatments, studies, academic papers, CLE courses, or other available resources. The process 1000 can then proceed to 1006.

[0344] At 1006, the process 1000 can persistently update the molecular report for a particular patient. For example, the process 1000 can update the associated clinical trials included in the molecular report. The process 1000 can then proceed to 1008.

[0345] At 1008, the process 1000 can automatically identify at least one intent and associated data action related to a general cancer knowledge database based on the molecular report data of the particular patient. The at least one intent can be related to a medicine, a gene, a test method, etc., as described above. The at least one intent can also be related to a particular cancer that the particular patient has been diagnosed with. At least some of the intents may be intents that the oncologist has not previously queried the collaboration device 20 about. The process 1000 can then proceed to 1010.

[0346] At 1010, the process 1000 can persistently perform associated data operations on the general cancer knowledge database to generate a new set of response data not previously generated. In some cases, the process 1000 can persistently perform multiple associated data operations on the general cancer knowledge database. Persistently executing the general cancer knowledge database allows the process 1000 to provide updated information (i.e., a new set of response data) to the oncologist. Additionally, the new set of response data may be used to provide information relevant to a particular patient that the oncologist may not have been aware of before. For example, the new set of response data can be used to inform the oncologist how various treatment options will perform for other patients with a similar genomic profile. As another example, the new set of data can be used to inform the oncologist of tests ordered for other patients diagnosed with the same cancer as the particular patient and with a similar genomic profile (such as the presence of a particular gene mutation). The process 1000 can then proceed to 1012.

[0347] At 1012, upon generating a new set of response data, the process 1000 can use the new set of response data to generate a notification for output to the oncologist. In some cases, the notification can be an audible response file that the process 1000 generates based on the new set of response data. In some cases, the notification can be a visual indicator that the process 1000 generates based on the new set of response data. The visual indicator can include a question related to the new set of response data. For example, the question can be a suggested question that can be answered using the new set of response data. In this example, if the new set of response data includes information regarding a patient's response to a particular treatment (e.g., patients with mutation XXX treated with YYY had a VVV% chance of surviving cancer type WWW), the suggested question can be "How many patients with mutation XXX survived when treated with YYY?" or "What is the most effective treatment for patients with mutation XXX and cancer type WWW?". The process 1000 can then proceed to 1014.

[0348] At 1014, the process 1000 may output the notification generated at 1012 to the oncologist. If the notification is an audible response file, the process may output the audible response file at the collaboration device 20 (e.g., speaker 44) and / or the mobile device 652 (e.g., speaker 654). If the notification is a visual indicator, the visual indicator may be output at the collaboration device 20 (e.g., display screen 48) and / or the mobile device 652 (e.g., touch screen 660). If the visual indicator is a suggested question, the suggested question may be displayed in the suggested questions section 714 above. The notification may function as a nudge. Thus, the process 1000 may generate and provide at least some of the above nudges to the oncologist.

[0349] 19, 24, and 28, and 30, a process 1050 for generating non-patient-specific supplemental content for physicians is shown. The process 1050 can be used to provide non-patient-specific nudges to oncologists. The process 1050 can identify information that may be generally relevant to oncologists, such as newly available treatments, research, and academic papers. The process 1050 can reduce the need for oncologists to search for new developments in the field in which the oncologist practices. For example, if an oncologist specializes in treating breast cancer patients, the process 1050 may provide information that may be useful in treating breast cancer patients. The process 1050 may be performed by a suitable system, such as the fifth exemplary system 750.

[0350] At 1052, process 1050 can determine one or more streams of interest for the oncologist. The streams of interest can include newly available clinical trials, treatments, studies, journal articles, CLE courses, or other suitable types of information and / or programs related to the cancer type that may be useful to the oncologist. In some embodiments, the oncologist can provide (e.g., audibly) the stream types of interest and / or the cancer type of interest to process 1050. Process 1050 may automatically determine the streams of interest based on the oncologist's history. For example, an oncologist may generally treat patients with breast and lung cancer, and process 1050 can select available streams of interest related to those cancer types. Process 1050 can then proceed to 1054.

[0351] At 1054, the process 1050 can store and maintain a general cancer knowledge database. The general cancer knowledge database can include raw and / or processed data on a large number of patients, including molecular reports, the presence of conditions such as diabetes, heart disease, etc., information about treatment history such as medications and / or treatments taken by each patient, and response to medications and / or treatments (e.g., the patient was successfully treated using the drug FFF), and / or other suitable data about the patient. The data associated with each patient can be permanently updated as additional information becomes available. The general cancer knowledge database can include non-patient specific information on a particular topic based on accepted industry standards (e.g., efficacy of a particular drug in treating a particular cancer condition, clinical trial information, drug class-mutation interactions, genes, etc.) or empirical information derived by the service provider, as well as information about the system capabilities of the service provider (e.g., information about specific tests and activities performed by the provider, testing requirements, etc.). The general cancer knowledge database can include the KDB 504 described above. The general cancer knowledge database can include information about available clinical trials, treatments, studies, academic papers, CLE courses, or other available resources. The process 1050 can then proceed to 1056.

[0352] At 1056, the process 1050 can automatically identify at least one intent and associated data action related to the general cancer knowledge database based on the flow of interest associated with the oncologist. For example, the at least one intent may relate to identifying whether a new journal article is newly available (e.g., published last week) on a particular type of cancer (e.g., breast cancer), identifying whether a new clinical trial is newly available for a particular type of cancer (e.g., lung cancer), identifying whether a new treatment option is newly available for a particular type of cancer (e.g., breast cancer, etc.), or other suitable intent. In this example, the associated data action may include searching for new available journal articles, clinical trials, and treatment options. At least some of the intents may be intents that the oncologist has not previously queried on the collaboration device 20. The process 1050 can then proceed to 1058.

[0353] At 1058, the process 1050 can persistently perform associated data operations on the general cancer knowledge database to generate a new set of response data not previously generated. In some cases, the process 1050 can persistently perform multiple associated data operations on the general cancer knowledge database. By persistently running the general cancer knowledge database, the process 1050 can provide updated information (i.e., a new set of response data) to the oncologist. The new set of data can be used to inform the oncologist of newly available journal articles, clinical trials, available treatment options, and the like. The process 1050 can then proceed to 1060.

[0354] At 1060, upon generating a new set of response data, the process 1050 can use the new set of response data to generate a notification for output to the oncologist. In some cases, the notification can be an audible response file that the process 1050 generates based on the new set of response data. In some cases, the notification can be a visual indicator that the process 1050 generates based on the new set of response data. The visual indicator can include a question related to the new set of response data. For example, the question can be a suggested question that can be answered using the new set of response data. In this example, if the new set of response data includes information about a newly available breast cancer treatment (e.g., treatment YYY is now available for breast cancer patients), the suggested question can be "Are there any new treatment options available for breast cancer patients?" The process 1050 can then proceed to 1062.

[0355] At 1062, the process 1050 may output the notification generated at 1060 to the oncologist. If the notification is an audible response file, the process may output the audible response file at the collaboration device 20 (e.g., speaker 44) and / or the mobile device 652 (e.g., speaker 654). If the notification is a visual indicator, the visual indicator may be output at the collaboration device 20 (e.g., display screen 48) and / or the mobile device 652 (e.g., touch screen 660). If the visual indicator is a suggested question, the suggested question may be displayed in suggested questions section 714 above.

[0356] 31, a process 800 that may be used for onboarding an oncologist is shown. At block 802, the process 800 may determine that a user (e.g., an oncologist) has opened a mobile application and that the collaboration device is turned on. The mobile application may be a mobile application included in the mobile device 652, and the collaboration device may be the collaboration device 20 described above. The collaboration device 20 may output, using the speaker 44, "Hello, your Tempus ONE is ready for setup. Download the Tempus ONE mobile app to begin setup." Control is passed to block 804, where the process 800 may display an option to confirm high level instructions to the oncologist. The option may be displayed on a user interface, such as the touch screen 660. Control is passed to block 806, where the process 800 may determine whether the oncologist has logged into the mobile application.

[0357] Once the oncologist has logged in, control is passed to block 808, where the process 800 may attempt to log the mobile device 652 into the wireless network to which the collaboration device is connected. After the mobile device 652 has logged into the wireless network, control is passed to block 810, where the process 800 displays an option to configure security settings to the oncologist in a user interface. Control is passed to block 812, where if the oncologist selects the option to configure security settings (i.e., “yes” at block 812), the process 800 proceeds to block 814. If the oncologist does not select the option to configure security settings (i.e., “no” at block 812), control is passed to block 816. At block 814, the process 800 may configure security settings of the mobile device 652 and / or the collaboration device 20. For example, the process 800 may set the oncologist's authentication preferences (e.g., fingerprint preferences, face recognition preferences, or entered password preferences).

[0358] Flow then passes to block 816, where process 800 may display an option to open an educational module to the oncologist in the user interface. Control then passes to block 818, where if the oncologist selects the option to open an educational module, process 800 may proceed to block 820. If the oncologist does not select the option to open an educational module, process 800 may proceed to block 822. In block 820, process 800 may display an instruction manual (i.e., a user manual) to the oncologist. Control then passes to block 822, where process 800 may display an FAQ menu, as well as a suggested pathway tutorial option, to the oncologist in the user interface. Control then passes to block 824, where if the oncologist selects the suggested pathway tutorial option, control passes to block 826. If the oncologist does not select the suggested pathway tutorial option, process 800 ends.

[0359] At block 826, the process 800 may run at least one tutorial, which may include a tutorial on how to use the collaboration device 20 (e.g., how to change the volume of the collaboration device 20) and suggest a “first question” that the oncologist may want to ask the collaboration device. In particular, the tutorial related to the suggested question may instruct the oncologist on how to query the collaboration device. After the oncologist performs some questions, the collaboration device 20 may end the tutorial and allow the oncologist to ask questions independently. The tutorial may be generated by recognizing the type of intent a particular physician may have and predicting questions based on various criteria (e.g., facility, specialty, questions from other physicians in the affiliation, patient molecular / clinical data and past order history, upcoming patients based on EMR scheduling integration, etc.). The tutorial may visually and / or audibly instruct the user on basic voice commands that the collaboration device can recognize: “volume up,” “volume down,” “start pairing,” “turn off,” or other suitable voice commands.

[0360] Referring now to FIG. 32, a screenshot 1100 is shown for use by a system administrator to specify system intents, intent parameters, and answer formats for provider panel types, consistent with at least some aspects of the present disclosure. As shown, a panel variable module 1104, an intent module 1108, and an answer module 1112 can be used by a user to specify an intent. Note that the modules may have different appearances for ease of identification. For example, the modules 1104-1112 can be different shapes. An intent module can have an intent (i.e., "Intent A") that may require variable inputs to answer. In this example, the variable is the panel type. The panel variable module 1104 corresponding to the type of panel can be linked to the intent module 1108, which can also be linked to an answer module 1112 that can automatically fill in the answer based on the panel variable module 1104. A user can drag and link the modules 1104-1112 using a mouse or touch screen to create an intent and associated answer.

[0361] 1 as well as FIG. 33, an intent extraction architecture 1150 is illustrated consistent with at least some aspects of the present disclosure. The intent extraction architecture 1150 may include an input module 1158 including a microphone and an output module 1162 including a speaker that may be included in the collaboration device 20. A user 1154 may provide an audible query to the input module 1158 and receive an audible answer from the output module 1162. The input module 1158 may process the audible query (e.g., perform text recognition) and transmit a query 1166 to an intent matching module 1174 included in the intent extraction architecture 1150. The intent matching module 1174 may include an intent matching application such as Dialogflow. The intent matching module 1174 may extract an intent from the query and transmit the query 1166 and the intent to a parameter extraction module 1178 included in the intent extraction architecture 1150. The parameter extraction module 1178 may extract any parameters from the query 1166 that are related to the intent. The parameter extraction module 1178 can then communicate with an API module 1182 and / or a database 1186 included in the intent extraction architecture 1150 to extract information related to the extracted parameters and / or intent from the database 1186. The information can be transmitted to the intent matching module 1174. The intent matching module 1174 can generate actionable data 1170 based on the information from the database 1186. The intent matching module 1174 can then transmit the actionable data 1170 to the output module 1162, which can output an audible answer to the user 1154 based on the actionable data 1170.

[0362] 1, 28, and 33, as well as 34, an example question and answer workflow 1200 is illustrated consistent with at least some aspects of the present disclosure. The workflow 1200 may include one or more collaboration devices 1204 (e.g., collaboration device 20 and / or mobile device 652), each including an input module 1208 and an output module 1212. The input module 1208 may include at least some of the components of the input module 1158, and the output module 1212 may include at least some of the components of the output module 1162. The input module 1208 may receive an audible query from an oncologist. The audible query may include a single question that may be formulated from a series of prompts displayed on one of the collaboration devices 1204. The input module 1208 may output the audible query (which may include a raw audio file) to an agent module 1216 included in the workflow 1200. The agent module 1216 may include several natural language understanding (NLU) modules that convert text or spoken user requests into actions. The agent module 1216 can convert the audible query into an action and transmit the action to an intent matching module 1224 included in the workflow 1200. The intent matching module 1224 can be substantially the same as the intent matching module 1174. The intent matching module 1224 can communicate with a fulfillment module 1228 included in the workflow 1200. Th...

Claims

1. 1. A method for providing a response to a user based on a user query, for use in a collaboration device including a processor, the method comprising: receiving a query from the user that can be answered in part by reference to past clinical or medical data related to a particular subject, the query including one or more parameters; identifying at least one candidate intent associated with the query, where if there are multiple candidate intents, the multiple candidate intents have a similar format and the at least one candidate intent is identified by a machine learning module that recognizes the query as sufficiently corresponding to an intent phrase provided to or generated by a machine learning module; identifying at least one intent from among a plurality of candidate intents based on one or more parameters included in the query; identifying at least one data action associated with the at least one intent; performing at least one of the identified at least one data action on a first data set contained in the historical clinical information or medical data related to the particular subject to generate a first responsive data set; using the first response data set to generate a response; providing said response; A method comprising:

2. 2. The method of claim 1, wherein the collaboration device includes a microphone and a speaker linked to the processor, the query is an audible query received via the microphone, and providing the response includes broadcasting an audible response via the speaker.

3. 10. The method of claim 1, wherein the historical clinical information or medical data related to the particular subject comprises one or more of a genomic biomarker or a treatment.

4. accessing molecular reporting information for said particular subject; The first data set included in the historical clinical information or medical data related to the specific subject includes molecular report information of the specific subject. The method of claim 1.

5. The step of identifying at least one candidate intent comprises: weighting medical condition-related intents higher than other possible intents; identifying a plurality of possible intents; determining whether any of the possible intents are medical condition-related intents; Enabling candidate weights for the at least one intent; The method of claim 1 , further comprising:

6. the at least one candidate intent comprises at least one eligible entity, and the method further comprises: identifying at least one parameter value from among the one or more parameters associated with an entity in the query that corresponds to the at least one eligible entity; the at least one data action is associated with the at least one eligible entity; The method of claim 1.

7. The at least one data movement comprises at least one first data movement, the historical clinical or medical data relating to the particular subject includes genomic biomarkers, and the method further comprises: identifying at least one second action associated with the genomic biomarker; performing at least one of the identified at least one data operation on the first data set included in the historical clinical information or medical data related to the particular subject, the historical clinical information or medical data including the genomic biomarkers; The method of claim 6.

8. 2. The method of claim 1, wherein the performing step further comprises performing at least one of the at least one data operation on a dataset including genomic variant or molecular report information for a plurality of subjects.

9. The method of claim 1 , wherein the query is answerable in part by reference to a treatment order, a standard of care, or a clinical trial.

10. 2. The method of claim 1, wherein the first response dataset relates to one or more of a testing method, a testing requirement, or a testing result, and optionally to a mutation test or an antibody-based test.

11. 13. The method of claim 1, further comprising the step of notifying the user at a future time that the subject has become newly eligible for one or more services, optionally the one or more services comprising a cancer board, a clinical trial, a diagnostic test, a treatment, or a drug.

12. generating one or more documents related to the subject matter, the one or more documents comprising one or more structured notes from among a plurality of structured note types, a transcription, a record of the subject matter, a billing slip, or a schedule appointment; identifying at least one second intent for generating the one or more documents, the at least one second intent being either included in a request from the user or inferred from an input from the user; The method of claim 1 , further comprising:

13. the collaboration device is in communication with one or more electronic document abstraction services, and the method comprises: and in response to a request from the user, identifying electronic documents associated with the request and providing summaries of some or all of the identified electronic documents. The method of claim 1.

14. A system for providing a response to a user based on a user query, comprising a server including a processor configured to perform the method of any one of claims 1 to 13.

15. A computer readable storage medium comprising instructions which, when executed by a processor, cause a computer system to perform the method of any one of claims 1 to 13.

Images