System and method for generating endoscopic report guidance using machine learning models

Abstract

The present disclosure relates to a system and method for generating endoscopic report guidance. The method can be applied to a computing system, or platform, having one or more component servers or computers. The computing system includes receiving patient data of a patient who has gone through a colonoscopy procedure. The computing device further receives endoscopic report data of the patient for the colonoscopy procedure. The endoscopic report data includes indication information including symptoms or reasons for the procedure, and findings and interventions information including details about abnormalities discovered and actions taken during the procedure. The computing device additionally generates, by one or more machine learning models, endoscopic report guidance based on the patient data and the endoscopic report data. The one or more machine learning models may analyze medical diagnostics to provide guidance and recommendations for a medical professional to prepare an endoscopic report.

Description

GAP24060-DUNV-W01 / 067655-202020SYSTEM AND METHOD FOR GENERATING ENDOSCOPIC REPORT GUIDANCE USING MACHINE LEARNING MODELSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of United States provisional application no. 63 / 709,156, filed 18 October 2024, which is hereby incorporated by reference as though fully set forth herein.BACKGROUND OF THE INVENTION

[0002] The present invention relates to the field of medical procedures and machine learning. In particular, the present invention relates to a system and method for generating guidance regarding medical procedures, particularly in the context of endoscopic examinations.

[0003] Endoscopic procedures are minimally invasive medical techniques used to examine the interior of a hollow organ or cavity in the body. These procedures involve the use of an endoscope, a flexible tube with a light and camera attached to it, which allows physicians to view images of the internal organs on a monitor.

[0004] During an endoscopic procedure, the physician documents various aspects of the examination. This documentation typically includes patient data, the indications for the procedure, and the findings observed during the examination. If any abnormalities or significant findings are detected, the physician will also document conclusions and recommendations, such as required medication, follow-up treatments, or surveillance protocols. It is a common practice for the conducting physician to document various aspects of the procedure. This documentation includes patient data, the reasons leading to the procedure, and the findings observed during the endoscopy. If any findings are observed, the physician will also document conclusions, such as required medication, follow-up treatments, or surveillance recommendations.

[0005] Traditionally, these conclusions have been created and recorded manually by the physician. This manual process is time-consuming and can lead to inefficiencies. For instance, during a colonoscopy, if certain findings are observed, the physician needs to document conclusions such as required medication and follow-up treatments or procedures. The manual nature of this documentation process can result in delays and increased workload for the physician.

[0006] Additionally, the information documented throughout the patient pathway is stored in either structured text or unstructured text formats. Structured text includes fact-based data, such asGAP24060-DUNV-W01 / 067655-202020 medication used and demographic information, but often lacks broader contextual information about the patient. Furthermore, while demographic information typically includes age and sex, structured text may miss other important contextual information such as lifestyle factors, social determinants of health, patient history nuances, and subjective notes from patient interactions. On the other hand, unstructured text provides context but is challenging to interpret due to the nonstandardized language used by different physicians across various disciplines. For example, a referring physician and an endoscopist may describe similar conditions using different terminologies, making it difficult to interpret and standardize the information.

[0007] Accordingly, there is a need for an automated system that can help the documentation process by providing guidance based on findings observed during endoscopic procedures. Such a system would streamline the documentation process, reduce the workload on physicians, and ensure that both structured and unstructured data are effectively utilized to provide comprehensive patient information.SUMMARY

[0008] Examples of the present disclosure provide systems and methods for generating endoscopic report guidance.

[0009] According to a first aspect of the present disclosure, a computer-implemented method for generating endoscopic report guidance is provided. The method may be implemented on a computing system or platform having one or more component servers or computers. The method may include receiving patient data of a patient who has gone through a colonoscopy procedure. The patient data includes date of birth, weight, and race of the patient. The method may include receiving endoscopic report data of the patient for the colonoscopy procedure. The endoscopic report data includes indication information including symptoms or reasons for the procedure, and findings and interventions information including details about abnormalities discovered and actions taken during the procedure. The method may also include generating, by a narrative model, a medical narrative based on the patient data. The narrative model is trained to analyze the patient data and translate patient data into natural language text in the form of a medical narrative. The method may additionally include generating, by a prompt model, a medical prompt based on the medical narrative and the endoscopic report data, wherein the prompt model analyzes medical narratives and endoscopic report data to generate medical prompts for generating medical guidance for a patient. This can be done, for example, by using Retrieval Augmented Generation (RAG)GAP24060-DUNV-W01 / 067655-202020 techniques. The method may further include generating, by a guidance model, endoscopic report guidance based on the generated medical prompt, wherein the guidance model analyzes medical diagnostics to provide guidance and recommendations for a medical professional to prepare an endoscopic report. The method may additionally include displaying the endoscopic report guidance.

[0010] According to a second aspect of the present disclosure, a computing device, such as a server, is provided. The computing device may include one or more processors, a non-transitory computer-readable memory storing instructions executable by the one or more processors. The one or more processors may be configured to receive the endoscopic report of a patient from the endoscope system. The one or more processors may also be configured to generate, using a prompt model, a medical prompt based on the endoscopic report. The prompt model may analyze endoscopic reports to generate medical prompts for generating medical guidance for a patient. The one or more processors may further be configured to generate, using a guidance model, endoscopic report guidance based on the generated medical prompt. The guidance model may analyze medical diagnostics to provide guidance and recommendations for a medical professional. The one or more processors may additionally be configured to display the endoscopic report guidance.

[0011] According to a third aspect of the present disclosure, a non-transitory computer- readable storage medium with instructions stored therein is provided. When the instructions are executed by one or more processors of the apparatus, the instructions may cause the apparatus to perform receiving an endoscopic report of a patient. The instructions may also cause the apparatus to generate, by a prompt model, a medical prompt based on the endoscopic report. The prompt model can be a natural language processing model trained on a corpus of endoscopic reports and corresponding medical prompts for generating medical guidance for a patient. The instructions may further cause the apparatus to generate, by a guidance model, endoscopic report guidance based on the generated medical prompt. The guidance model can be a large language model finetuned on medical guidelines and clinical best practices to provide guidance and recommendations for a medical professional. The instructions may additionally cause the apparatus to display the endoscopic report guidance.

[0012] These and other aspects and advantages will become apparent to those of ordinary skill in the art by reading the following detailed description, with reference where appropriate to the accompanying drawings. Further, it should be understood that the foregoing summary is merelyGAP24060-DUNV-W01 / 067655-202020 illustrative and is not intended to limit in any manner the scope or range of equivalents to which the appended claims are lawfully entitled.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples consistent with the present disclosure and, together with the description, serve to explain the principles of the disclosure.

[0014] FIG. 1 is a block diagram of an endoscopic report guidance system, according to an example of the present disclosure.

[0015] FIG. 2A is a flow chart illustrating an auto-reporting functionality process, according to an example of the present disclosure.

[0016] FIG. 2B is a flow chart illustrating an auto-reporting functionality process, according to an example of the present disclosure.

[0017] FIG. 3 is a flow chart illustrating a text classification network process, according to an example of the present disclosure.

[0018] FIG. 4 is a flow chart illustrating a patient feature vector refinement process, according to an example of the present disclosure.

[0019] FIG. 5 is a flow chart illustrating an optimal clinical pathway prediction process, according to an example of the present disclosure.

[0020] FIG. 6 is a flow chart illustrating a method for automatic generation of endoscopic report guidance, according to an example of the present disclosure.

[0021] FIG. 7 is a flow chart illustrating a method for automatic generation of endoscopic report guidance, according to an example of the present disclosure.

[0022] FIG. 8 is a flow chart illustrating a method for automatic generation of endoscopic report guidance, according to an example of the present disclosure.DETAILED DESCRIPTION

[0023] While the present invention is capable of being embodied in various forms, for simplicity and illustrative purposes, the principles of the invention are described by referring to certain embodiments thereof. It is understood, however, that the present disclosure is to be considered as an exemplification of the claimed subject matter and is not intended to limit theGAP24060-DUNV-W01 / 067655-202020 appended claims to the specific embodiments illustrated. It will be apparent to one of ordinary skill in the art that the invention may be practiced without limitation to these specific details. In other instances, well-known methods and structures have not been described in detail so as not to unnecessarily obscure the invention. For example, while embodiments herein are described with reference to endoscopic examinations, it is understood that the systems and methods may be implemented similarly with respect to medical procedures more generally.

[0024] The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to limit the present disclosure. As used in the present disclosure and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It shall also be understood that the terms “and” and “or” used herein are intended to signify and include any or all possible combinations of one or more of the items listed in the associated list. As used herein, the term “if’ may be understood to mean “when,” “upon,” or “in response to a judgment,” depending on the context.

[0025] The present disclosure relates to systems and methods for generating endoscopic report guidance using machine learning models. In one or more embodiments, a computing system or platform can determine appropriate medical guidance and send recommendations to a medical professional based on an endoscopic report. In this way, a healthcare provider or medical institution can streamline the process of documenting endoscopic procedures. The endoscopic report can include findings, observations, and patient data from an endoscopic examination. In such embodiments, the medical professional conducting the endoscopic procedure can gain automated assistance in generating report guidance upon completing the examination. The guidance can include an “auto-conclude” feature by which the medical professional receives guidance and recommendations based on the endoscopic findings so that the user can then use the guidance to finalize a medical report. For example, a physician may perform a colonoscopy and document the findings in an endoscopic report. The report may be processed using a computing system that is capable of generating guidance for the physician by providing recommended follow-up procedures, medications, or treatment plans. In one embodiment, the guidance may come from a machine learning model that is trained on clinical guidelines and best practices. This model can contain a repository of medical knowledge and decision-making logic for generating appropriate recommendations. In another embodiment, Natural Language Processing (NLP) techniques can beGAP24060-DUNV-W01 / 067655-202020 used to extract and classify meaningful information from unstructured text data in patient records and generate guidance. Thus, the field of medical procedures and machine learning is improved.

[0026] In one or more embodiments, a computing system or platform can provide a comprehensive and thorough characterization of a patient to automatically generate guidance regarding the optimal further treatment of the patient according to clinical evidence (as derived from clinical guidelines). Besides the preoperative information, the computing system or platform can include the intraoperative findings gathered during endoscopic examination in the generation of a recommendation for follow-up steps.

[0027] In one or more embodiments, a language model is trained based on a history of existing medical case documentations, where a probabilistic relationship between one or several of the following input items such as patient sex, patient age, indication for procedure, findings (i.e. observed physiologic or pathologic parameters, abnormalities, pathology) and interventions (e.g., removed tissue, cauterization, dilation, catheterization, etc.) are linked to the output of reported guidance and recommended or prescribed follow-up procedures. The trained model thus is able to provide a plurality of likely guidance and recommendations for helping determine guidance which a medical professional can either dismiss, reject, or use for drafting documentation or finalizing and a colonoscopy report.

[0028] In an embodiment, the model can be implemented as functionality within an endoscopic documentation tool, such as EndoBase or ZipReport, further enhancing the efficiency of the documentation process.

[0029] Turning now to the figures, FIG. 1 shows a guidance process 10 with an endoscopic system 20, a network 40, a medical records database 60, and a computing system 80According to example embodiments shown schematically in FIG. 1, the endoscopic system 20 includes an endoscope 22, processor 24, memory 26, VO interface 28, and display 30. The computing system 80 includes a processor 82, memory 84, I / O interface 86, and display 88. The endoscopic system 20 can communicate with the computing system 80 and the medical records database 60 through the network 40.

[0030] The endoscopic system 20 functions as an integrated unit for performing endoscopic procedures. The endoscope 22 is the primary instrument used to examine the patient, capturing visual data and potentially other sensor information from inside the body. This data is then processed by the processor 24, which analyzes the incoming information and generates preliminaryGAP24060-DUNV-W01 / 067655-202020 findings. Memory 26 stores the captured endoscopic images, video footage, and sensor data from the current procedure. It may also temporarily store patient information, procedure protocols, and system software necessary for the immediate operation of the endoscopic system. This local storage in the memory 26 ensures quick access to critical data during the procedure and allows for initial processing and analysis to occur within the endoscopic system itself.

[0031] The I / O interface 28 allows the medical professional to input additional information, control the endoscope, and interact with the system’s software. Display 30 presents the live endoscopic imagery, as well as medical information or other relevant information, to the medical professional in real-time. This integrated approach allows for seamless data collection, analysis, and documentation throughout the endoscopic procedure.

[0032] The computing system 80 serves as a central processing and analysis hub for performing and documenting endoscopic procedures. It includes a processor 82, which can run machine learning models and algorithms that generate the endoscopic report guidance. Memory 84 stores patient data, clinical guidelines, and the trained language models used for generating recommendations.

[0033] The I / O interface 86 facilitates communication with the endoscopic system 20 and the medical records database 60 through the network 40. This allows the computing system 80 to receive real-time data from the endoscope 22 and access relevant patient history and medical records. Display 88 provides a user interface for medical professionals to review generated guidance, interact with the system, and finalize reports.

[0034] The endoscopic system 20 can transmit procedure data, images, and preliminary findings to the computing system 80 via the network 40. Simultaneously, the computing system 80 can retrieve relevant patient information from the medical records database 60. This interconnected setup enables comprehensive analysis and guidance generation, combining realtime endoscopic data with historical patient information and clinical guidelines stored in the computing system 80.

[0035] The processors 24 and 82 can typically control the overall operations of the endoscopic system 20 and computing system 30, such as the operations associated with data acquisition, data processing, and data communications. Processors 24 and 82 can include one or more processors to execute instructions to perform all or some of the steps in the below-described methods. Moreover, processors 24 and 82 can include one or more modules that facilitate the interactionGAP24060-DUNV-W01 / 067655-202020 between processors 24, 82 and other components. The processor may be or include a central processing unit (CPU), a microprocessor, a single chip machine, a graphical processing unit (GPU), a System on a Chip (SoC), Tensor Processing Unit (TPU), Quantum Processor, Vision Processing Unit (VPU) or the like.

[0036] Memory 26 and 86 can store various types of data to support the operation of the endoscopic system 20 and computing system 30. Memory 26 and 86 can include predetermined software. Examples of such data comprise instructions for any applications or methods operated on the endoscopic system 20 and computing system 30. The memory 26, 86 may be implemented by using any type of volatile or non-volatile memory devices, or a combination thereof.

[0037] In some embodiments, there is also provided a non-transitory computer-readable storage medium comprising a plurality of programs, such as comprised in the memory 26, 86, executable by the processors 24, 82, for performing the above-described methods. For example, the non-transitory computer-readable storage medium may be a ROM, a RAM, or the like.

[0038] The non-transitory computer-readable storage medium has stored therein a plurality of programs for execution by a computing device having one or more processors, where the plurality of programs for execution by a computing device having one or more processors, where the plurality of programs when executed by the one or more processors, cause the computing device to perform the above-described method for motion prediction.

[0039] In a first embodiment, it is proposed to combine at least two different language models to generate medical guidance based on medical reports. The first model is fed with entries from the different text fields of the report / reporting tool and is programmed or instructed to create a prompt that contains a question as input for a second language model. Optionally, a third language model can convert data points to language as another input to the first model. The first model is specifically optimized to act as a question generator, and the third model is a generative question- and-answer model. In an embodiment, the second model would be a large language model specifically trained with clinical guidelines.

[0040] FIG. 2A shows an example process 100A for auto-reporting functionality based on a combination of multiple models in accordance with the present disclosure. Process 100A includes an endoscopic report 120, patient data 130, first model 140, second model 150, third model 160, and prompt output 170A. The endoscopic report 120 can be generated by a healthcare professional using the endoscopic system 20 and can include, for example, “abdominal bloating, FOBTGAP24060-DUNV-W01 / 067655-202020 positive, national bowel cancer screening program. There were four sessile polyps in the transverse colon (size of the largest: 5mm).” The patient data 130 can be retrieved from the medical records database 60 and can include, for example, “DOB = 01.01.1970, sex = male.” The first model 140 is a prompt model that is trained to generate medical prompts based on medical reports. The second model 150 is a narrative model that is trained to generate a patient narrative based on the patient’s medical data. The second model 150 intakes patient data 130 and generates a narrative about the patient that can be used by the first model 140 to generate a medical prompt for generating guidance relative to the endoscopic report. The output or narrative of the second model 150 can be an input to the first model 140. The second model 150 can use the endoscopic report 120 and narrative from the first model 140 to generate a prompt for the third model 160. The prompt can include, for example, “provide summary and recommendation for treatment and follow-up on a male patient, aged 54 years who reported abdominal pain and is FOBT positive and has been diagnosed with four sessile polyps in the transverse colon with a size of up to 5mm?” The third model 160 can be a generative question-and-answer model trained on clinical guidelines and can output guidance 170A. The guidance can include guidance about the patient’s diagnosis and recommendations for next steps. For example, the guidance 170A can include “Guidance: four sessile polyps in the transverse colon, up to 5mm in size. Recommendations: colonoscopy with Polypectomy: Immediate colonoscopy for polyp removal. Histopathological Examination: Analysis of removed polyps to determine benign or malignant nature. Follow-up Colonoscopy: Regular follow-up colonoscopies as per the gastroenterologist’s recommendation, considering polyp characteristics and patient history. Lifestyle Modifications: Adopting a healthy lifestyle, including diet and exercise, and avoiding risk factors like smoking and excessive alcohol consumption. Consultation: Continued follow-up with a gastroenterologist for ongoing management and surveillance.”

[0041] The advantage of having multiple models to generate medical guidance includes more tailored guidance for the patient. For example, the first model 140 can help convert the varied and potentially unstructured entries from a medical report into a more structured, focused question or prompt. This provides a clearer, more specific input for the third model 160 to work with for generating guidance. Another advantage is by formulating a question or prompt, the first model 140 can capture the context and relationships between different pieces of information in the report, which might not be apparent from the raw entries alone. Additionally, having a specific questionGAP24060-DUNV-W01 / 067655-202020 allows the third model 160 to generate more precise and relevant answers or guidance, potentially improving the accuracy and relevance of the final output. This approach allows for more flexibility in the system. The first model 140 can be fine-tuned to generate different types of questions based on the specific needs of the medical specialty or procedure type. Similarly, the first model 140 can be specifically trained on medical report interpretation, while the third model 160 can focus on medical diagnosis and recommendations, allowing each model to specialize in its task. The second model 150 can similarly be specifically trained to interpret unstructured medical data and provide relevant information needed to understand the patient’s history and generate a prompt for medical guidance.

[0042] In another embodiment, the process may only use a first model 140 and a third model 160. In such an embodiment, the first model 140 can generate a prompt based on the endoscopic report 120. In a similar embodiment, the first model 140 may also use both the endoscopic report 120 and the patient data 130 to generate a prompt for the third model 160.

[0043] FIG. 2B shows an example process 100B for auto-reporting functionality based on a combination of multiple models and a user interface in accordance with the present disclosure. Process 100B includes a colonoscopy report user interface HOB, the first model 140, the second model 150, and the third model 160. The colonoscopy report user interface HOB includes information about the hospital, patient, and colonoscopy procedure. The hospital and patient data may be auto populated based on stored information about the patient and hospital the procedure was performed or where the report is being generated. The information about the patient includes patient demographic data displayed in a patient data section 112B. Patient demographic data includes a patient’s age, weight, height, race, ethnicity, gender, marital status, employment status, education level, income level, primary language, any disabilities or special needs, and any other information that can be used for risk stratification or risk assessment. The information about the colonoscopy procedure can be displayed in a colonoscopy report section that includes an indication information section 114B, a findings and interventions information section 116B, and a conclusion section 118B. The indication information section 114B includes details about the patient’s symptoms or reasons for the procedure. For example, the section can read “Abdominal bloating. FOBT positive - National Bowel Cancer Screening Program.” The findings and interventions information section 116B includes details about any abnormalities discovered during the procedure and any actions taken to address them. For example, the section can read “There were four sampleGAP24060-DUNV-W01 / 067655-202020 polyps in the transverse colon (size of the largest: 5mm).” Other information displayed includes information about the endoscopist who performed the procedure and preparation for the procedure, as well as images of the procedure and other information related to the patient, their medical history, and colonoscopy procedure. The conclusion section 118B, for example, may display “Colonic Polyp(s) - recommended follow-up for ongoing management and surveillance.”

[0044] The process 100B involves extracting patient demographic data from the patient data section 112B and extracting report specific data associated with the specific procedure being analyzed from the indication information section 114B and the findings and interventions information section 116B. The patient demographic data from the patient data section 112B is input to the first model 140 for generating patient data context or medical narrative to be used by the second model 150 (as described above). The report specific data from the indication information section 114B and the findings and interventions information section 116B is input to the second model 150, along with the patient data context from the first model 140 for generating a prompt for the third model 160 (as described above). The third model 160 intakes the prompt, which includes relevant information about the patient and procedure, to output a conclusion 170B about the procedure for the report (as described above). Thus, the process 100B can intake information from the report to auto generate guidance for the conclusion to be used in the report.

[0045] In an embodiment, information within the report can be auto generated based on patient information and procedure information. In another embodiment, a user can manually enter information about the patient and the procedure. For example, a user can input information in sections 112B, 114B, and 116B for real-time report generation. The user may further edit the conclusion section 118B for finalizing the report. In another embodiment, edits to section 118B can be an input to the third model 160 or an additional model that can generate further guidance or respond to a question within the section 118B about the guidance, patient, medical history, or procedure.

[0046] In a second embodiment, guidance may be generated based on NLP techniques, specifically Text Classification and Named Entity Recognition, to extract and classify meaningful information from unstructured text data in patient records. This approach creates a comprehensive patient feature vector that is continuously refined throughout the patient pathway and can be compared to encoded clinical guideline personas, enabling the system to predict the optimal next steps in the patient’s treatment process.GAP24060-DUNV-W01 / 067655-202020

[0047] NLP -based text classification involves extracting and processing information from endoscopic procedure reports. NLP -based text classification is a technique in computational linguistics and machine learning that involves assigning predefined categories or labels to text data. This method leverages NLP algorithms to analyze and interpret the content of the text, allowing for automated categorization based on its semantic and syntactic properties. The NLPbased Text Classification can be used to extract and classify meaningful features from unstructured text data (e.g., referral letter). For example, classification labels for colonoscopy can be typical indications, such as known or occult gastrointestinal bleeding, stool positive for occult blood, screening etc. This features vector (or embedding) contains the encoded patient information. The feature vector can then be compared (classified) against the class labels learned during the training phase of the text classification network. The class label with the highest similarity to the encoded feature vector of the patient will be the predicted class label.

[0048] In an embodiment, text encoding networks, such as Bidirectional Encoder Representations from Transformers (BERT) or Generative Pre-trained Transformers (GPT) can be used to encode unstructured text into a feature vector in latent embedding space. These text encoding networks are advanced models in NLP that convert unstructured text into structured numerical representations known as feature vectors. These vectors reside in a high-dimensional latent embedding space where semantically similar texts are positioned close to each other. The process begins with tokenization, breaking down text into smaller units called tokens. These tokens are then converted into dense vector representations through an embedding layer. Multiple transformer layers subsequently process these token vectors, capturing contextual information by attending to different parts of the text simultaneously. BERT, for instance, captures context from both directions — left-to-right and right-to-left — while GPT predicts the next word in a sequence to understand the text’s flow and structure. The final output is a feature vector for each token or the entire text, which can be used for various NLP tasks such as text classification, similarity analysis, information retrieval, and sentiment analysis.

[0049] FIG. 3 shows a simplified process 200 of a text classification network in accordance with the present disclosure. The process includes an unstructured text 210, encoded information 220, and feature vectors 230. The unstructured text 210 can be part of a medical report and includes text stating, “the patient had 3 previous endoscopic examinations without any suspicious findings.” Feature extraction can be done from the unstructured data. The encoded information 220 is aGAP24060-DUNV-W01 / 067655-202020 feature vector that can extract the highest similarity for each label. The feature vectors 230 is feature vector of learned / trained class labels and includes labels 232, 234, 236, 238, and 240. A comparison of feature vectors against those feature vectors learned during training of the network is used. The labels 232 can be an “occult gastrointestinal bleeding” label, label 234 can include, a “stool positive for occult blood” label, and label 236 can include a “screening” label. FIG. 3 illustrates this process with an example of unstructured text that includes information about a patient who had three previous endoscopic examinations without any suspicious findings. This unstructured text is input to the encoded information, which includes a feature vector. The encoded information then outputs the highest similarity with a label for the encoded information to the feature vectors of learned / trained class labels. These learned / trained class labels include occult gastrointestinal bleeding, stool positive for occult blood, and screening.

[0050] NLP -based Named Entity Recognition can be used to extract meaningful entities (i.e., objects, such as medications) from unstructured text data. Named Entity Recognition works in a similar way as text classification (see FIG. 4 described below), but the network is trained to detect individual entities from unstructured text. Technically, each individual word in the unstructured text is compared against the entities learned during training. Comparison is made based on feature vectors. Whereas Text Classification aims to learn the “essence” (general intention) of the unstructured text data (context, but less precise), Named Entity Recognition is able to detect special words (features) in the unstructured text (precise, but less context). The combination (fusion) of the predicted feature vectors from Text Classification and Named Entity Recognition can give a more precise characterization of the patient (i.e., combined feature vector).

[0051] Every single piece of data (either in unstructured or in structured text format) that will be added throughout the patient pathway to the patient’s health record or examination report (such as endoscopic findings) is encoded in the same way (using Text Classification and Named Entity Recognition) and hence attributes to the refinement of the patient’s feature vector. The simplified process of information encoding throughout the process is illustrated in FIG. 4.

[0052] FIG. 4 illustrates the evolution of a patient’s feature vector at different stages of care models in accordance with the present disclosure. Specifically, FIG. 4 shows three distinct time points: preoperative stage (tO), intraoperative stage (tl), and postoperative stage (t2). This includes refinement of the patient’s feature vector that encodes all patient-specific information throughout the patient pathway. Feature Vector (tO) Feature Vector (tl) Feature Vector (t2).GAP24060-DUNV-W01 / 067655-202020

[0053] The patient’s feature vector can be used to map the patient’s characteristics to the proposed diagnostic and therapeutic pathway as provided by clinical guidelines. Clinical guidelines are often based on descriptions of typical patient personas (i.e., patients with this and that condition should get this and that treatment etc.). These personas can be encoded in the same way as the patient information is encoded (using Text Classification and Named Entity Recognition). Subsequently, the individual patient’s feature vector can be compared to the typical feature vectors of the personas (similarity matching, e.g., determination of cosine similarity) described in the clinical guidelines at different stages. Hence, the next optimal clinical step for the individual patient can be predicted in line with the clinical guidelines. The process of feature matching comparison to predict the next step in the clinical workflow is illustrated in FIG. 5, described below.

[0054] FIG. 5 shows a process 400 for predicting the correct patient path on the clinical pathway in accordance with the present disclosure. Process 400 includes a patient 410, first feature vector 420, similarity values 422, 424, second feature vector 430, third feature vector 440, first persona 450, and second persona 460. Patient 410 can have a medical report including information such as a patient with stool positive for occult blood and 4 sessile polyps of size 5mm. The information analyzed through the feature vector 420 and the similarity percentage for other feature vectors 422 and 424 are derived. For example, similarity 422 is 97% and similarity 424 is 20%. The second feature vector 430 has a first persona 450 with a patient with stool positive for occult blood and polyps. The third feature vector 440 has a second persona 460 with a patient with occult GI bleeding and no polyps. A feature vector is calculated, reflecting all information from unstructured and structured patient data. Feature vectors are also calculated for hypothetical personas on the clinical pathway (based on their pre-conditions and intraoperative findings). The feature vector of the individual patient can be compared to the feature vectors of the personas for different clinical pathways to identify the persona with the most similarity to the patient. This way, a recommendation / prediction for the most likely next step within the treatment process can be realized for an individual patient.

[0055] In one or more embodiments, several additional approaches can be used to further enhance the capabilities of automated guidance generation based on medical records. These advanced embodiments focus on extracting, analyzing, and synthesizing information from various sections of the report to generate comprehensive and accurate medical guidance. For instance, oneGAP24060-DUNV-W01 / 067655-202020 approach could involve deep semantic analysis of the procedure description, findings, and physician notes to identify key medical entities and their relationships. This could enable the system to automatically infer potential diagnoses, complications, or necessary follow-up actions based on the specific language and context used in the report.

[0056] Another embodiment could incorporate a temporal aspect, analyzing multiple medical reports for the same patient over time. This longitudinal analysis could help identify trends in the patient’s condition, track the progression of diseases, or evaluate the effectiveness of previous treatments. The system could also compare the current report with a database of similar cases to generate evidence-based guidance and recommendations. Additionally, an advanced embodiment might employ a hybrid approach, combining rule-based systems (based on established medical guidelines) with machine learning models trained on large datasets of anonymized medical reports. This could allow the system to generate guidance that is both clinically sound and tailored to the specific nuances of each case.

[0057] This approach employs text classification to extract and classify meaningful features from unstructured text data, helping to understand the general context and intention of the text. It also uses named entity recognition to identify and extract specific entities from the unstructured text. The system combines the results from these techniques to create a comprehensive feature vector that characterizes the patient’s condition and procedure findings. This feature vector is continuously updated and refined as new information is added throughout the patient’s care pathway.

[0058] Another innovative aspect is the similarity matching for treatment recommendations. By comparing the patient’s feature vector to those of guideline-based personas, the system can predict the optimal next steps in the clinical workflow. This automated matching process represents a new way to generate evidence-based recommendations. Furthermore, the system integrates both structured and unstructured data, providing a more holistic view of the patient’s condition than systems that rely on only one type of data.

[0059] Additionally, this approach is adaptable across the patient pathway, unlike systems that focus on a single point in care. It can be applied at various stages (preoperative, intraoperative, postoperative), making it more versatile and comprehensive. These innovations collectively represent a novel approach to automating and improving the accuracy of guidance and recommendations in endoscopic procedure reports, providing a more automated, consistent, andGAP24060-DUNV-W01 / 067655-202020 evidence-based method for generating guidance and recommendations.

[0060] In one or more embodiments, several additional approaches can be used to further enhance the capabilities of automated guidance generation based on medical records. One approach could involve developing a more sophisticated question generation model that can create more nuanced and context-specific prompts based on the various fields in the medical report. This enhanced model could be trained to recognize complex patterns and relationships between different data points in the report, allowing it to generate more targeted questions for the guidance model.

[0061] Another embodiment might focus on improving the guidance model by fine-tuning it with a larger and more diverse dataset of endoscopic reports and clinical guidelines. This could enable the model to provide more precise and clinically relevant guidance based on the specific details documented in each report. Additionally, an advanced embodiment could incorporate a feedback loop mechanism, where the system learns from physician feedback on its generated guidance, continuously improving its accuracy and relevance over time.

[0062] FIG. 6 shows an example method for automatic generation of endoscopic report guidance based on the first embodiment. The method can be applied on a computer or server.

[0063] In step 510, the computer receives an endoscopic report of a patient.

[0064] In step 512, the computer generates, by a prompt model, a medical prompt based on the endoscopic report, wherein the prompt model analyzes endoscopic reports to generate medical prompts for generating medical guidance for a patient. The prompt model is a natural language processing model trained on a corpus of endoscopic reports and corresponding medical prompts.

[0065] In step 514, the computer generates, by a guidance model, endoscopic report guidance based on the generated medical prompt, wherein the guidance model analyzes medical diagnostics to provide guidance and recommendations for a medical professional. The guidance model is a large language model fine-tuned on medical guidelines and clinical best practices.

[0066] In step 516, the computer displays the endoscopic report guidance. The endoscopic report guidance includes recommended follow-up procedures, medications, or treatment plans.

[0067] In another embodiment, the computer can generate a confidence score for the endoscopic report guidance; it can display the confidence score along with the endoscopic report guidance.

[0068] In another embodiment, the computer can receive imaging data from an endoscopicGAP24060-DUNV-W01 / 067655-202020 system, and the prompt model can generate the medical prompt based on the endoscopic report and the imaging data. In this embodiment, the prompt model is able to analyze images or a separate model, like the narrative model, an imaging model, can be used to analyze the imaging data and provide the prompt model an imaging narrative for generating the prompt.

[0069] FIG. 7 shows an example method for automatic generation of endoscopic report guidance based on the first embodiment. The method can be applied on a computer or server.

[0070] In step 610, the computer receives the patient data of the patient. The patient data includes at least one of the patient demographics, medical history, laboratory results, or imaging studies.

[0071] In step 612, the computer generates, by a narrative model, a medical narrative based on the patient data. The narrative model is trained to analyze the patient data of a patient and translate patient data into natural language text.

[0072] In step 614, the computer receives an endoscopic report of a patient.

[0073] In step 616, the computer generates, by a prompt model, a medical prompt based on the endoscopic report and the medical narrative. The prompt model analyzes endoscopic reports to generate medical prompts for generating medical guidance for a patient.

[0074] In step 618, the computer generates, by a guidance model, endoscopic report guidance based on the generated medical prompt. The guidance model analyzes medical diagnostics to provide guidance and recommendations for a medical professional.

[0075] In step 620, the computer displays the endoscopic report guidance.

[0076] FIG. 8 shows an example method for automatic generation of endoscopic report guidance based on the second embodiment. The method can be applied on a computer or server.

[0077] In step 710, the computer performs NLP -based text classification to extract and classify meaningful features from unstructured text data in the reports. This can be used to understand the general context and intention of the text.

[0078] In step 712, the computer performs NLP-based named entity recognition to identify and extract specific entities (like medications, conditions, etc.) from the unstructured text.

[0079] In step 714, the computer performs feature vector creation by combining the results from text classification and named entity recognition to create a comprehensive feature vector that characterizes the patient’s condition and the procedure findings.

[0080] In step 716, the computer performs guideline matching by comparing the patient’sGAP24060-DUNV-W01 / 067655-202020 feature vector to encoded representations of clinical guidelines and typical patient personas to determine the most appropriate next steps in the patient’s care.

[0081] In step 718, the computer performs, based on the matching process, recommendation generation for follow-up care, treatments, or further procedures that align with clinical guidelines.

[0082] Additionally, the computer can perform continuous refinement to continuously update and refine the patient’s feature vector as new information is added throughout the patient’s care pathway.

[0083] Multiple embodiments are described herein, including the best mode known to the inventors for practicing the claimed invention. Of these, variations of the disclosed embodiments will become apparent to those of ordinary skill in the art upon reading the foregoing disclosure. The inventors expect skilled artisans to employ such variations as appropriate (e.g., altering or combining features or embodiments), and the inventors intend for the invention to be practiced otherwise than as specifically described herein. In addition, while the invention has been described in terms of several preferred embodiments, it should be understood that there are many alterations, permutations, and equivalents that fall within the scope of this invention. It should also be noted that there are alternative ways of implementing both the process and apparatus of the present invention. For example, steps do not necessarily need to occur in the orders shown in the accompanying figures and may be rearranged as appropriate. It is therefore intended that the appended claim includes all such alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.

Claims

GAP24060-DUNV-W01 / 067655-202020CLAIMSWhat is claimed is:

1. A method for automatic generation of endoscopic report guidance, comprising: receiving patient data of a patient who has gone through a colonoscopy procedure, wherein the patient data comprises date of birth, weight, and race of the patient; receiving endoscopic report data of the patient for the colonoscopy procedure, wherein the endoscopic report data comprises indication information including symptoms or reasons for the procedure, and findings and interventions information including details about abnormalities discovered and actions taken during the procedure; and generating, by one or more machine learning models, endoscopic report guidance based on the patient data and the endoscopic report data, wherein the one or more machine learning models analyzes medical diagnostics to provide guidance and recommendations for a medical professional to prepare an endoscopic report.

2. The method of claim 1, wherein generating, by one or more machine learning models, the endoscopic report guidance comprises: generating, by a narrative model, a medical narrative based on the patient data, wherein the narrative model is trained to analyze the patient data and translate patient data into natural language text in a form of a medical narrative; generating, by a prompt model, a medical prompt based on the medical narrative and the endoscopic report data, wherein the prompt model analyzes medical narratives and endoscopic report data to generate medical prompts for generating medical guidance for a patient; and generating, by a guidance model, the endoscopic report guidance based on the generatedGAP24060-DUNV-W01 / 067655-202020 medical prompt, wherein the guidance model analyzes medical diagnostics to provide guidance and recommendations for a medical professional to prepare an endoscopic report.

3. The method of claim 1, further comprising: receiving a user input modifying the endoscopic report data; generating, by the prompt model, an updated medical prompt based on the medical narrative and the modified endoscopic report data; generating, by the guidance model, an updated endoscopic report guidance based on the updated generated medical prompt; and displaying the updated endoscopic report guidance.

4. The method of claim 1, wherein the prompt model is a natural language processing model trained on a corpus of endoscopic reports and corresponding medical prompts.

5. The method of claim 1, wherein the guidance model is a large language model fine-tuned on medical guidelines and clinical best practices.

6. The method of claim 1, further comprising: receiving user input modifying the endoscopic report guidance; and updating the displayed endoscopic report guidance in real-time based on the user input.

7. The method of claim 1, wherein the endoscopic report guidance includes recommended follow-up procedures, medications, or treatment plans.GAP24060-DUNV-W01 / 067655-2020208. The method of claim 1, wherein the patient data further comprises laboratory results or imaging studies.

9. The method of claim 1, further comprising: generating a confidence score for the endoscopic report guidance; and displaying the confidence score along with the endoscopic report guidance.

10. The method of claim 1, wherein generating the medical prompt further comprises: extracting key features from the endoscopic report using natural language processing techniques.

11. The method of claim 1, further comprising: receiving imaging data from an endoscopic system, and wherein generating the medical prompt comprises: generating, by the prompt model, the medical prompt based on the endoscopic report and the imaging data.

12. A system for automatic generation of endoscopic report guidance, comprising: an endoscope system configured to perform an endoscopic procedure and create an endoscopic report; a memory storing instructions; and a processor configured to execute the instructions to:GAP24060-DUNV-W01 / 067655-202020 receive the endoscopic report of a patient from the endoscope system; and generate, by one or more machine learning models, endoscopic report guidance based on the patient data and the endoscopic report data, wherein the one or more machine learning models analyze medical diagnostics to provide guidance and recommendations for a medical professional to prepare an endoscopic report.

13. The system of claim 12, generating, by one or more machine learning models, the endoscopic report guidance comprises: generate, by a narrative model, a medical narrative based on the patient data, wherein the narrative model is trained to analyze the patient data and translate patient data into natural language text in a form of a medical narrative; generate, by a prompt model, a medical prompt based on the medical narrative and the endoscopic report data, wherein the prompt model analyzes medical narratives and endoscopic report data to generate medical prompts for generating medical guidance for a patient; and generate, by a guidance model, the endoscopic report guidance based on the generated medical prompt, wherein the guidance model analyzes medical diagnostics to provide guidance and recommendations for a medical professional to prepare an endoscopic report.GAP24060-DUNV-W01 / 067655-20202014. The system of claim 12, wherein the processor is further configured to: display the endoscopic report guidance; receive patient data of the patient; and generate, using a narrative model, a medical narrative based on the patient data, wherein the narrative model is trained to analyze patient data of a patient and translate patient data into natural language text, and wherein generating the medical prompt comprises: generating, using the prompt model, the medical prompt based on the endoscopic report and the medical narrative.

15. The system of claim 14, wherein the patient data includes at least one of patient demographics, medical history, laboratory results, or imaging studies.

16. The system of claim 12, wherein the prompt model is a natural language processing model trained on a corpus of endoscopic reports and corresponding medical prompts.

17. The system of claim 12, wherein the guidance model is a large language model finetuned on medical guidelines and clinical best practices.

18. The system of claim 12, wherein the processor is further configured to: receive user input modifying the endoscopic report guidance; and update the displayed endoscopic report guidance based on the user input.GAP24060-DUNV-W01 / 067655-20202019. A non-transitory computer-readable storage medium storing a plurality of programs for execution by a computing device having one or more processors, wherein the plurality of programs, when executed by the one or more processors, cause the computing device to perform: receiving an endoscopic report of a patient; generating, by one or more machine learning models, endoscopic report guidance based on the patient data and the endoscopic report data, wherein the one or more machine learning models analyze medical diagnostics to provide guidance and recommendations for a medical professional to prepare an endoscopic report.

20. The non-transitory computer-readable storage medium of claim 19, wherein the plurality of programs further cause the computing device to perform: displaying the endoscopic report guidance; receiving patient data of the patient; and generating, by a narrative model, a medical narrative based on the patient data, wherein the narrative model is trained to analyze patient data of a patient and translate patient data into natural language text, and wherein generating the medical prompt comprises: generating, by the prompt model, the medical prompt based on the endoscopic report and the medical narrative.

Images