Consultation assistance method and device, electronic equipment and storage medium

Abstract

The application provides a consultation auxiliary method and device, electronic equipment and storage medium, wherein the method comprises: in response to a consultation request, obtaining historical medical record data of a target patient and real-time illness data of a consultation site, performing semantic induction on the historical medical record data to generate a historical medical record abstract; identifying a screen image to extract multi-modal monitoring information of the target patient; and based on the historical medical record abstract and the multi-modal monitoring information, obtaining a target illness abstract and sending the target illness abstract to a consultation physician terminal. Through semantic induction on the historical medical record data and visual recognition on the screen image, the application can quickly extract and associate the patient's past medical history background and multi-modal monitoring information from multi-source heterogeneous medical data, thereby instantaneously fusing fragmented medical information into a target illness abstract and immediately pushing the target illness abstract to the consultation physician, so as to improve the automation and timeliness of medical information integration, thereby significantly shortening the response and decision-making time of the consultation.

Description

Technical Field

[0001] This invention relates to the field of smart healthcare technology, and in particular to a consultation assistance method, device, electronic device, and storage medium. Background Technology

[0002] The hospital's internal consultation system is a crucial link in ensuring medical quality and resolving complex and difficult cases. Consultations are generally divided into "regular consultations" and "emergency consultations." Regular consultations are typically required to be completed within 24 hours, while emergency consultations require the consulting physician to arrive on-site within 5-10 minutes of receiving the request, with the core purpose of saving lives as quickly as possible. In the current process, the requesting party needs to manually write or enter an electronic consultation request form, summarizing the patient's lengthy medical history, various auxiliary examination results, and the patient's current emergency condition. Before arriving on-site, the invited consulting physician often has limited knowledge of the patient's condition and needs to spend a significant amount of time reviewing paper or electronic medical records, reports from the Picture Archiving and Communication System (PACS), and Laboratory Information System (LIS), and checking on the patient again by phone or at the bedside to obtain the latest information. This process is particularly inefficient in emergency consultation scenarios, severely consuming precious rescue time, potentially leading to delayed diagnosis and treatment, thus affecting the patient's prognosis.

[0003] Currently, the consultation process within hospitals mainly relies on hospital information systems. These medical information systems primarily include consultation modules based on traditional electronic medical records, mobile consultation systems, and medical record summary systems based on rules or simple natural language processing.

[0004] However, existing hospital information systems have significant shortcomings in terms of information integration and timeliness. On the one hand, when physicians conduct consultations, they need to manually search for key information in multiple systems such as the Hospital Information System (HIS), Electronic Medical Record (EMR), PACS, and LIS, as well as a large amount of medical records, examination results, and imaging data. This process is time-consuming and prone to omissions. Especially in emergency consultations, a search delay of several minutes to more than ten minutes may directly affect the patient's treatment. On the other hand, the medical records that traditional consultation requests rely on are outdated and cannot reflect the latest critical changes in the patient's condition in a timely manner. The current system also lacks the function of proactively integrating and pushing the latest information before the physician arrives at the consultation site. Summary of the Invention

[0005] This invention provides a consultation assistance method, device, electronic device, and storage medium to address the shortcomings of existing hospital information systems, such as low information integration efficiency and insufficient real-time performance. This not only forces physicians to spend time searching for key information across multiple systems, increasing clinical risks, but also restricts the efficiency of emergency and critical care consultations because the system cannot proactively and promptly synchronize the latest critical changes in the patient's condition.

[0006] This invention provides a consultation assistance method, comprising the following steps.

[0007] In response to a consultation request, the system acquires the target patient's historical medical records and real-time condition data from the consultation site, including screen images from the bedside monitoring device's display interface. The historical medical record data is semantically summarized to generate a historical medical record summary; The screen image is identified to extract the multimodal monitoring information of the target patient; Based on the historical medical record summary and the multimodal monitoring information, the target patient's target condition summary is obtained; The target medical condition summary is sent to the consulting physician's terminal.

[0008] According to a consultation assistance method provided by the present invention, the real-time medical data further includes voice description information describing the current vital signs of the target patient; The process of obtaining the target patient's condition summary based on the historical medical record summary and the multimodal monitoring information includes: Semantic understanding is performed on the voice description information to obtain real-time medical condition text; Based on the historical medical record summary, the multimodal monitoring information, and the real-time medical condition text, a target medical condition summary for the target patient is obtained.

[0009] According to a consultation assistance method provided by the present invention, obtaining a target patient's medical condition summary based on the historical medical record summary, the multimodal monitoring information, and the real-time medical condition text includes: Extract the first text features of the historical medical record summary and the second text features of the real-time medical condition text; Calculate the first attention weight of the first text feature relative to the second text feature, and the second attention weight of the first text feature relative to the multimodal monitoring information, respectively. Based on the first attention weight and the second attention weight, the target attention weight is determined; Based on the target attention weight, the first text feature is adjusted to obtain the third text feature; Based on the third text feature, the multimodal monitoring information, and the second text feature, a target patient's disease summary is obtained.

[0010] According to a consultation assistance method provided by the present invention, the step of performing semantic understanding on the voice description information to obtain real-time medical condition text includes: Convert the voice description information into a text stream; Extract key clinical event information and vital sign information from the text stream; Based on the key clinical event information and the vital sign information, a real-time medical condition text is generated.

[0011] According to a consultation assistance method provided by the present invention, the step of recognizing the screen image and extracting the multimodal monitoring information of the target patient includes: The screen image is segmented into regions to obtain a numerical display sub-image and a waveform display sub-image; Numerical parameters are extracted from the numerical display sub-image, and waveform morphology parameters are extracted from the waveform display sub-image; the waveform morphology parameters include the displacement amplitude, interval duration, and wave group morphology characteristics of the electrocardiogram waveform; Based on the waveform morphology parameters, a diagnostic conclusion including anomaly status indication is generated; Based on the numerical parameters and the diagnostic conclusions, the multimodal monitoring information of the target patient is determined.

[0012] According to a consultation assistance method provided by the present invention, generating a diagnostic conclusion including abnormal state indications based on the waveform morphology parameters includes: Based on the waveform morphology parameters, the time-series waveform features are determined, and the time-series waveform features are input into the sequence classification model to obtain the pathological probability distribution output by the sequence classification model. Based on the pathological category corresponding to the maximum probability value in the pathological probability distribution, the abnormal state prompt is determined, and the diagnostic conclusion is generated based on the abnormal state prompt. The sequence classification model is trained based on the sample waveform morphological parameters and the label diagnostic conclusions of the sample waveform morphological parameters.

[0013] According to a consultation assistance method provided by the present invention, the step of semantically summarizing the historical medical record data to generate a historical medical record summary includes: Extract key medical entities from the historical medical record data; Based on the temporal attributes of the key medical entities, the key medical entities are temporally associated to construct a disease evolution relationship chain; Based on the disease progression relationship chain, the historical medical record data is semantically summarized to generate the historical medical record summary.

[0014] According to a consultation assistance method provided by the present invention, the step of sending the target medical condition summary to the consulting physician's terminal includes: Retrieve auxiliary diagnostic opinions and recommended treatment plans that match the target disease summary from a preset medical knowledge base; Based on the auxiliary diagnostic opinions and recommended treatment plans related to the target disease summary, an enhanced consultation form is generated; The enhanced consultation form and the target disease summary are sent to the consulting physician's terminal.

[0015] According to a consultation assistance method provided by the present invention, after sending the enhanced consultation form and the target disease summary to the consulting physician's terminal, the method further includes: Receive a request for help from the consulting physician's terminal; the request for help includes the text of the difficult question in the enhanced consultation form; Based on the text of the difficult problem and the summary of the target condition, a tiered consultation request is generated; The tiered consultation request is sent to the superior physician terminal of the consulting physician terminal to obtain consultation assistance results.

[0016] According to a consultation assistance method provided by the present invention, the step of sending the target medical condition summary to the consulting physician's terminal includes: Determine the semantic correlation between each information point in the target disease summary and the real-time disease data; Information points whose semantic relevance exceeds a preset threshold are marked as high-priority content; The high-priority content is visually enhanced, and the target disease summary of the visually enhanced content is sent to the consulting physician's terminal.

[0017] According to a consultation assistance method provided by the present invention, the step of sending the visually enhanced target disease summary to the consulting physician's terminal includes: The visually enhanced summary of the target condition, the historical medical record data, and the multimodal monitoring information are sent to the consulting physician's terminal.

[0018] The present invention also provides a consultation auxiliary device, comprising the following units: The acquisition unit is used to acquire the target patient's historical medical record data and real-time condition data at the consultation site in response to a consultation request. The real-time condition data includes the screen image of the bedside monitoring device display interface. A semantic summarization unit is used to perform semantic summarization on the historical medical record data and generate a historical medical record summary. The recognition unit is used to recognize the screen image and extract the multimodal monitoring information of the target patient; A summary determination unit is used to obtain a target condition summary of the target patient based on the historical medical record summary and the multimodal monitoring information; The sending unit is used to send the target medical condition summary to the consulting physician's terminal.

[0019] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the consultation assistance method as described above.

[0020] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the consultation assistance method as described above.

[0021] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the consultation assistance method as described above.

[0022] The consultation assistance method, device, electronic device, and storage medium provided by this invention, in response to a consultation request, acquire the historical medical record data of the target patient and the real-time condition data at the consultation site, wherein the real-time condition data includes the screen image of the bedside monitoring device display interface; perform semantic summarization on the historical medical record data to generate a historical medical record summary; identify the screen image to extract the multimodal monitoring information of the target patient; obtain the target condition summary of the target patient based on the historical medical record summary and the multimodal monitoring information; and send the target condition summary to the consulting physician terminal. This invention, through semantic induction of historical medical record data and visual recognition of images from bedside monitoring equipment screens at consultation sites, can quickly extract and associate patients' past medical history and multimodal monitoring information from multi-source heterogeneous medical data. This allows fragmented medical information to be instantly fused into a target disease summary and pushed to the consulting physician in a timely manner, thereby improving the automation and timeliness of medical information integration. It effectively solves the technical problems of existing hospital information systems, which require physicians to manually search through a large number of medical records in multiple systems, resulting in time consumption and easy omissions, as well as the technical problems of traditional medical record updates being lagging behind and unable to reflect the latest critical changes in the patient's condition. This ensures that the consulting physician can accurately grasp the patient's core condition before arriving at the scene, thereby significantly shortening the response and decision-making time of consultations. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the consultation assistance method provided by the present invention.

[0025] Figure 2 This is a flowchart illustrating the process of determining a target disease summary provided by the present invention.

[0026] Figure 3 This is a schematic diagram of the process for determining multimodal monitoring information provided by the present invention.

[0027] Figure 4 This is a schematic diagram of the visual enhancement processing provided by the present invention.

[0028] Figure 5 This is a schematic diagram of the architecture of the consultation assistance system provided by the present invention.

[0029] Figure 6 This is a schematic diagram of the interactive feedback and hierarchical consultation process provided by the present invention.

[0030] Figure 7 This is a schematic diagram of the consultation auxiliary device provided by the present invention.

[0031] Figure 8 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0033] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and that the objects distinguished by "first," "second," etc., are generally of the same class.

[0034] Currently, in related technologies, the consultation process within hospitals mainly relies on hospital information systems. These medical information systems mainly include consultation modules based on traditional electronic medical records, mobile consultation systems, and medical record summary systems based on rules or simple natural language processing.

[0035] The consultation module based on traditional electronic medical records is currently the most common solution. The hospital information system integrates a consultation management module. The requesting physician fills out a standardized consultation request form on an electronic workstation. The form typically includes basic patient information (such as age, gender, and hospital number), a brief medical history, physical examination, auxiliary examinations, preliminary diagnosis, current treatment, and the purpose of the consultation. After the consultation invitation is sent, the consulting physician receives a notification on their workstation or mobile device. They then need to actively log into the electronic medical record system and review the patient's admission records, progress notes, laboratory reports, medical images, temperature charts, and medication orders, manually reading and integrating these documents to form an understanding of the patient's condition. Simultaneously, they need to contact the patient by phone or check on them at the bedside to obtain the latest information.

[0036] Some hospitals have developed mobile consultation systems, such as mobile consultation applications (APPs) or mini-programs, extending the consultation process from fixed workstations to mobile terminals, such as tablets and smartphones. Consulting physicians can receive consultation request notifications through these apps and access electronic medical records on their mobile devices. This improves the responsiveness of consulting physicians to some extent, but its essence remains the mobilization of "information retrieval," and it does not change the core pain point that consulting physicians still need to spend a considerable amount of time manually and proactively filtering and integrating fragmented medical information. The presentation of information is still a mere accumulation of raw data, lacking intelligent refinement and summarization. Furthermore, information regarding urgent changes in a patient's condition still needs to be obtained through telephone inquiries or bedside visits.

[0037] Furthermore, rule-based or simple natural language processing-based medical record summarization systems generate a summary text by identifying and extracting key fields (such as diagnosis, medication, and surgical records) and specific keywords from electronic medical records through research or rudimentary products. Natural language processing technology is used to automatically generate medical record summaries. However, these methods have weak abilities to understand context and semantics, and the generated summaries are often stiff, incoherent, and lack integration of the latest subjective information from the requesting physician and urgently needed information. They cannot accurately capture the logical relationships of the disease's evolution and the core contradictions of the current emergency situation, thus limiting their value in assisting clinical decision-making for urgent consultations.

[0038] To address the aforementioned problems, this invention provides a consultation assistance method. This method aims to resolve the technical issues in existing emergency medical consultation processes, such as the inability of consulting physicians to fully and in real-time grasp the patient's condition before arriving at the scene, and the low efficiency of information integration and the loss of crucial rescue time due to data barriers between hospital internal information systems and bedside monitoring equipment. The method in this embodiment is based on a multimodal instantaneous fusion large-scale model architecture, enabling automated data aggregation and intelligent analysis. Figure 1 This is a flowchart illustrating the consultation assistance method provided by the present invention, as shown below. Figure 1 As shown, the method includes the following: Step 110: In response to the consultation request, obtain the target patient's historical medical records and real-time condition data at the consultation site. The real-time condition data includes the screen image of the bedside monitoring device display interface.

[0039] Specifically, firstly, in response to a consultation request, the system acquires the target patient's historical medical records and real-time condition data from the consultation site. The real-time condition data may include screen images from the bedside monitoring equipment's display interface.

[0040] Here, consultation requests are typically triggered by the requesting physician on a bedside terminal, such as a mobile device or a doctor's workstation. When a critically ill patient requires assistance from other departments, the requesting physician issues a consultation request, and the system then initiates the data collection process.

[0041] The target patient refers to the patient who needs a consultation. After receiving a consultation request, the system will identify the patient's identity through a preset interface, such as a privacy anonymization module.

[0042] Obtaining the target patient's historical medical record data refers to the system automatically extracting all or key medical records of the target patient from the hospital's existing information system through multi-source data interface modules, such as EMR / HIS and LIS / PACS interfaces. This historical medical record data may include admission records, progress notes, surgical records, past medical history, physical examination records, laboratory test results, and medical imaging reports, etc. Past medical history includes medical history and allergy history, etc., but this embodiment of the invention does not specifically limit this.

[0043] Considering that bedside monitoring equipment at emergency scenes, such as electrocardiogram monitors and ventilators, is often manufactured by different companies and may not all be able to connect to the network in real time to upload waveform data to the central server, this embodiment uses visual reading to break down this data barrier. Specifically, real-time patient data includes the screen image of the bedside monitoring equipment's display interface. This acquisition process can be achieved by requesting a physician to use a handheld mobile terminal to photograph the monitor screen, or by using a fixed camera pre-installed at the bedside. The screen image intuitively reflects the target patient's current vital signs and their dynamic changes. The current vital signs include the patient's heart rate, blood pressure, and blood oxygen saturation, while the dynamic change areas include electrocardiogram waveforms and respiratory waveforms, which are not specifically limited in this embodiment.

[0044] Step 120: Semantic summarization is performed on the historical medical record data to generate a historical medical record summary.

[0045] Specifically, after acquiring historical medical record data, the original historical medical record data is often lengthy, sometimes containing hundreds of pages of documents. In urgent consultation scenarios, such as when a response is typically required within 5-10 minutes, the consulting physician cannot read through each document individually. Therefore, semantic summarization of the historical medical record data can be performed to generate historical medical record summaries.

[0046] Here, an artificial intelligence processing engine can be used, specifically a large language model fine-tuned with medical knowledge to perform semantic induction on historical medical record data. Semantic induction refers to the process by which the large language model extracts key information, removes duplicates, and logically organizes massive amounts of medical record text. The system will identify key medical entities strongly related to the current emergency symptoms, such as specific diagnostic names, abnormal test values, and key surgical time points. Generating a historical medical record summary involves reorganizing the above-summarized information into a concise, logically clear, and focused short text. The historical medical record summary aims to provide consulting physicians with a quick patient profile, for example: "Male patient, 65 years old, admitted to the hospital due to sudden chest pain for 3 hours, with a history of hypertension and diabetes. The admission electrocardiogram indicated acute anterior wall myocardial infarction, and an emergency PCI procedure has been performed."

[0047] Step 130: Recognize the screen image and extract the multimodal monitoring information of the target patient.

[0048] Specifically, after acquiring the screen image, the screen image can be recognized to extract the multimodal monitoring information of the target patient. Here, a medical image recognition engine can be used to recognize the screen image and extract the multimodal monitoring information of the target patient. Multimodal monitoring information refers to a comprehensive set of data obtained from the monitoring site that reflects the current physiological function and life activity trends of the target patient. Multimodal monitoring information may include the instrument modes and results of bedside monitoring equipment, such as core vital signs parameters like heart rate, blood oxygen saturation, respiratory rate, body temperature, and blood pressure, as well as visually discernible vital signs indicators such as pupillary response, skin color, and peripheral circulation that need attention in specific medical scenarios, display / view modes, and equipment operation / function modes. This embodiment of the invention does not specifically limit these aspects.

[0049] Here, screen image recognition can be performed by recognizing the numerical values ​​displayed on the screen image, or by extracting morphological features from the waveform image displayed on the screen image, or by recognizing both the numerical values ​​displayed on the screen image and extracting morphological features from the waveform image displayed on the screen image. This embodiment of the invention does not specifically limit the specifics of this method.

[0050] Step 140: Based on the historical medical record summary and the multimodal monitoring information, obtain the target patient's target condition summary.

[0051] Specifically, after obtaining historical medical record summaries and multimodal monitoring information, a target patient's condition summary can be obtained based on these summaries.

[0052] For example, when the historical medical record summary indicates that "the patient recently underwent percutaneous coronary intervention," while multimodal monitoring information shows "significant decrease in blood pressure and sharp increase in heart rate," the two types of information can be combined to infer the current potential high-risk clinical condition, such as suspected cardiac tamponade or acute cardiogenic shock.

[0053] Here, the target patient summary is a comprehensive clinical assessment conclusion that integrates the target patient's historical medical records and multimodal monitoring information. The target patient summary goes beyond a simple display of single-dimensional information; it is a comprehensive summary of the patient's condition, formed through multimodal data fusion and artificial intelligence clinical reasoning, possessing clear clinical guidance value.

[0054] In addition, real-time medical data can also include voice descriptions of the target patient's current vital signs. Accordingly, a target medical summary of the target patient can be obtained based on historical medical record summaries, multimodal monitoring information, and voice description information.

[0055] Step 150: Send the target medical condition summary to the consulting physician's terminal.

[0056] Specifically, after generating the target patient condition summary, it can be sent to the consulting physician's terminal. The system, through its communication module, preferably using a 5G network to ensure low latency and high security, instantly pushes the target patient condition summary to the invited consulting physician's terminal, such as a dedicated mobile app. At this time, the consulting physician can access the target patient condition summary on their terminal while en route to the department. This process achieves instantaneous information synchronization, allowing the consulting physician to grasp the core patient condition before arriving at the scene, thus enabling them to dedicate all their time upon arrival to emergency procedures rather than reviewing medical records again.

[0057] Here, auxiliary diagnostic opinions and recommended treatment plans related to the target disease summary can be obtained. Then, based on the auxiliary diagnostic opinions and recommended treatment plans related to the target disease summary, an enhanced consultation form is generated, and the enhanced consultation form and the target disease summary are sent to the consulting physician's terminal.

[0058] The method provided in this embodiment of the invention, in response to a consultation request, acquires the historical medical record data of the target patient and the real-time condition data at the consultation site, wherein the real-time condition data includes the screen image of the bedside monitoring device display interface; performs semantic summarization on the historical medical record data to generate a historical medical record summary; identifies the screen image to extract the multimodal monitoring information of the target patient; obtains the target condition summary of the target patient based on the historical medical record summary and the multimodal monitoring information; and sends the target condition summary to the consulting physician's terminal. This invention, through semantic induction of historical medical record data and visual recognition of images from bedside monitoring equipment screens at consultation sites, can quickly extract and associate patients' past medical history and multimodal monitoring information from multi-source heterogeneous medical data. This allows fragmented medical information to be instantly fused into a target disease summary and pushed to the consulting physician in a timely manner, thereby improving the automation and timeliness of medical information integration. It effectively solves the technical problems of existing hospital information systems, which require physicians to manually search through a large number of medical records in multiple systems, resulting in time consumption and easy omissions, as well as the technical problems of traditional medical record updates being lagging behind and unable to reflect the latest critical changes in the patient's condition. This ensures that the consulting physician can accurately grasp the patient's core condition before arriving at the scene, thereby significantly shortening the response and decision-making time of consultations.

[0059] In related technologies, under emergency rescue conditions, when the requesting physician's hands are occupied and time is of the essence, manually entering a description of the patient's condition via keyboard is both impractical and prone to errors. Existing systems lack efficient and convenient information entry methods.

[0060] Based on the above embodiments, the real-time medical data also includes voice description information describing the current vital signs of the target patient; Step 140 includes: Step 1401: Perform semantic understanding on the voice description information to obtain real-time medical condition text; Step 1402: Based on the historical medical record summary, the multimodal monitoring information, and the real-time medical condition text, obtain the target medical condition summary for the target patient.

[0061] Specifically, in actual emergency or consultation settings, in addition to objective instrument data, the subjective descriptions from on-site medical personnel often contain the most direct and urgent clues about the patient's condition, such as the patient's chief complaint and details of the sudden onset of symptoms. Therefore, in this embodiment, real-time patient data also includes voice descriptions of the target patient's current vital signs.

[0062] The voice description information reflects audio data about the patient's current emergency situation, recorded orally by the requesting physician or on-site medical staff. For example, the requesting physician might say into a mobile terminal: "The patient suddenly became agitated two hours after surgery, suspected of having cardiac tamponade."

[0063] Accordingly, semantic understanding can be performed on the spoken description information to obtain real-time medical condition text. This step utilizes Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) technologies. Real-time medical condition text reflects a structured text record describing the patient's current emergency condition, after transformation and refinement, and can be processed by a computer program. Real-time medical condition text is not merely a simple record of speech to text; it is text after key information has been extracted, such as converting spoken content into a text format like "Symptoms: restlessness; Suspected diagnosis: cardiac tamponade."

[0064] Then, based on historical medical record summaries, multimodal monitoring information, and real-time condition text, a target condition summary for the target patient can be obtained. In this step, the system integrates information from three different sources: historical medical record summaries representing past baselines, multimodal monitoring information representing objective monitoring data, and real-time condition text representing subjective on-site observations.

[0065] The method provided in this invention, on the one hand, by introducing voice description information and converting it into real-time medical condition text, can capture on-site emergencies that cannot be monitored by instruments alone. On the other hand, based on historical medical record summaries, multimodal monitoring information, and real-time medical condition text, a target medical condition summary for the target patient is obtained, making the final generated target medical condition summary more comprehensive and solving the problem of incomplete information that may be caused by a single data source, thereby assisting consulting physicians in more accurately judging the severity of the patient's condition.

[0066] Based on the above embodiments, Figure 2 This is a flowchart illustrating the process of determining a target disease summary provided by the present invention, such as... Figure 2As shown, step 1402 includes: Step 1402-1: Extract the first text features of the historical medical record summary and the second text features of the real-time medical condition text; Step 1402-2: Calculate the first attention weight of the first text feature relative to the second text feature, and the second attention weight of the first text feature relative to the multimodal monitoring information. Step 1402-3: Determine the target attention weight based on the first attention weight and the second attention weight; Step 1402-4: Based on the target attention weight, adjust the first text feature to obtain the third text feature; Step 1402-5: Based on the third text feature, the multimodal monitoring information, and the second text feature, obtain the target patient's disease summary.

[0067] Specifically, in order to ensure that the generated target disease summary does not omit important historical information and highlights the current critical situation, a fusion algorithm based on an attention mechanism is adopted.

[0068] First, the first textual features of the historical medical record summary and the second textual features of the real-time medical condition text are extracted. The first textual feature refers to the mathematical representation of the historical medical record summary in vector space, implicitly containing information about the target patient's long-term health background and underlying diseases. The second textual feature refers to the mathematical representation of the real-time medical condition text in vector space, implicitly containing information about the current sudden and urgent symptoms. Both the first and second textual features are typically obtained through pre-trained language models, such as Bidirectional Encoder Representations from Transformers (BERT) models; however, this embodiment of the invention does not specifically limit the specific methods used.

[0069] After obtaining the first text feature and the second text feature, the first attention weight of the first text feature relative to the second text feature and the second attention weight of the first text feature relative to the multimodal monitoring information can be calculated respectively.

[0070] The first attention weight is used to reflect key information in the historical medical record summary that is relevant to the real-time medical condition text. For example, when the real-time medical condition text mentions chest pain symptoms, descriptions related to coronary artery disease in the historical medical record summary will receive higher weight. The second attention weight is used to reflect information in the historical medical record summary that is closely related to real-time physiological monitoring data. For example, when multimodal monitoring information shows ST-segment elevation, records related to the patient's past myocardial infarction will be given higher attention.

[0071] After obtaining the first and second attention weights, a target attention weight can be determined based on them. The target attention weight reflects the final importance of each information point in the historical medical record summary after comprehensively considering both verbal description information and multimodal monitoring information. This step aims to integrate the first and second attention weights to ensure that the screening and weighting process of the historical medical record summary accurately reflects the actual needs of the current emergency situation.

[0072] After obtaining the target attention weights, the first text features can be adjusted based on these weights to obtain the third text features. These third text features reflect the feature information of the historical medical record summary after weighting the voice description information and multimodal monitoring information. During this process, feature information of historical medical record summaries irrelevant to the current emergency scenario is suppressed, while feature information of historical medical record summaries closely related to the current critical situation is significantly enhanced.

[0073] After obtaining the third text features, a summary of the target patient's condition can be derived based on the third text features, multimodal monitoring information, and second text features. For example, the third text features, multimodal monitoring information, and second text features can be concatenated to obtain fused features, which can then be decoded by a decoder to obtain a summary of the target patient's condition.

[0074] Here, the fusion features are used to reflect the target patient's current condition, integrating historical context, objective on-site vital signs data, and subjective on-site speech descriptions into a comprehensive mathematical representation. The decoder refers to the generative component in the multimodal large-scale model architecture. The decoder understands the semantic and numerical relationships in the input fusion features and, following the language conventions of medical documents, generates fluent and accurate natural language text word by word using an autoregressive approach.

[0075] The method provided in this invention dynamically adjusts the first text features by calculating attention weights, thereby achieving historical information filtering guided by the current condition. The generated target condition summary can automatically focus on the historical causes that led to the current crisis, avoiding interference from irrelevant historical information. This greatly improves the signal-to-noise ratio when consulting physicians read the target condition summary, enabling them to quickly grasp the core cause of the disease.

[0076] Based on the above embodiments, step 1401 includes: Step 1401-1: Convert the voice description information into a text stream; Step 1401-2: Extract key clinical event information and vital sign information from the text stream; Step 1401-3: Generate real-time medical condition text based on the key clinical event information and the vital signs information.

[0077] Specifically, in emergency consultation scenarios, doctors' dictation is often rapid, information-dense, and may include non-medical terminology. The method in this embodiment aims to accurately extract structured real-time patient information from this complex speech stream.

[0078] First, the spoken description information can be converted into a text stream. This step is typically accomplished using an automatic speech recognition engine. The system receives the spoken description information entered by the physician and transcribes it into a text stream using acoustic and language models. The text stream refers to the unstructured, continuous string output by the automatic speech recognition engine. For example, the text stream obtained from the original speech might be: "That patient isn't doing well right now; we couldn't measure his blood pressure just now, and he's sweating profusely, he might be in shock." After obtaining the text stream, key clinical event information and vital sign information can be extracted from it. This step utilizes Named Entity Recognition (NER) and relation extraction techniques from Natural Language Processing.

[0079] Here, key clinical event information refers to event descriptions in the text stream that have clear clinical diagnostic significance or indicate a rapid change in the patient's condition. For example, phrases like "not good condition," "profuse sweating," or "shock" extracted from the text stream.

[0080] Vital signs information refers to the specific values ​​or states of a patient's physiological indicators described in the text stream. For example, blood pressure or unmeasurable information extracted from the text stream; this embodiment of the invention does not specifically limit this.

[0081] To achieve accurate extraction, a dictionary or knowledge graph containing emergency medical terminology can be pre-built to help the model filter out colloquial words such as "that" and "I see him" that have no actual clinical significance.

[0082] Finally, real-time medical condition text can be generated based on key clinical event information and vital sign information. The system reorganizes the extracted key clinical event information and vital sign information according to a preset medical expression template or semantic logic. Here, the real-time medical condition text reflects the short text after cleaning, denoising, and structured reorganization. The real-time medical condition text removes the redundancy of colloquial language and directly states the core condition. For the above example, the generated real-time medical condition text could be: "[Chief Complaint / Symptom] Profuse sweating; [Vital Signs] Blood pressure undetectable; [Suspected Diagnosis] Shock."

[0083] In related technologies, existing systems typically only provide raw images or numerical lists for dynamic physiological signals such as electrocardiograms, cardiac monitoring waveforms, and vital sign trend charts. Consulting physicians need to possess professional expertise to interpret these images, as more intuitive interpretation results cannot be obtained through auxiliary machines or knowledge retrieval.

[0084] Based on the above embodiments, Figure 3 This is a schematic diagram of the process for determining multimodal monitoring information provided by the present invention, such as... Figure 3 As shown, step 130 includes: Step 1301: Perform region segmentation on the screen image to obtain a numerical display sub-image and a waveform display sub-image; Step 1302: Extract numerical parameters from the numerical display sub-image and extract waveform morphology parameters from the waveform display sub-image; the waveform morphology parameters include the displacement amplitude, interval duration, and wave group morphology characteristics of the electrocardiogram waveform; Step 1303: Based on the waveform morphology parameters, generate a diagnostic conclusion including an abnormal state indication; Step 1304: Based on the numerical parameters and the diagnostic conclusions, determine the multimodal monitoring information of the target patient.

[0085] Specifically, firstly, the screen image can be segmented into numerical display sub-images and waveform display sub-images. This step utilizes object detection models, such as YOLO and Faster R-CNN, to perform layout analysis on the input screen image.

[0086] The numerical display sub-image reflects the rectangular area on the screen specifically designed for displaying digital readings, such as the area in the upper right corner showing HR 120 or SpO2 98%. The waveform display sub-image reflects the area on the screen specifically designed for displaying dynamic scan trajectories, such as the area in the center of the screen showing a green ECG waveform or a yellow respiratory waveform. The system automatically crops these different functional areas based on the common layout characteristics of the monitor, so that different algorithms can be used for subsequent processing.

[0087] Then, numerical parameters can be extracted from the numerical display sub-image, and waveform morphology parameters can be extracted from the waveform display sub-image. Among them, waveform morphology parameters include the displacement amplitude, interval duration, and wave group morphology characteristics of the ECG waveform.

[0088] For numerical display sub-images, Optical Character Recognition (OCR) technology can be used to identify the numbers and units to obtain numerical parameters. For waveform display sub-images, signal extraction or skeletonization algorithms from computer vision are used to restore the pixels in the waveform display sub-images to time-series signals, and then morphological indices are calculated.

[0089] Here, waveform morphology parameters are used to reflect the quantitative characteristics with electrophysiological significance extracted from the waveform display sub-image. The displacement amplitude of the ECG waveform reflects the vertical distance of each segment in the ECG, such as the ST segment and T wave, relative to the isoelectric line; for example, a 2mm upward shift of the ST segment corresponds to 0.2mV. Interval duration reflects the time interval between various waveforms in the ECG, such as the PR interval, QRS complex duration, and QT interval, usually measured in milliseconds. Wave group morphology features reflect the overall geometry of the waveform, such as whether the QRS complex is wide and deformed, whether there are notches, and whether the P wave is absent; however, this embodiment of the invention does not specifically limit these features.

[0090] After obtaining the waveform morphology parameters, diagnostic conclusions, including abnormal state indications, can be generated based on these parameters. The abnormal state indications reflect specific manifestations of the waveform morphology parameters deviating from the normal range, such as significant ST segment elevation and wide, distorted QRS complexes.

[0091] Here, diagnostic conclusions are used to reflect preliminary pathological judgments derived from abnormal cues, such as those suggesting acute myocardial infarction (ST-segment elevation) or ventricular tachycardia.

[0092] Finally, multimodal monitoring information for the target patient can be determined based on numerical parameters and diagnostic conclusions. Multimodal monitoring information reflects a complete set of objective vital signs data for the target patient at the current moment. For example, {heart rate: 120, blood pressure: 80 / 50, diagnosis: ventricular tachycardia}.

[0093] The method provided in this invention performs region segmentation on a screen image to obtain a numerical display sub-image and a waveform display sub-image. Numerical parameters are extracted from the numerical display sub-image, and waveform morphological parameters are extracted from the waveform display sub-image. The waveform morphological parameters include the displacement amplitude, interval duration, and wave group morphological characteristics of the electrocardiogram waveform. Based on these parameters, a diagnostic conclusion containing abnormal indications is generated. By fusing numerical parameters and diagnostic conclusions, structured multimodal monitoring information that combines objective data with clinical semantics is formed. This transforms the original image into a high-information-density summary that can directly assist in decision-making, effectively overcoming the deficiency of existing systems that can only provide raw data and cannot provide machine interpretation results.

[0094] Based on the above embodiments, step 1303 includes: Step 1303-1: Determine the time-series waveform features based on the waveform morphology parameters, input the time-series waveform features into the sequence classification model, and obtain the pathological probability distribution output by the sequence classification model; Step 1303-2: Based on the pathological category corresponding to the maximum probability value in the pathological probability distribution, determine the abnormal state prompt, and generate the diagnostic conclusion based on the abnormal state prompt; The sequence classification model is trained based on the sample waveform morphological parameters and the label diagnostic conclusions of the sample waveform morphological parameters.

[0095] Specifically, for complex time-series waveform signals, traditional rule-based judgments often fail to cover all pathological conditions. Therefore, this embodiment introduces a deep learning sequence model.

[0096] Correspondingly, the time-series waveform characteristics can be determined based on the waveform morphology parameters, and the time-series waveform characteristics can be input into the sequence classification model to obtain the pathological probability distribution output by the sequence classification model.

[0097] Among them, the time-series waveform features are used to reflect the feature vector formed by serializing the discrete waveform points or parameters extracted from the waveform display sub-image, and the time-series waveform features are used to reflect the dynamic law of waveform change over time.

[0098] Here, the sequence classification model can be a Long Short-Term Memory (LSTM) network or a Transformer model; this embodiment of the invention does not specifically limit it. The sequence classification model is a deep neural network trained on a large amount of medical waveform data, and it is capable of capturing long-distance temporal dependencies.

[0099] Here, the pathological probability distribution is used to reflect the confidence vector of the sequence classification model in predicting that the current input waveform belongs to various preset pathological categories, such as sinus rhythm, atrial fibrillation, ventricular fibrillation, atrioventricular block, etc.

[0100] After obtaining the pathological probability distribution, the abnormal state prompt can be determined based on the pathological category corresponding to the highest probability value in the distribution, and a diagnostic conclusion can be generated based on the abnormal state prompt. For example, if "ventricular tachycardia" has the highest probability, then the abnormal state prompt is determined to be "ventricular tachycardia". Subsequently, the system encapsulates this prompt into a diagnostic conclusion in natural language, such as "Waveform analysis prompt: ventricular tachycardia".

[0101] The training steps for the sequence classification model include: First, sample waveform morphological parameters and labeled diagnostic conclusions of the sample waveform morphological parameters can be collected.

[0102] Then, the temporal waveform features of the samples can be determined based on the sample waveform morphology parameters. These features are then input into the initial sequence classification model to obtain the predicted pathological probability distribution output by the model. Based on the predicted pathological category corresponding to the maximum probability value in the predicted pathological probability distribution, a predicted abnormal state indication is determined, and a predicted diagnostic conclusion is generated based on this indication.

[0103] After obtaining the predicted diagnostic conclusion, the classification loss can be determined based on the difference between the predicted and labeled diagnostic conclusions. The model parameters of the initial sequence classification model are then updated based on this loss to obtain the sequence classification model. It is understandable that during the training phase, a large number of sample waveform morphological parameters and their corresponding labeled diagnostic conclusions are used to supervise the learning of the mapping rules from waveform features to pathological conclusions in the initial sequence classification model.

[0104] Understandably, the greater the difference between the predicted diagnostic conclusion and the labeled diagnostic conclusion, the greater the classification loss; conversely, the smaller the difference between the predicted diagnostic conclusion and the labeled diagnostic conclusion, the smaller the classification loss.

[0105] The method provided in this invention enables a sequence classification model to automatically learn from time-series waveform features and identify complex pathological patterns that are difficult to cover by traditional rules, thereby achieving accurate classification and probabilistic assessment of abnormal states. This not only improves the objectivity and reliability of the diagnostic process but also significantly enhances the efficiency of waveform data parsing and diagnostic prompt generation, thereby improving the accuracy and reliability of consultation assistance.

[0106] Based on the above embodiments, step 120 includes: Step 1201: Extract key medical entities from the historical medical record data; Step 1202: Based on the time attributes of the key medical entities, perform temporal association on the key medical entities to construct a disease evolution relationship chain; Step 1203: Based on the disease progression relationship chain, perform semantic induction on the historical medical record data to generate the historical medical record summary.

[0107] Specifically, firstly, key medical entities can be extracted from historical medical record data. Here, an entity extraction model can be used to extract key medical entities from historical medical record data.

[0108] Here, key medical entities refer to the lexical units in historical medical record data that carry core medical information. Key medical entities may include disease names, symptom descriptions, examination items, examination results, treatment methods, drug names, and related time points, etc. This embodiment of the invention does not specifically limit these.

[0109] After obtaining the key medical entities, the system can establish temporal associations based on their time attributes to construct a chain of disease progression relationships. The system arranges and associates the key medical entities in chronological order according to their corresponding timestamps.

[0110] Here, the disease progression chain is used to reflect the causal development path of the patient's condition over time. For example, the disease progression chain can be: [Time T1: Sudden chest pain] -> [Time T2: Hospitalization] -> [Time T3: Diagnosis of myocardial infarction] -> [Time T4: Surgery] -> [Time T5: Postoperative blood pressure drop].

[0111] Finally, based on the disease progression chain, semantic summaries can be generated from historical medical record data. Here, the disease progression chain can be input into a summary generation model, such as a Transformer-based model. The summary generation model can organize coherent language based on the key nodes in the disease progression chain to generate a concise historical medical record summary. This historical medical record summary serves to reflect a highly condensed, chronologically logical overview of the patient's medical history.

[0112] Furthermore, to enhance the clinical value of the constructed disease progression chain, the system deeply integrates the dynamic trends of laboratory test values ​​and the patient's background characteristics during time-series correlation. Specifically, the system tracks the fluctuations of key laboratory test values, such as troponin, white blood cell count, and hemoglobin, over time, mapping these dynamic changes to disease progression or diagnostic criteria. For example, a sustained increase in troponin levels is associated with the diagnostic node of acute myocardial infarction progression. Simultaneously, the system attaches the patient's past medical history, comorbidities (such as long-term hypertension and diabetes), and key medication records as background context nodes to corresponding positions on the timeline. This constructs a composite disease progression chain that not only includes discrete events but also comprehensively integrates numerical trends and background characteristics, enabling accurate revelation of disease outcomes and treatment responses during subsequent summary generation.

[0113] The method provided in this invention extracts key medical entities and constructs a disease progression relationship chain based on their temporal attributes, enabling the automatic extraction of a disease development trajectory with clear temporal logic from loosely structured historical medical record data. This not only achieves a coherent and structured presentation of the patient's disease course but also makes the subsequent semantic induction-generated medical record summaries more logical, significantly improving the efficiency of medical record information integration and the depth of disease understanding.

[0114] Based on the above embodiments, step 150 includes: Step 1501: Retrieve auxiliary diagnostic opinions and recommended treatment plans that match the target disease summary from a preset medical knowledge base; Step 1502: Based on the auxiliary diagnostic opinions and recommended treatment plans related to the target disease summary, generate an enhanced consultation form; Step 1503: Send the enhanced consultation form and the target disease summary to the consulting physician's terminal.

[0115] Specifically, the system can retrieve auxiliary diagnostic opinions and recommended treatment plans that match the target disease summary from a pre-defined medical knowledge base. The system uses the target disease summary as a query vector to perform a similarity search within the vectorized medical knowledge base. The pre-defined medical knowledge base includes clinical guidelines, expert consensus, and a database of classic cases.

[0116] Here, auxiliary diagnostic opinions refer to suggestions on possible diagnostic directions for the current condition based on similar cases or guideline entries found in the search. For example, if a similar case is found to be "cardiac tamponade," then this condition should be ruled out. Recommended treatment plans refer to standardized management suggestions for this suspected diagnosis, such as recommending immediate ultrasound-guided puncture and drainage.

[0117] Then, based on the auxiliary diagnostic opinions and recommended treatment plans related to the target disease summary, an enhanced consultation form is generated. The enhanced consultation form is a composite clinical decision support document that integrates a disease summary, differential diagnostic suggestions, and standardized treatment recommendations. It systematically presents the patient's disease progression, provides evidence-based diagnostic guidance, and links to corresponding standardized treatment pathways, thereby improving the completeness of consultation information, the effectiveness of decision support, and the timeliness of clinical intervention.

[0118] Finally, the enhanced consultation form and the target disease summary can be sent to the consulting physician's terminal.

[0119] The method provided in this invention retrieves matching auxiliary diagnostic opinions and recommended treatment plans from a vectorized medical knowledge base based on a target disease summary, achieving efficient and accurate retrieval of evidence-based medicine knowledge and clinical guidelines. The generated enhanced consultation form not only integrates the patient's own disease context but also systematically links relevant differential diagnostic prompts and standardized treatment suggestions, thereby significantly improving the completeness of consultation information and the evidence-based level of decision support.

[0120] In related technologies, for difficult cases that the consulting physician cannot resolve, due to the inability to synchronize information, it is necessary to contact the consulting physician's superior again to communicate the patient's condition, which is a cumbersome process.

[0121] Based on the above embodiments, step 1503 further includes: Step 1503-1: Receive a request for help from the consulting physician's terminal; the request for help includes the text of the difficult question in the enhanced consultation form; Step 1503-2: Based on the text of the difficult problem and the summary of the target condition, generate a tiered consultation request; Step 1503-3: Send the tiered consultation request to the superior physician terminal of the consulting physician terminal to obtain the consultation assistance result.

[0122] Specifically, in emergency consultations, extremely complex cases are often encountered that require the intervention of higher-level experts. This embodiment provides a convenient upgrade channel.

[0123] First, it can receive assistance requests from consulting physicians' terminals, including textual requests for complex issues from enhanced consultation reports. When a frontline consulting physician finds they cannot handle a case independently after reviewing a consultation report or arriving at the scene, they can click the "Request Assistance from Higher Authorities" button on their terminal.

[0124] Here, the "Help Request" command reflects a control signal initiated by a frontline physician requesting intervention from a higher-level expert. The "Difficult Question Text" refers to a frontline physician's specific description of the current predicament, such as "The patient's circulation cannot be maintained, the vasopressor dosage has been reached, requesting guidance on the next steps." This can be obtained through text input or voice-to-text conversion.

[0125] Then, based on the text of the difficult question and the summary of the target condition, a tiered consultation request is generated. The system automatically packages the original summary of the target condition with the text of the newly raised difficult question from the physician. The tiered consultation request refers to a high-priority task package containing complete contextual information.

[0126] Finally, the tiered consultation request can be sent to the superior physician's terminal of the consulting physician's terminal to obtain consultation assistance results.

[0127] It should be noted that the consultation physician terminal, serving as the physician's operating interface, not only clearly displays the generated target disease summary but also features interactive functions. Physicians can directly contact the requesting department or patient's bedside via a one-click call function for quick communication of details; they can access original PACS images or detailed laboratory reports for in-depth analysis by clicking links to view historical medical records; and after deciding to accept the patient, they can confirm the appointment and estimated arrival time by clicking the confirmation button, ensuring the requesting party is informed. Furthermore, after or during the consultation, physicians can verbally express preliminary diagnostic opinions via voice input, and the system supports voice-to-text transcription for manual editing, thereby quickly generating consultation records.

[0128] Specifically, upon receiving a tiered consultation request, senior physicians can assess the situation using their designated terminal and generate guidance via voice or text input. When the system receives voice input from a senior physician, it automatically performs speech recognition and semantic correction, transing it into structured text suggestions; if text input is received, it is directly formatted. Subsequently, the system encapsulates these processed guidance opinions as consultation support results and pushes them in real-time to the terminal of the physician who initiated the request. This achieves a closed-loop remote collaboration of junior physician asking questions and senior physician providing answers, ensuring that on-site physicians receive timely and authoritative treatment plans.

[0129] The method provided in this invention constructs an instant, hierarchical medical collaboration closed loop. By automatically packaging the target patient's condition summary and the text of complex issues, senior experts can quickly grasp the core of the problem without re-inquiring about the patient's medical history, greatly improving the efficiency of remote guidance. This mechanism can fully mobilize medical resources when dealing with complex and difficult cases, providing patients with a higher level of treatment and protection.

[0130] Based on the above embodiments, Figure 4 This is a schematic diagram of the visual enhancement processing provided by the present invention, such as... Figure 4 As shown, step 150 includes: Step 150-1: Determine the semantic correlation between each information point in the target disease summary and the real-time disease data; Step 150-2: Mark the information points whose semantic relevance exceeds a preset threshold as high-priority content; Step 150-3: Perform visual enhancement processing on the high-priority content, and send the visually enhanced target disease summary to the consulting physician's terminal.

[0131] Specifically, in the fast-paced emergency environment, the way information is presented directly impacts the efficiency of physicians' reception. This system achieves this goal through a large-scale multimodal instantaneous fusion model. This model, as the core engine, is responsible for deeply fusing three types of heterogeneous data: historical static data (historical medical record data summarized by a large language model), current subjective descriptions (real-time patient condition data obtained from speech-to-text conversion), and current objective vital signs (multimodal monitoring information obtained from image recognition monitors / ECGs). To ensure the medical safety of the generated content, the fused target patient condition summary is further verified by a result validation module to eliminate potential logical contradictions or misleading information.

[0132] When generating the final display content, the system follows the principle of highlighting key points and performs the following steps: First, the semantic relevance between each information point in the target medical summary and the real-time medical data can be determined. Semantic relevance reflects the explanatory power or relevance of each information point in the target medical summary to the real-time medical data of the current critical state. For example, if the real-time data shows "hypotensive shock," then the relevance of the information point "large intraoperative bleeding" in the target medical summary will be very high, while the relevance of the information point "previous history of athlete's foot" will be extremely low.

[0133] Then, information points with semantic relevance exceeding a preset threshold can be marked as high-priority content. High-priority content reflects key information segments in the target medical summary that can directly explain, warn of, or relate to the current critical situation. For example, content prioritizes highlighting the primary purpose of this consultation, as well as the patient's background, baseline data, and auxiliary examination results most relevant to the emergency.

[0134] Finally, high-priority content can be visually enhanced, and the enhanced summary of the patient's condition can be sent to the consulting physician's terminal. Visual enhancement techniques include bolding the font, highlighting it in red, brightening the background, increasing the font size, or moving it to the top of the summary for display. This embodiment of the invention does not specifically limit these techniques.

[0135] Here, the visually enhanced summary of the patient's condition is used to reflect the final text display form after layout optimization and highlighting of key points.

[0136] The method provided in this invention determines the semantic correlation between each information point in the target medical condition summary and real-time medical condition data, marks information points with semantic correlation exceeding a preset threshold as high-priority content, performs visual enhancement processing on the high-priority content, and sends the visually enhanced target medical condition summary to the consulting physician's terminal. This allows the physician's gaze to be automatically guided to the most critical information the moment they scan the screen, thereby establishing an understanding of the core of the medical condition in a very short time, significantly reducing the physician's cognitive load and improving the agility of emergency decision-making.

[0137] Based on the above embodiments, step 150-3, which involves sending the visually enhanced target disease summary to the consulting physician's terminal, includes: Steps 150-31: The target disease summary of the visual enhancement processing, the historical medical record data, and the multimodal monitoring information are sent to the consulting physician terminal.

[0138] Specifically, the visually enhanced summary of the target patient's condition, historical medical record data, and multimodal monitoring information can be sent to the consulting physician's terminal. That is, while sending the visually enhanced summary of the target patient's condition to the consulting physician's terminal, the system will also package and send the historical medical record data and multimodal monitoring information used as the basis for generation.

[0139] On the display interface of the consulting physician's terminal, the visually enhanced summary of the target patient's condition is displayed first, serving as the physician's first visual focus. Historical medical record data and multimodal monitoring information can be folded and stored in the details area below the target patient's summary, or implicitly stored as background data on the current page. Alternatively, the visually enhanced summary of the target patient's condition, historical medical record data, and multimodal monitoring information can be displayed simultaneously, but this embodiment of the invention does not specifically limit this.

[0140] Furthermore, to facilitate physicians in verifying the accuracy of the summary, the system establishes a hyperlink jump or floating preview mechanism between high-priority content, namely the visually enhanced target disease summary, and the corresponding historical medical record data and multimodal monitoring information.

[0141] For example, when the target medical condition summary displays the highlighted red text "sudden onset of ventricular tachycardia," the physician can click on the text, and the consulting physician's terminal interface will immediately expand or pop up to display the corresponding multimodal monitoring information segment, specifically a screenshot of the abnormal electrocardiogram waveform captured by the monitor at the corresponding moment. As another example, when the target medical condition summary displays the bold text "history of aspirin allergy," clicking on the text will directly jump to the specific allergy history record document page in the historical medical record data.

[0142] In this way, the embodiments of the present invention not only provide physicians with a summary of the target condition, but also simultaneously provide objective evidence supporting the conclusion, namely historical medical record data and multimodal monitoring information, thereby constructing a data interaction mechanism that combines conclusions and evidence. This mechanism, while ensuring the efficiency of information transmission in emergency scenarios, strengthens the traceability and security of medical decisions, helps enhance physicians' trust in the system's output, and reduces medical risks that may arise from the uncertainty of model-generated content.

[0143] Based on any of the above embodiments Figure 5 This is a schematic diagram of the architecture of the consultation assistance system provided by the present invention, as shown below. Figure 5 As shown, this system mainly includes an application layer (responsible for multimodal data collection), a preprocessing layer (responsible for privacy and compliance preprocessing), an AI core layer (responsible for multimodal intelligent fusion reasoning), an output and parallel retrieval layer (responsible for result generation and knowledge enhancement), and an interaction and upgrade closed-loop layer (responsible for difficult case triage and feedback).

[0144] Specifically, the multimodal instantaneous fusion big data model emergency consultation process first collects multimodal data at the request end. The emergency consultation request is triggered and initiated in three parallel streams: historical medical records / auxiliary examination results are captured through the HIS / EMR system interface to obtain text background data; the voice input module receives the requesting physician's dictated symptoms to generate an audio stream; and the image acquisition module captures raw medical images from the monitor / ECG. Then, the data enters the preprocessing layer to perform privacy and compliance preprocessing. The above data is then fed into the privacy de-identification gateway for de-identification processing to generate a compliant data package. Finally, the AI ​​core layer performs multimodal intelligent fusion inference, and the compliant data package is distributed and processed. Structured text features are obtained by extracting chief complaint / reason for emergency consultation through the ASR speech recognition engine, and structured vital sign data is obtained by extracting vital sign data / waveform features through the image recognition engine. These features, along with historical data, are input into the LLM large language model inference engine. The Prompt command is used to fuse historical data, current status, and vital signs to achieve multimodal semantic alignment and summary generation. Finally, the output and parallel retrieval stage is entered to perform result generation and knowledge enhancement. First, the target disease summary is generated and processed by the result verification module. The content includes patient profile / critical value highlighting / urgent consultation purpose. It is submitted to the requesting physician for confirmation. After confirmation, it enters the SplitNode for parallel processing: one path is transmitted through a 5G encrypted transmission channel, and the other path calls the RAG (Retrieval-Augmented Generation) retrieval engine based on the medical knowledge base to retrieve emergency diagnosis / treatment plan / source basis and generate knowledge enhancement package. Finally, the data is merged and packaged in the MergeNode.

[0145] The EMR / HIS interface automatically retrieves all structured and unstructured medical record data of patients from the hospital information system, such as admission records, progress notes, and surgical records. The LIS / PACS interface acquires laboratory test results and medical imaging reports. The bedside device image acquisition interface collects real-time patient data via wired or wireless (preferably Wi-Fi 6 or 5G IoT). This includes screen images from bedside monitoring devices, such as monitors, ventilators, and infusion pumps.

[0146] The voice input interface uses wearable devices or mobile terminal microphones to collect the patient's urgent changes in condition as described by the requesting physician, as part of the real-time condition data.

[0147] Before the data enters the AI ​​core layer, the privacy desensitization module automatically identifies and removes direct identifiers such as patient names and ID numbers, replacing them with unique identifiers within the hospital to ensure compliance with the Personal Information Protection Law and medical data security management regulations.

[0148] Based on any of the above embodiments Figure 6This is a schematic diagram of the interactive feedback and hierarchical consultation process provided by the present invention, such as... Figure 6 As shown, the process begins with the data merging and packaging stage. The system integrates the generated consultation information with the screen images and then distributes it to the receiving end. On one hand, the system pushes the information to the consulting physician's terminal, displaying the target condition summary, knowledge base, and screen images on the terminal interface. At this point, the physician determines whether the case is complex based on the condition. If not, standard procedures are followed, and the consultation record is improved through voice guidance / manual modification. After electronic signature confirmation, the final consultation opinion is generated. If the case is complex and requires assistance from a higher level, the physician enters the text of the difficult question. The text of the difficult question is combined with the target condition summary to generate a tiered consultation request, and an emergency reminder notification is issued. On the other hand, the system simultaneously pushes the aforementioned emergency reminder notification to the higher-level physician's terminal. The higher-level physician's terminal also displays the target condition summary, knowledge base, and screen images. After reviewing the specific reason for the request for assistance, the higher-level physician provides consultation assistance results, and the process ends.

[0149] Based on any of the above embodiments, this embodiment provides an overall application flow of an emergency consultation assistance system and method based on a multimodal instantaneous fusion large model. This system can be widely applied in general departments, emergency rooms, and intensive care units. Its core capability lies in its ability to proactively aggregate, understand, and generate multi-source heterogeneous medical data, combined with timely voice input, and, while strictly protecting patient privacy, integrate and generate a highly refined, real-time intelligent case summary tailored to emergency consultation scenarios. Furthermore, the system uses RAG technology to search the knowledge base, providing urgent condition analysis and treatment plans. This allows consulting physicians to determine their consultation opinions from an interface containing the target condition summary, screen images, and medical knowledge base references, and to decide whether to trigger emergency assistance from senior physicians. After the consultation, physicians can input their consultation opinions via voice and can manually rewrite them. Finally, electronic signature confirmation achieves a closed-loop feedback process, significantly improving the efficiency of physicians' clinical work.

[0150] The overall workflow of the system includes the following steps: First, the requesting physician selects an urgent consultation and target department on the bedside terminal to trigger a consultation request; then, the system automatically obtains the patient's complete electronic medical record through a data interface and calls the medical record summary generation engine to generate a refined medical record summary with privacy anonymization within seconds; simultaneously, the requesting physician verbally describes the patient's latest condition changes, and the speech recognition and understanding engine works in real time to generate an anonymized "real-time condition snapshot"; next, the requesting physician uses the terminal camera to photograph the bedside monitor screen, the electrocardiogram printed by the electrocardiograph, or the ventilator screen, and the medical image recognition engine analyzes the images and outputs anonymized structured data; based on this, The instantaneous fusion module integrates the outputs of the above three parts to generate a final consultation request form containing three major sections: "historical static summary," "real-time snapshot," and "image conclusion." The system further performs RAG knowledge retrieval based on the intelligent summary to provide diagnostic, treatment, and knowledge basis, and pushes the final consultation form instantly, quickly, securely, and confidentially to the mobile consultation terminal APP of the designated consulting physician via the 5G network. Finally, the consulting physician can read this intelligent consultation form on their mobile phone while en route to the site, understand the patient's condition in advance, and consider diagnostic and treatment plans. For difficult cases requiring assistance from senior physicians, an emergency notification can be triggered in a timely manner to the senior physician's terminal, realizing a rapid, accurate, and hierarchical collaboration mechanism.

[0151] The beneficial effects of this invention are mainly reflected in four aspects. First, by using multimodal artificial intelligence to deeply understand, refine, and summarize the raw data, the time for integrating medical record information is greatly shortened. The information retrieval and integration process, which originally required consulting physicians to spend 10 to 15 minutes or even longer, is compressed to a near real-time, second-level process completed by the system. This allows consulting physicians to grasp the patient's basic condition in just 1 to 2 minutes en route, significantly improving time efficiency. Second, real-time information is achieved. By using voice input and image recognition to obtain the patient's latest status and the main purpose of the emergency consultation, the problem of information lag in traditional application forms is solved, making the consultation... Decision-making can be based on the latest and most accurate on-site information; thirdly, it improves the convenience and humanization of medical services, providing a simplified voice information input method for requesting physicians in high-pressure emergency environments, reducing the time spent on manual input, and providing clinical decision-making through knowledge retrieval, which greatly facilitates clinical scenarios; fourthly, it brings potential improvements in medical quality and safety, reducing the risk of human oversight or misjudgment caused by time constraints and complex information through artificial intelligence assistance, standardizing information formats helps reduce communication errors, and the tiered consultation and collaboration mechanism for difficult cases improves teamwork efficiency, ultimately directly translating into an increase in the success rate of patient treatment.

[0152] The consultation assistance device provided by the present invention is described below. The consultation assistance device described below and the consultation assistance method described above can be referred to in correspondence.

[0153] Based on any of the above embodiments, the present invention provides a consultation auxiliary device. Figure 7 This is a schematic diagram of the consultation auxiliary device provided by the present invention, as shown below. Figure 7 As shown, the device includes: The acquisition unit 710 is used to acquire the target patient's historical medical record data and the real-time condition data at the consultation site in response to a consultation request. The real-time condition data includes the screen image of the bedside monitoring device display interface. Semantic summarization unit 720 is used to perform semantic summarization on the historical medical record data and generate a historical medical record summary. The recognition unit 730 is used to recognize the screen image and extract the multimodal monitoring information of the target patient; The summary unit 740 is used to obtain a target condition summary of the target patient based on the historical medical record summary and the multimodal monitoring information. The sending unit 750 is used to send the target medical condition summary to the consulting physician terminal.

[0154] The device provided in this embodiment of the invention, in response to a consultation request, acquires the historical medical record data of the target patient and the real-time condition data at the consultation site, wherein the real-time condition data includes the screen image of the bedside monitoring device display interface; performs semantic summarization on the historical medical record data to generate a historical medical record summary; identifies the screen image to extract the multimodal monitoring information of the target patient; obtains the target condition summary of the target patient based on the historical medical record summary and the multimodal monitoring information; and sends the target condition summary to the consulting physician's terminal. This invention, through semantic induction of historical medical record data and visual recognition of images from bedside monitoring equipment screens at consultation sites, can quickly extract and associate patients' past medical history and multimodal monitoring information from multi-source heterogeneous medical data. This allows fragmented medical information to be instantly fused into a target disease summary and pushed to the consulting physician in a timely manner, thereby improving the automation and timeliness of medical information integration. It effectively solves the technical problems of existing hospital information systems, which require physicians to manually search through a large number of medical records in multiple systems, resulting in time consumption and easy omissions, as well as the technical problems of traditional medical record updates being lagging behind and unable to reflect the latest critical changes in the patient's condition. This ensures that the consulting physician can accurately grasp the patient's core condition before arriving at the scene, thereby significantly shortening the response and decision-making time of consultations.

[0155] Based on any of the above embodiments, the real-time medical data further includes voice description information describing the current vital signs of the target patient; The summary determination unit 740 specifically includes: A semantic understanding unit is used to perform semantic understanding on the speech description information to obtain real-time medical condition text; A summary subunit is defined to obtain a target patient's condition summary based on the historical medical record summary, the multimodal monitoring information, and the real-time condition text.

[0156] Based on any of the above embodiments, the method for determining the summary subunit is specifically used for: Extract the first text features of the historical medical record summary and the second text features of the real-time medical condition text; Calculate the first attention weight of the first text feature relative to the second text feature, and the second attention weight of the first text feature relative to the multimodal monitoring information, respectively. Based on the first attention weight and the second attention weight, the target attention weight is determined; Based on the target attention weight, the first text feature is adjusted to obtain the third text feature; Based on the third text feature, the multimodal monitoring information, and the second text feature, a target patient's disease summary is obtained.

[0157] Based on any of the above embodiments, the semantic understanding unit is specifically used for: Convert the voice description information into a text stream; Extract key clinical event information and vital sign information from the text stream; Based on the key clinical event information and the vital sign information, a real-time medical condition text is generated.

[0158] Based on any of the above embodiments, the identification unit 730 specifically includes: A region segmentation unit is used to segment the screen image into regions to obtain a numerical display sub-image and a waveform display sub-image. The extraction unit is used to extract numerical parameters from the numerical display sub-image and waveform morphology parameters from the waveform display sub-image; the waveform morphology parameters include the displacement amplitude, interval duration, and wave group morphology characteristics of the electrocardiogram waveform; The generation unit is used to generate a diagnostic conclusion, including an abnormal state indication, based on the waveform morphology parameters. A feature unit is defined for determining the multimodal monitoring information of the target patient based on the numerical parameters and the diagnostic conclusion.

[0159] Based on any of the above embodiments, the generation unit is specifically used for: Based on the waveform morphology parameters, the time-series waveform features are determined, and the time-series waveform features are input into the sequence classification model to obtain the pathological probability distribution output by the sequence classification model. Based on the pathological category corresponding to the maximum probability value in the pathological probability distribution, the abnormal state prompt is determined, and the diagnostic conclusion is generated based on the abnormal state prompt. The sequence classification model is trained based on the sample waveform morphological parameters and the label diagnostic conclusions of the sample waveform morphological parameters.

[0160] Based on any of the above embodiments, the semantic induction unit 720 is specifically used for: Extract key medical entities from the historical medical record data; Based on the temporal attributes of the key medical entities, the key medical entities are temporally associated to construct a disease evolution relationship chain; Based on the disease progression relationship chain, the historical medical record data is semantically summarized to generate the historical medical record summary.

[0161] Based on any of the above embodiments, the sending unit 750 is specifically used for: Retrieve auxiliary diagnostic opinions and recommended treatment plans that match the target disease summary from a preset medical knowledge base; Based on the auxiliary diagnostic opinions and recommended treatment plans related to the target disease summary, an enhanced consultation form is generated; The enhanced consultation form and the target disease summary are sent to the consulting physician's terminal.

[0162] Based on any of the above embodiments, a help-seeking unit is further included, wherein the help-seeking unit is specifically used for: Receive a request for help from the consulting physician's terminal; the request for help includes the text of the difficult question in the enhanced consultation form; Based on the text of the difficult problem and the summary of the target condition, a tiered consultation request is generated; The tiered consultation request is sent to the superior physician terminal of the consulting physician terminal to obtain consultation assistance results.

[0163] Based on any of the above embodiments, the sending unit 750 specifically includes: A semantic relevance unit is used to determine the semantic relevance between each information point in the target disease summary and the real-time disease data. A marking unit is used to mark information points whose semantic relevance exceeds a preset threshold as high-priority content; A visual enhancement unit is used to perform visual enhancement processing on the high-priority content and send the target disease summary of the visual enhancement processing to the consulting physician terminal.

[0164] Based on any of the above embodiments, the visual enhancement unit is specifically used for: The visually enhanced summary of the target condition, the historical medical record data, and the multimodal monitoring information are sent to the consulting physician's terminal.

[0165] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a consultation assistance method, which includes: in response to a consultation request, acquiring the target patient's historical medical record data and real-time condition data at the consultation site, the real-time condition data including screen images from the bedside monitoring device display interface; performing semantic summarization on the historical medical record data to generate a historical medical record summary; recognizing the screen images to extract the target patient's multimodal monitoring information; obtaining a target condition summary for the target patient based on the historical medical record summary and the multimodal monitoring information; and sending the target condition summary to the consulting physician's terminal.

[0166] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0167] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the consultation assistance method provided by the above methods. The method includes: in response to a consultation request, acquiring historical medical record data of the target patient and real-time condition data at the consultation site, wherein the real-time condition data includes screen images of the bedside monitoring device display interface; performing semantic summarization on the historical medical record data to generate a historical medical record summary; recognizing the screen images to extract multimodal monitoring information of the target patient; obtaining a target condition summary of the target patient based on the historical medical record summary and the multimodal monitoring information; and sending the target condition summary to the consulting physician terminal.

[0168] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the consultation assistance method provided by the above methods. The method includes: in response to a consultation request, acquiring historical medical record data of a target patient and real-time condition data at the consultation site, the real-time condition data including screen images of a bedside monitoring device display interface; semantically summarizing the historical medical record data to generate a historical medical record summary; recognizing the screen images to extract multimodal monitoring information of the target patient; obtaining a target condition summary of the target patient based on the historical medical record summary and the multimodal monitoring information; and sending the target condition summary to a consulting physician terminal.

[0169] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A consultation assistance method, characterized in that, include: In response to a consultation request, the system acquires the target patient's historical medical records and real-time condition data from the consultation site, including screen images from the bedside monitoring device's display interface. The historical medical record data is semantically summarized to generate a historical medical record summary; The screen image is identified to extract the multimodal monitoring information of the target patient; Based on the historical medical record summary and the multimodal monitoring information, the target patient's target condition summary is obtained; The target medical condition summary is sent to the consulting physician's terminal.

2. The consultation assistance method according to claim 1, characterized in that, The real-time medical data also includes voice description information describing the current vital signs of the target patient; The process of obtaining the target patient's condition summary based on the historical medical record summary and the multimodal monitoring information includes: Semantic understanding is performed on the voice description information to obtain real-time medical condition text; Based on the historical medical record summary, the multimodal monitoring information, and the real-time medical condition text, a target medical condition summary for the target patient is obtained.

3. The consultation assistance method according to claim 2, characterized in that, The process of obtaining the target patient's target medical condition summary based on the historical medical record summary, the multimodal monitoring information, and the real-time medical condition text includes: Extract the first text features of the historical medical record summary and the second text features of the real-time medical condition text; Calculate the first attention weight of the first text feature relative to the second text feature, and the second attention weight of the first text feature relative to the multimodal monitoring information, respectively. Based on the first attention weight and the second attention weight, the target attention weight is determined; Based on the target attention weight, the first text feature is adjusted to obtain the third text feature; Based on the third text feature, the multimodal monitoring information, and the second text feature, a target patient's disease summary is obtained.

4. The consultation assistance method according to claim 2, characterized in that, The step of performing semantic understanding on the speech description information to obtain real-time medical condition text includes: Convert the voice description information into a text stream; Extract key clinical event information and vital sign information from the text stream; Based on the key clinical event information and the vital sign information, a real-time medical condition text is generated.

5. The consultation assistance method according to any one of claims 1 to 4, characterized in that, The step of recognizing the screen image and extracting the multimodal monitoring information of the target patient includes: The screen image is segmented into regions to obtain a numerical display sub-image and a waveform display sub-image; Numerical parameters are extracted from the numerical display sub-image, and waveform morphology parameters are extracted from the waveform display sub-image; the waveform morphology parameters include the displacement amplitude, interval duration, and wave group morphology characteristics of the electrocardiogram waveform; Based on the waveform morphology parameters, a diagnostic conclusion including anomaly status indication is generated; Based on the numerical parameters and the diagnostic conclusions, the multimodal monitoring information of the target patient is determined.

6. The consultation assistance method according to claim 5, characterized in that, The step of generating a diagnostic conclusion based on the waveform morphology parameters, including an abnormal state indication, includes: Based on the waveform morphology parameters, the time-series waveform features are determined, and the time-series waveform features are input into the sequence classification model to obtain the pathological probability distribution output by the sequence classification model. Based on the pathological category corresponding to the maximum probability value in the pathological probability distribution, the abnormal state prompt is determined, and the diagnostic conclusion is generated based on the abnormal state prompt. The sequence classification model is trained based on the sample waveform morphological parameters and the label diagnostic conclusions of the sample waveform morphological parameters.

7. The consultation assistance method according to any one of claims 1 to 4, characterized in that, The step of semantically summarizing the historical medical record data to generate a historical medical record summary includes: Extract key medical entities from the historical medical record data; Based on the temporal attributes of the key medical entities, the key medical entities are temporally associated to construct a disease evolution relationship chain; Based on the disease progression relationship chain, the historical medical record data is semantically summarized to generate the historical medical record summary.

8. The consultation assistance method according to any one of claims 1 to 4, characterized in that, Sending the target disease summary to the consulting physician's terminal includes: Retrieve auxiliary diagnostic opinions and recommended treatment plans that match the target disease summary from a preset medical knowledge base; Based on the auxiliary diagnostic opinions and recommended treatment plans related to the target disease summary, an enhanced consultation form is generated; The enhanced consultation form and the target disease summary are sent to the consulting physician's terminal.

9. The consultation assistance method according to claim 8, characterized in that, After sending the enhanced consultation form and the target disease summary to the consulting physician's terminal, the process also includes: Receive a request for help from the consulting physician's terminal; the request for help includes the text of the difficult question in the enhanced consultation form; Based on the text of the difficult problem and the summary of the target condition, a tiered consultation request is generated; The tiered consultation request is sent to the superior physician terminal of the consulting physician terminal to obtain consultation assistance results.

10. The consultation assistance method according to any one of claims 1 to 4, characterized in that, Sending the target disease summary to the consulting physician's terminal includes: Determine the semantic correlation between each information point in the target disease summary and the real-time disease data; Information points whose semantic relevance exceeds a preset threshold are marked as high-priority content; The high-priority content is visually enhanced, and the target disease summary of the visually enhanced content is sent to the consulting physician's terminal.

11. The consultation assistance method according to claim 10, characterized in that, Sending the visually enhanced summary of the target condition to the consulting physician's terminal includes: The visually enhanced summary of the target condition, the historical medical record data, and the multimodal monitoring information are sent to the consulting physician's terminal.

12. A consultation auxiliary device, characterized in that, include: The acquisition unit is used to acquire the target patient's historical medical record data and real-time condition data at the consultation site in response to a consultation request. The real-time condition data includes the screen image of the bedside monitoring device display interface. A semantic summarization unit is used to perform semantic summarization on the historical medical record data and generate a historical medical record summary. The recognition unit is used to recognize the screen image and extract the multimodal monitoring information of the target patient; A summary determination unit is used to obtain a target condition summary of the target patient based on the historical medical record summary and the multimodal monitoring information; The sending unit is used to send the target medical condition summary to the consulting physician's terminal.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the consultation assistance method as described in any one of claims 1 to 11.

14. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the consultation assistance method as described in any one of claims 1 to 11.

Images