Hospital hospital guide method, system and equipment based on multi-modal data and storage medium
The hospital triage method, which integrates multimodal data fusion and dynamic interactive decision-making, solves the problems of low efficiency and insufficient accuracy in traditional triage services, and achieves efficient and accurate triage results output.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-30
- Publication Date
- 2026-03-27
AI Technical Summary
Traditional hospital triage services rely on manual labor, which is inefficient and has a high rate of misdiagnosis. Existing intelligent triage tools cannot effectively integrate multimodal information and lack dynamic adjustment judgment strategies, resulting in insufficient accuracy and personalization of triage results.
The hospital triage method based on multimodal data acquires text descriptions, medical images, and physiological signal data, extracts feature vectors, and fuses them into a patient state representation vector. It then uses a probabilistic inference model to calculate the confidence distribution, iteratively executes information gain actions to obtain new data, and dynamically optimizes the diagnostic path.
It significantly improves the accuracy and efficiency of intelligent triage, enabling comprehensive evaluation by utilizing multi-source information, reducing misdiagnosis rates, and achieving accurate and efficient triage.
Smart Images
Figure CN121747862A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data processing technology, specifically relating to a hospital triage method, system, device, and storage medium based on multimodal data. Background Technology
[0002] Traditional hospital triage services rely heavily on manual intervention. Patients, unfamiliar with departmental divisions or unable to accurately describe their symptoms, often face problems such as low triage efficiency, long waiting times, and misdiagnosis. Existing online intelligent triage tools are mostly based on simple keyword matching or static decision trees, with limited interaction methods and difficulty handling complex, multi-source medical information. These systems typically fail to effectively integrate multimodal information such as patient text descriptions, medical images, and real-time physiological data, and lack an intelligent mechanism that can simulate a doctor's proactive questioning and dynamic adjustment of judgment strategies during diagnostic reasoning. Their rigid decision-making process lacks quantitative assessment of uncertainty, resulting in limited accuracy and personalization of triage results, failing to meet the urgent needs for precise and efficient triage in real-world clinical scenarios. Summary of the Invention
[0003] In view of the above-mentioned shortcomings of the prior art, the present invention provides a hospital triage method, system, device and storage medium based on multimodal data to solve the above-mentioned technical problems.
[0004] In a first aspect, the present invention provides a hospital triage method based on multimodal data, comprising: Acquire multimodal input data provided by the patient, wherein the multimodal input data includes at least one of text description, medical image and physiological signal data; Multiple feature vectors corresponding to different modalities are extracted from the multimodal input data, and the multiple feature vectors are fused into a patient state representation vector; Based on the patient state representation vector, the confidence distribution of at least one target department is calculated using a probabilistic inference model; When the confidence distribution does not meet the preset termination condition, the following steps are iteratively executed: calculate the information gain of each potential action for a predefined set of actions; select and execute an action accordingly to obtain new patient data; update the patient state representation vector based on the new data and calculate the updated confidence distribution again through the probabilistic inference model; Output the triage results corresponding to the confidence distribution when the termination condition is met.
[0005] In an optional implementation, the method further includes: If the confidence distribution of the target department meets the preset termination condition, the target department will be directly output as the triage result and the process will be terminated.
[0006] In an optional implementation, multiple feature vectors corresponding to different modalities are extracted from the multimodal input data, and the multiple feature vectors are fused into a patient state representation vector, including: Medical named entity recognition and semantic feature extraction are performed on the text description to obtain the text feature vector; Visual feature extraction and lesion classification are performed on medical images to obtain visual feature vectors; Temporal feature extraction is performed on physiological signal data to obtain physiological feature vectors; The text feature vector, visual feature vector, and physiological feature vector are mapped to a shared semantic space and then weighted and fused to generate the patient state representation vector.
[0007] In an optional implementation, the text feature vector, visual feature vector, and physiological feature vector are mapped to a shared semantic space and then weighted and fused to generate the patient state representation vector, including: The text feature vectors, visual feature vectors, and physiological feature vectors of different dimensions are respectively input into the fully connected layer and mapped to a shared semantic space of a unified dimension. The patient state representation vector is generated by weighted summation of the mapped feature vectors using an attention mechanism.
[0008] In an optional implementation, based on the patient state representation vector, the confidence distribution of at least one target department is calculated using a probabilistic inference model, including: Transform the patient state representation vector into the state of one or more evidence nodes in the probabilistic inference model; Perform probabilistic inference to calculate the posterior probability of the target department or disease node, and form the confidence distribution; The probabilistic reasoning model is a pre-constructed Bayesian network; the Bayesian network contains nodes representing the target department or disease, and nodes representing patient evidence; the prior probabilities and conditional probabilities in the Bayesian network are pre-set based on medical knowledge.
[0009] In one optional implementation, the information gain of each potential action is calculated for a predefined set of actions; based on this, an action is selected and executed to obtain new patient data, including: Based on the current confidence distribution, calculate the information gain of each potential action in the predefined action set, wherein the predefined action set includes multiple predefined questions and multiple types of data to be requested; The calculation of information gain includes: calculating the entropy of the current confidence distribution, denoted as the current entropy; for each potential action, simulating all possible output results and the probability of each result occurring; based on the probability, calculating the expected entropy after executing the action; and using the difference between the expected entropy and the current entropy as the information gain of the potential action. The action with the highest information gain is selected from the set of actions and executed; if the selected action is to ask a question, the corresponding question is retrieved from the preset question template library and output to the patient; if the selected action is to request data, a request instruction is generated to prompt the patient to provide new data of the specified type.
[0010] In one optional implementation, the termination condition includes at least one of the following: The confidence threshold condition includes that, in the confidence distribution, the confidence level of at least one target department exceeds a preset first threshold. Information gain depletion condition, which includes the information gain of all potential actions being lower than a preset second threshold; Interaction limit conditions include the number of iterations of the process reaching a preset maximum.
[0011] Secondly, the present invention provides a hospital triage system based on multimodal data, comprising: The data acquisition module is used to acquire multimodal input data provided by the patient, wherein the multimodal input data includes at least one of text description, medical image and physiological signal data; The feature extraction module is used to extract multiple feature vectors corresponding to different modalities from the multimodal input data, and to fuse the multiple feature vectors into a patient state representation vector; The probabilistic reasoning module is used to calculate the confidence distribution of at least one target department based on the patient state representation vector using a probabilistic reasoning model. The iterative calculation module is used to iteratively execute the following steps when the confidence distribution does not meet the preset termination condition: calculate the information gain of each potential action for a predefined set of actions; select and execute an action accordingly to obtain new patient data; update the patient state representation vector based on the new data and calculate the updated confidence distribution again through the probabilistic inference model; The result output module is used to output the triage result corresponding to the confidence distribution when the termination condition is met.
[0012] Thirdly, a device is provided, comprising: Memory, used to store hospital triage programs based on multimodal data; A processor is configured to implement the steps of the hospital triage method based on multimodal data as provided in the first aspect when executing the hospital triage program based on multimodal data.
[0013] Fourthly, a computer-readable storage medium is provided, on which a hospital triage program based on multimodal data is stored, wherein when the hospital triage program based on multimodal data is executed by a processor, the hospital triage method based on multimodal data provided in the first aspect is implemented.
[0014] The beneficial effects of this invention are that the hospital triage method, system, device, and storage medium based on multimodal data provided by this invention significantly improve the accuracy and efficiency of intelligent triage through multimodal data fusion and dynamic interactive decision-making. It can comprehensively utilize multi-source information such as text, images, and physiological signals to achieve a comprehensive assessment of the patient's condition; through an iterative question-and-answer mechanism based on probabilistic reasoning and information gain, the system can proactively acquire key information, dynamically optimize the diagnostic path, and significantly reduce the misdiagnosis rate. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a schematic flowchart of a method according to an embodiment of the present invention.
[0017] Figure 2 This is a schematic block diagram of a system according to an embodiment of the present invention.
[0018] Figure 3 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation
[0019] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.
[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
[0021] The hospital triage method based on multimodal data provided in this embodiment of the invention is executed by a computer device, and correspondingly, the hospital triage system based on multimodal data runs on the computer device.
[0022] Figure 1 This is a schematic flowchart illustrating a method according to an embodiment of the present invention. Wherein, Figure 1 The implementing entity can be a hospital triage system based on multimodal data. Depending on different needs, the order of the steps in this flowchart can be changed, and some can be omitted.
[0023] like Figure 1 As shown, the method includes: S1. Obtain multimodal input data provided by the patient, wherein the multimodal input data includes at least one of text description, medical image and physiological signal data; S2. Extract multiple feature vectors corresponding to different modalities from the multimodal input data, and fuse the multiple feature vectors into a patient state representation vector; S3. Based on the patient state representation vector, calculate the confidence distribution of at least one target department using a probabilistic inference model; S4. When the confidence distribution does not meet the preset termination condition, iteratively execute the following steps: calculate the information gain of each potential action for a predefined set of actions; select and execute an action accordingly to obtain new patient data; update the patient state representation vector based on the new data and calculate the updated confidence distribution again through the probabilistic inference model; S5. Output the triage results corresponding to the confidence distribution when the termination condition is met.
[0024] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.
[0025] S101. Obtaining the text description: Input interface: Provide explicit text input boxes in the mobile app, WeChat mini program or web interface used by patients, allowing patients to describe their symptoms directly in text form (e.g., "I have had a fever, sore throat and cough since the day before yesterday").
[0026] Technical implementation: The front end uses standard text box components (such as HTML's...) <textarea> (Or the native input box on the mobile device) receives user input and asynchronously transmits the text data to the backend server via HTTP / HTTPS protocol.< / textarea>
[0027] S102. Acquisition of medical images: Input Interface: Within the same user interface, "Take Photo" and "Upload from Album" buttons are provided to guide patients in taking or uploading photos of the affected area. Typical application scenarios include taking clear photos of skin rashes, swollen areas, throat, tongue coating, or traumatic sites.
[0028] Technical Implementation: The front-end calls the mobile device's camera API (such as the browser's MediaDevices.getUserMedia() interface or the WeChat mini-program's wx.chooseImage() interface) to obtain the image. The uploaded image file (usually in JPG or PNG format) is uploaded to the server via a multimedia form.
[0029] S103. Acquisition of physiological signal data: Manual input: Standardized forms are provided for patients to manually fill in key physiological indicators, such as current body temperature and blood pressure.
[0030] Automatic synchronization: Interacting with wearable devices: After obtaining explicit authorization from the user, the SDK and API provided by health apps (such as Apple Health Kit, Google Fit, and Mi Fit) are called to read historical physiological time-series data synchronized by the device or recorded by the user, including heart rate, heart rate variability, blood oxygen saturation, sleep data, etc.
[0031] Integration with hospital systems: Through standardized interfaces (such as HL7 FHIR) provided by hospital information systems (HIS) or electronic medical record (EMR) systems, patients' past medical history, diagnostic records, laboratory test results and other structured data can be queried and retrieved under the premise of security and compliance.
[0032] S104. Data Preprocessing and Integration: After receiving the multimodal data, the server backend will create a unique session ID for this triage session and bind all data to this session ID, storing it in a temporary database or cache to prepare for subsequent feature extraction and fusion modules.
[0033] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.
[0034] S201. Text Feature Vector Extraction Medical Named Entity Recognition (NER): Technology selection: We adopted the BERT-Base pre-trained model, which was finely tuned on a medical corpus (such as MIMIC-III). We defined a set of medical entity labels, including: SYMPTOM, BODY_PART, QUALITY, DURATION, etc.
[0035] Implementation process: Input the patient's text description (such as "I have intermittent pain in my temples") into the NER model, and output a list of structured entities, such as: [(temples, BODY_PART), (intermittent, QUALITY), (pain, SYMPTOM)].
[0036] Text semantic feature extraction: Technology selection: The Sentence-BERT model is used, specifically the all-MiniLM-L6-v2 version, which generates a 384-dimensional semantic vector for each sentence.
[0037] Implementation process: The model inputs the patient's entire description and directly outputs a high-dimensional, floating-point text feature vector. This vector can capture subtle semantic differences such as "mild headache" and "severe headache".
[0038] S202. Visual Feature Vector Extraction Technology selection: The EfficientNet-B0 model, pre-trained on ImageNet and skin disease datasets (such as ISIC), is used as the feature extractor.
[0039] Implementation process: Image preprocessing: The uploaded medical images are automatically corrected, centrally cropped, and uniformly scaled to 224x224 pixels, and finally the pixel values are normalized.
[0040] Feature extraction and classification: The preprocessed image is input into the EfficientNet-B0 model. The last classification layer is removed, and the output of the global average pooling layer is taken as a 1024-dimensional visual feature vector. At the same time, a small fully connected classifier is added to the top of the model to classify several common skin lesions (such as erythema, papules, and vesicles), and output the lesion type and the corresponding confidence score.
[0041] S203. Physiological Feature Vector Extraction Implementation process: Data cleaning: Preprocessing time-series data (such as body temperature and heart rate) obtained from wearable devices or electronic medical records, including handling missing values and removing noise such as motion artifacts using sliding window filtering.
[0042] Temporal feature engineering: Extracting a set of statistical features for each signal within a fixed time window (e.g., the past 6 hours). For example: Body temperature: current value, average value, maximum value, and rise / fall slope. Heart rate: average heart rate, maximum and minimum values, SDNN (standard deviation), and other time-domain features. Blood oxygen saturation: current value, minimum value, and the percentage of time the saturation level is below 95%.
[0043] All statistical features are concatenated into a unified physiological feature vector.
[0044] S204. Feature fusion into a patient state representation vector Feature alignment (mapping to a shared semantic space): Technology selection: Use three independent fully connected neural network layers.
[0045] Implementation process: Text feature vectors (384-dimensional), visual feature vectors (1024-dimensional), and physiological feature vectors (e.g., 20-dimensional) of different dimensions are input into the corresponding fully connected layers. These three fully connected layers are all designed to output 256-dimensional vectors, thereby mapping the features of different modalities to a unified shared semantic space.
[0046] Weighted fusion (attention mechanism): Technology selection: A simple additive attention network is adopted.
[0047] Implementation Process: The three mapped 256-dimensional vectors are concatenated into a single 768-dimensional vector. This concatenated vector is then passed through an attention network, which outputs a 3-dimensional attention weight vector (corresponding to text, visual, and physiological modalities, respectively), with the weights summing to 1. Using the calculated weights, the three mapped 256-dimensional vectors are weighted and summed to generate a final 256-dimensional, comprehensive patient state representation vector. This vector encapsulates all available patient information at the current moment, used for subsequent probabilistic reasoning and decision-making.
[0048] In one embodiment of the present invention, based on steps S3, S4 and S5, a possible embodiment will be given below, and its specific implementation will be described in a non-limiting manner.
[0049] S3. Implementation of probabilistic inference based on Bayesian networks Construction of Bayesian Networks: Node definition: A Bayesian network is built using the open-source library pgmpy. Network nodes include: Hypothesis nodes: Represent the target disease or department to be inferred, such as D_Migraine (migraine), D_TensionHeadache (tension headache), and D_Glaucoma (glaucoma). Each disease node is associated with a recommended department.
[0050] Evidence nodes: represent symptoms or signs that can be observed from patient data, such as E_Photophobia, E_UnilateralPain, E_Rash, and E_Fever.
[0051] Probability settings: The prior probability P(Disease) in the network is set based on disease epidemiological data. The conditional probability P(Evidence | Disease) is extracted from medical textbooks, clinical guidelines, and expert knowledge and encoded into a conditional probability table. For example: P(Photophobia=True | D_Migraine) = 0.8.
[0052] Calculation of confidence distribution: Evidence Input: The "patient state representation vector" is transformed into the state of the corresponding evidence node in a Bayesian network through a simple classifier or rule engine. For example, when visual features in the vector indicate the presence of a rash and textual features mention "photophobia," the states of nodes E_Rash and E_Photophobia are set to Observed=True.
[0053] Probabilistic inference: Input the pre-set evidence into the variable elimination method or belief propagation algorithm provided by pgmpy for Bayesian inference. This algorithm calculates the posterior probability of all disease hypothesis nodes, forming a confidence distribution, for example: {"Neurology - Migraine": 0.75, "Neurology - Tension Headache": 0.20, "Ophthalmology - Glaucoma": 0.05}.
[0054] S4. Iterative decision-making based on information gain Information gain calculation: 1. Calculate the current uncertainty (current entropy) Input: The current confidence distribution P(D) from the probabilistic reasoning module, for example: {"Neurology-Migraine":0.6, "Neurology-Tension Headache":0.3, "Ophthalmology-Glaucoma":0.1}.
[0055] Data processing method: The system uses the information entropy formula. To calculate the uncertainty in the current state. Represents the hypothesis of the i-th disease or department. It is its probability.
[0056] Procedure: Substitute each probability value from the confidence distribution into the above formula and sum them. For example, for the above distribution, the calculation process is as follows: Calculation results It is a specific scalar value; the higher the entropy value, the more uncertain the system's judgment of the condition.
[0057] 2. The result of simulating potential actions Input: A predefined set of actions (e.g., asking "Are you afraid of light?" or requesting "Upload a photo of your tongue coating").
[0058] Data processing methods and processes: For each potential action in the set, the system initiates a simulated reasoning process: a. Enumerate possible results: For question-type actions, the result is usually "yes" or "no"; for data request-type actions, the result is the possible classification result (such as "image classified as erythema" or "image classified as normal").
[0059] b. Calculate the probability P(Answer): This step utilizes knowledge from a pre-constructed Bayesian network. The system queries the evidence node (e.g., E) in the network. Photophobia The marginal probability of each outcome is calculated using a conditional probability table (CPT) between the disease node and its parent node (disease node).
[0060] Calculation formula:
[0061] Example: For the question "Are you afraid of light?", calculate the probability of the answer being "yes": P(yes) = P(migraine) * P(photophobia|migraine) + P(tension headache) * P(photophobia|tension headache) + P(glaucoma) * P(photophobia|glaucoma) P(yes) = P(migraine) * P(photophobia|migraine) + P(tension headache) * P(photophobia|tension headache) + P(glaucoma) * P(photophobia|glaucoma). Where P(photophobia|disease) is directly read from the conditional probability table of the Bayesian network.
[0062] c. The new entropy H after the calculation result occurs answer : Updated Evidence: In the simulation environment, the system inputs "assuming this outcome occurs" as new evidence into the Bayesian network. For example, assuming the answer is "yes," then node E... Photophobia The state is set to Observed=True.
[0063] Perform probability updates: In the simulated Bayesian network, use the same probabilistic inference algorithm (such as variable elimination) to compute the posterior probability distribution P(D|Answer=A) of the disease node under new evidence. j ).
[0064] Calculate the new entropy: based on this updated probability distribution P(D|A) jUsing the entropy formula from step 1, calculate the new entropy value H for this result. answer .
[0065] 3. Calculate the expected information gain Data processing method: The system calculates the expected entropy of each action, which is the weighted average of the entropy values after all possible results occur.
[0066] Calculation formula: Where m is the number of possible outcomes of the action.
[0067] Final information gain: The information gain IG(A) of action A is obtained by the following formula:
[0068] This difference quantifies the expected reduction in system uncertainty after performing action A. A larger IG(A) value indicates that the action is more valuable for a definitive diagnosis.
[0069] Action selection and execution: Selection: The system selects the action with the highest information gain.
[0070] If the action is to ask a question, the system retrieves the corresponding question from a pre-set natural language question template library that is bound to the evidence node (such as the question bound to the E_Photophobia node: "When you have a headache, do you feel afraid of light or sound?"), and presents it to the patient through the front-end interface.
[0071] If the action is a data request, the system generates a standardized instruction (such as {"action": "request_image", "type": "skin_lesion", "hint": "Please take a clear photo in a well-lit place"}), which is then converted into a natural language prompt by the front end to guide the patient to provide new data.
[0072] S5. Termination Judgment and Result Output Termination condition monitoring: After each iteration, the system checks the following conditions, and terminates the process if any one of them is met: Confidence threshold: The confidence level of any target department exceeds the threshold (e.g., max(P(D_i))>0.90).
[0073] Information gain depletion: The information gain of all remaining potential actions is below a very small threshold (e.g., IG_max < 0.01), meaning that further queries will yield very little benefit.
[0074] Interaction limit: To prevent infinite loops, a safety limit is set (e.g., a maximum of 10 iterations).
[0075] Output Results: When the process terminates, the system outputs the department with the highest confidence level as the referral result. Simultaneously, an interpretable report is generated, which, through backtracking a Bayesian network, lists the key evidence that contributed most to the final conclusion (e.g., "Judgment criteria: unilateral headache, photophobia"), thereby enhancing the system's transparency and credibility.
[0076] Example process: The patient's initial description was "headache," and the initial confidence distribution entropy value was very high.
[0077] The system calculations showed that the question "Is the pain unilateral?" had the highest information gain, so it asked that question. The patient answered "yes".
[0078] The confidence distribution is updated (probability of migraine increases), and the entropy value decreases. The system then asks the question "Are you afraid of light?" in the next round.
[0079] The patient answered "afraid of light," raising the confidence level for migraine to 0.92, which exceeds the threshold of 0.9, and the process was terminated.
[0080] The system recommends "Neurology" and generates a report stating that "unilateral pain and photophobia are key diagnostic factors."
[0081] In some embodiments, the hospital triage system based on multimodal data may include multiple functional modules composed of computer program segments. The computer programs for each program segment in the hospital triage system based on multimodal data may be stored in the memory of a computer device and executed by at least one processor to perform (see details). Figure 1 (Description) The function of hospital triage based on multimodal data.
[0082] In this embodiment, the hospital triage system based on multimodal data can be divided into multiple functional modules according to the functions it performs, such as... Figure 2 As shown. The module referred to in this invention is a series of computer program segments that can be executed by at least one processor and perform a fixed function, and is stored in memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.
[0083] The data acquisition module is used to acquire multimodal input data provided by the patient, wherein the multimodal input data includes at least one of text description, medical image and physiological signal data; The feature extraction module is used to extract multiple feature vectors corresponding to different modalities from the multimodal input data, and to fuse the multiple feature vectors into a patient state representation vector; The probabilistic reasoning module is used to calculate the confidence distribution of at least one target department based on the patient state representation vector using a probabilistic reasoning model. The iterative calculation module is used to iteratively execute the following steps when the confidence distribution does not meet the preset termination condition: calculate the information gain of each potential action for a predefined set of actions; select and execute an action accordingly to obtain new patient data; update the patient state representation vector based on the new data and calculate the updated confidence distribution again through the probabilistic inference model; The result output module is used to output the triage result corresponding to the confidence distribution when the termination condition is met.
[0084] Figure 3 The hospital triage method based on multimodal data provided in the embodiments of this application can be applied to devices. Those skilled in the art will understand that the device structure involved in the embodiments of this invention does not constitute a limitation on the device. A device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In the embodiments of this invention, the device includes, but is not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of this application described and / or claimed herein.
[0085] The device 300 may include a processor 310, a memory 320, and a communication unit 330. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.
[0086] The memory 320 can be used to store execution instructions of the processor 310. The memory 320 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 320 are executed by the processor 310, the device 300 is able to perform some or all of the steps in the above method embodiments.
[0087] The processor 310 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 320, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 310 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.
[0088] The communication unit 330 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.
[0089] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.
[0090] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.
[0091] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.
[0092] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.
[0093] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0094] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.
[0095] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.
Claims
1. A hospital triage method based on multimodal data, characterized in that, include: Acquire multimodal input data provided by the patient, wherein the multimodal input data includes at least one of text description, medical image and physiological signal data; Multiple feature vectors corresponding to different modalities are extracted from the multimodal input data, and the multiple feature vectors are fused into a patient state representation vector; Based on the patient state representation vector, the confidence distribution of at least one target department is calculated using a probabilistic inference model; When the confidence distribution does not meet the preset termination condition, the following steps are iteratively executed: calculate the information gain of each potential action for a predefined set of actions; select and execute an action accordingly to obtain new patient data; update the patient state representation vector based on the new data and calculate the updated confidence distribution again through the probabilistic inference model; Output the triage results corresponding to the confidence distribution when the termination condition is met.
2. The method according to claim 1, characterized in that, The method further includes: If the confidence distribution of the target department meets the preset termination condition, the target department will be directly output as the triage result and the process will be terminated.
3. The method according to claim 1, characterized in that, Multiple feature vectors corresponding to different modalities are extracted from the multimodal input data, and the multiple feature vectors are fused into a patient state representation vector, including: Medical named entity recognition and semantic feature extraction are performed on the text description to obtain the text feature vector; Visual feature extraction and lesion classification are performed on medical images to obtain visual feature vectors; Temporal feature extraction is performed on physiological signal data to obtain physiological feature vectors; The text feature vector, visual feature vector, and physiological feature vector are mapped to a shared semantic space and then weighted and fused to generate the patient state representation vector.
4. The method according to claim 3, characterized in that, The text feature vector, visual feature vector, and physiological feature vector are mapped to a shared semantic space and then weighted and fused to generate the patient state representation vector, including: The text feature vectors, visual feature vectors, and physiological feature vectors of different dimensions are respectively input into the fully connected layer and mapped to a shared semantic space of a unified dimension. The patient state representation vector is generated by weighted summation of the mapped feature vectors using an attention mechanism.
5. The method according to claim 1, characterized in that, Based on the patient state representation vector, the confidence distribution of at least one target department is calculated using a probabilistic inference model, including: Transform the patient state representation vector into the state of one or more evidence nodes in the probabilistic inference model; Perform probabilistic inference to calculate the posterior probability of the target department or disease node, and form the confidence distribution; The probabilistic reasoning model is a pre-constructed Bayesian network; the Bayesian network contains nodes representing the target department or disease, and nodes representing patient evidence; the prior probabilities and conditional probabilities in the Bayesian network are pre-set based on medical knowledge.
6. The method according to claim 1, characterized in that, Calculate the information gain of each potential action for a predefined set of actions; select and execute an action accordingly to obtain new patient data, including: Based on the current confidence distribution, calculate the information gain of each potential action in the predefined action set, wherein the predefined action set includes multiple predefined questions and multiple types of data to be requested; The calculation of information gain includes: calculating the entropy of the current confidence distribution, denoted as the current entropy; for each potential action, simulating all possible output results and the probability of each result occurring; based on the probability, calculating the expected entropy after executing the action; and using the difference between the expected entropy and the current entropy as the information gain of the potential action. The action with the highest information gain is selected from the set of actions and executed; if the selected action is to ask a question, the corresponding question is retrieved from the preset question template library and output to the patient; if the selected action is to request data, a request instruction is generated to prompt the patient to provide new data of the specified type.
7. The method according to claim 1, characterized in that, The termination condition includes at least one of the following: The confidence threshold condition includes that, in the confidence distribution, the confidence level of at least one target department exceeds a preset first threshold. Information gain depletion condition, which includes the information gain of all potential actions being lower than a preset second threshold; Interaction limit conditions include the number of iterations of the process reaching a preset maximum.
8. A hospital triage system based on multimodal data, characterized in that, include: The data acquisition module is used to acquire multimodal input data provided by the patient, wherein the multimodal input data includes at least one of text description, medical image and physiological signal data; The feature extraction module is used to extract multiple feature vectors corresponding to different modalities from the multimodal input data, and to fuse the multiple feature vectors into a patient state representation vector; The probabilistic reasoning module is used to calculate the confidence distribution of at least one target department based on the patient state representation vector using a probabilistic reasoning model. The iterative calculation module is used to iteratively execute the following steps when the confidence distribution does not meet the preset termination condition: calculate the information gain of each potential action for a predefined set of actions; select and execute an action accordingly to obtain new patient data; update the patient state representation vector based on the new data and calculate the updated confidence distribution again through the probabilistic inference model; The result output module is used to output the triage result corresponding to the confidence distribution when the termination condition is met.
9. A hospital triage device based on multimodal data, characterized in that, include: Memory, used to store hospital triage programs based on multimodal data; A processor is configured to implement the steps of the hospital triage method based on multimodal data as described in any one of claims 1-7 when executing the hospital triage program based on multimodal data.
10. A computer-readable storage medium storing a computer program, characterized in that, The readable storage medium stores a hospital triage program based on multimodal data, which, when executed by a processor, implements the steps of the hospital triage method based on multimodal data as described in any one of claims 1-7.