Robot hospital guide method based on multi-modal data and intelligent hospital guide robot

Abstract

The invention provides a robot hospital guide method based on multi-modal data and an intelligent hospital guide robot, and relates to the technical field of robots. The intelligent hospital guide robot comprises a multi-mode sensing module, an intention understanding and decision center module and a behavior execution and guide module. The method is applied to the intelligent hospital guide robot, and comprises the following steps: collecting multi-modal data through a multi-modal sensing module; performing information fusion and intention understanding on the acquired multi-modal data through an intention understanding and decision-making center module, and generating a decision-making instruction; executing a decision instruction through a behavior execution and guidance module, so as to execute personification interaction by controlling joint movement of the intelligent hospital guide robot and / or execute visual guidance by rendering and superposing an augmented reality navigation arrow on a display screen of the intelligent hospital guide robot; the problems of unfriendly interaction, single perception, rigid planning and the like in the prior art are solved, and the triage accuracy, the user experience and the adaptability in complex medical scenes are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a robot-based triage method and an intelligent triage robot based on multimodal data. Background Technology

[0002] Currently, the triage technologies widely used in hospitals mainly fall into two categories: touchscreen-based self-service terminals and early-generation guidance robots. Self-service terminals rely on graphical interfaces and touch controls, which are unfriendly to the elderly and users unfamiliar with electronic devices. They also cannot accurately understand vague symptom descriptions and lack contextual awareness and proactive service capabilities. Early-generation robots mostly employ single-modal interaction (such as voice or touchscreen only), with limited perception dimensions, unable to integrate multi-dimensional information such as vision and voice. Their task planning relies on fixed scripts, making it impossible to provide multi-round clarification or dynamically respond to emergencies. The guidance method is rigid, resulting in limited overall triage accuracy and user experience. Summary of the Invention

[0003] The purpose of this invention is to provide a robot-based triage method and intelligent triage robot based on multimodal data. The method uses a multimodal perception module to simultaneously collect voice, visual, and environmental context data. The intention understanding and decision center performs multimodal information fusion and deep intention understanding. Finally, the behavior execution and guidance module realizes human-like interaction and AR visual guidance. This invention overcomes the problems of unfriendly interaction, single perception, and rigid planning in existing technologies, and significantly improves triage accuracy, user experience, and adaptability in complex medical scenarios.

[0004] In a first aspect, the present invention provides a robot-based triage method based on multimodal data, applied to an intelligent triage robot. The intelligent triage robot includes: a multimodal perception module, an intent understanding and decision-making center module, and a behavior execution and guidance module; the method includes: Multimodal data is acquired through a multimodal perception module; the multimodal data includes: the user's voice signal, visual images, and environmental context data; The intent understanding and decision center module performs information fusion and intent understanding on the collected multimodal data to generate decision instructions; The decision-making instructions are executed through the behavior execution and guidance module to perform anthropomorphic interactions by controlling the joint movements of the intelligent triage robot and / or to render and overlay augmented reality navigation arrows on the display screen of the intelligent triage robot to perform visual guidance.

[0005] In some preferred embodiments of the present invention, the multimodal perception module includes: a voice interaction submodule, a visual perception submodule, and a context perception submodule; the voice interaction submodule includes: an end-to-end speech recognition model based on a convolutional recurrent neural network and a lightweight acoustic feature analysis model; Speech recognition models are used to convert a user's speech signals into text; Acoustic feature analysis model is used to extract fundamental frequency, energy and speech rate features based on the user's speech signal to quantify the user's state; The visual perception submodule is used to extract key points of the user's body in visual images to analyze posture, and to use a pre-set convolutional neural network to classify facial images in visual images to recognize expressions. The context-aware submodule is used to process environmental context data to analyze the crowd density around the robot.

[0006] In some preferred embodiments of the present invention, the context-aware submodule includes: an environment-aware model with a preset clustering algorithm; the context-aware submodule is used to set the neighborhood radius and minimum number of points parameters to distinguish pedestrian point cloud clusters, and to calculate the crowd density value based on the number of clusters and the area of ​​the monitoring area based on the environment-aware model, and to classify the current environmental state into sparse, medium or crowded levels based on the crowd density value.

[0007] In some preferred embodiments of the present invention, the intent understanding and decision center module includes: a multimodal information fusion unit, an intent recognition and state evaluator, and a dynamic dialogue management and task planner. A multimodal information fusion unit is used to dynamically weight and fuse feature vectors from different modalities using a pre-defined attention-based feature fusion algorithm to generate a user state embedding vector. The intent recognition and state evaluator is used to perform semantic similarity matching and reasoning between the user's state embedding vector and a pre-built medical knowledge graph, and output a comprehensive state label. The comprehensive state label includes: confidence level, recommended department, urgency level, and whether assistance is needed. A dynamic dialogue management and task planner is used to dynamically generate or adjust task sequences based on comprehensive status labels and a hierarchical task network planner.

[0008] In some preferred embodiments of the present invention, a multimodal information fusion processor is used to calculate the attention score of the feature vector of each modality, normalize the attention score using a preset function to obtain the weight of each modality, and perform a weighted summation of the feature vectors of all modalities according to the weights to generate a user state embedding vector.

[0009] In some preferred embodiments of the present invention, a hierarchical task network planner is used to dynamically generate or adjust task sequences, including: When the reliability is lower than a preset threshold, multiple rounds of dialogue are inserted into the task sequence to clarify the sub-tasks; When an event with high urgency is received, the current task sequence is immediately stopped, and the task corresponding to the high urgency event is put on top for execution.

[0010] In some preferred embodiments of the present invention, the behavior execution and guidance module includes: an anthropomorphic interaction unit, an AR visual guidance unit, and a system collaboration interface; the anthropomorphic interaction unit is used to generate control commands based on a preset semantic-action mapping library to drive the neck and waist joints of the intelligent triage robot to perform nodding, turning to the side, or indicating gestures. The AR visual guidance unit is used to obtain the real-time pose of the intelligent triage robot based on the real-time localization and mapping technology of LiDAR and vision fusion, and to render dynamic 3D navigation arrows on the display screen according to the planned path. The system collaboration interface is used to exchange data with the hospital information system to send structured alarm messages.

[0011] In some preferred embodiments of the present invention, the anthropomorphic interaction unit is used to trigger a nodding action when the voice content contains affirmative semantics; to trigger the intelligent triage robot to make a slightly sideways posture when it is recognized that the intelligent triage robot is listening to the user's statement; and to simultaneously execute indicative gestures when providing directional guidance.

[0012] In some preferred embodiments of the present invention, the AR visual guidance unit is used to achieve positioning in a pre-constructed hospital map by fusing lidar point cloud data and visual data, and to calculate the optimal path based on the navigation target point.

[0013] Secondly, the present invention provides an intelligent triage robot for performing the robot triage method based on multimodal data provided in the first aspect above.

[0014] This invention brings the following beneficial effects: This invention provides a robot-based triage method and an intelligent triage robot based on multimodal data. The intelligent triage robot includes a multimodal perception module, an intent understanding and decision center module, and a behavior execution and guidance module. The method is applied to the intelligent triage robot and includes: collecting multimodal data through the multimodal perception module; wherein the multimodal data includes: user voice signals, visual images, and environmental context data; performing information fusion and intent understanding on the collected multimodal data through the intent understanding and decision center module to generate decision commands; and executing the decision commands through the behavior execution and guidance module to perform human-like interaction by controlling the joint movements of the intelligent triage robot and / or rendering and overlaying augmented reality navigation arrows on the intelligent triage robot's display screen to perform visual guidance. This method overcomes the problems of unfriendly interaction, limited perception, and rigid planning in existing technologies, significantly improving triage accuracy, user experience, and adaptability in complex medical scenarios. Attached Figure Description

[0015] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 This is a schematic diagram of the system structure of an intelligent triage robot provided in an embodiment of the present invention; Figure 2 A flowchart illustrating a robot-based triage method based on multimodal data provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.

[0017] Icons: 400 - Memory; 401 - Processor; 402 - Bus; 403 - Communication Interface. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0019] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0020] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0021] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. In addition, the terms "first," "second," "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] Furthermore, terms such as "horizontal," "vertical," and "sag" do not imply that components must be absolutely horizontal or suspended, but rather that they can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal relative to "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0023] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0024] With the rapid development of artificial intelligence and robotics, service robots have begun to be used in public places such as hospitals and banks to alleviate the repetitive and manpower-intensive front desk consultation and guidance work. In hospital settings, efficiently and accurately triaging patients to the correct departments (i.e., "triage") is a key aspect of improving medical service efficiency and patient satisfaction.

[0025] Currently, the existing technologies most similar to the technical solution of this invention can be mainly divided into the following two categories: 1. Touchscreen-based self-service terminal (self-service machine) These devices are currently the most widely used patient guidance tools in hospitals. Patients select options and enter keywords related to their symptoms on a touchscreen, and the system recommends the appropriate department to visit.

[0026] This technology has the following drawbacks: First, the interaction is unfriendly and there is a digital divide: the interaction method relies entirely on the graphical user interface (GUI) and touch operation. For the elderly, those unfamiliar with electronic devices, or illiterate patients, the learning cost is high, the operation is difficult, and they are prone to resistance, which makes them prefer to wait in long lines for human assistance.

[0027] Secondly, the system suffers from weak intent comprehension and poor error tolerance: it relies on patients accurately translating their symptoms into abstract text keywords. When patients cannot accurately describe their condition (e.g., "stomach ache" but cannot distinguish whether it's a stomach or intestinal problem), triage errors are highly likely. The system lacks the ability to ask follow-up questions and clarify issues, resulting in a rigid, one-off, and discontinuous interaction process.

[0028] Finally, there is a lack of situational awareness and initiative: self-service machines are completely passive devices, unable to perceive the patient's physical state (such as whether they have difficulty moving or whether they appear to be in pain) or psychological state (such as whether they are anxious). Therefore, they cannot identify emergencies and activate priority handling mechanisms, nor can they provide humane care and comfort.

[0029] 2. First-generation guided robot These robots typically have a tablet computer mounted on a mobile chassis, enabling basic voice dialogue and mobility. They can respond to simple voice commands (such as "Take me to the internal medicine department") and provide path guidance.

[0030] This technology has the following drawbacks: First, its perception is limited and its understanding is one-sided: its interaction relies heavily on a single voice module or touchscreen input, making it a "single-modal" interaction. It cannot integrate and analyze the patient's voice information with visual information such as body posture, facial expressions, and behavior. For example, when a patient calmly says "I'm fine," but visually appears unsteady, the robot will accept the voice content and ignore the potential risk.

[0031] Secondly, their task planning is rigid, making them unable to handle complex scenarios: the behavior of these robots is mostly driven by pre-set scripts, lacking deep intent understanding and dynamic decision-making capabilities. Their task planning is static, unable to conduct multi-turn, context-sensitive intelligent dialogue to clarify ambiguous intents. When faced with multiple patients asking questions simultaneously or encountering sudden emergencies, their programs are prone to becoming chaotic or rigidly executing predetermined tasks, lacking the ability to dynamically adjust interruptions and priorities.

[0032] Finally, the triage process is rudimentary and the guidance methods are rigid: the triage logic is often a simple replication of the self-service machine logic, failing to leverage the potential of humanoid robot-like interaction. Guidance methods are usually limited to voice prompts or on-screen maps, which are not intuitive or engaging enough, resulting in a poor experience for patients with poor sense of direction.

[0033] In summary, existing hospital triage technologies, whether self-service terminals or early-generation robots, suffer from a series of problems due to limitations in their interaction modes, singular perceptual capabilities, and rigid decision-making logic, including low triage accuracy, poor user experience, and poor adaptability in complex real-world medical scenarios. Therefore, there is an urgent need for a robotic system capable of multi-dimensional observation, deep intent understanding, and intelligent dynamic response, much like a human triage guide.

[0034] The core of this application lies in constructing an intelligent triage method with a closed-loop feedback mechanism encompassing perception, decision-making, and execution. It acquires rich information through multimodal perception, performs in-depth understanding and planning via a central decision-maker, and finally outputs the results through a human-like execution module, continuously adapting and adjusting based on feedback throughout the process.

[0035] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0036] Example 1 This invention provides a robot-based triage method based on multimodal data, applicable to intelligent triage robots. (See also...) Figure 1 The diagram shown is a system structure diagram of an intelligent triage robot provided by an embodiment of the present invention; the intelligent triage robot includes: a multimodal perception module, an intent understanding and decision-making center module, and a behavior execution and guidance module.

[0037] See Figure 2 The flowchart shown in this embodiment of the invention provides a robot-based triage method based on multimodal data. The method includes: Step S102: Collect multimodal data through the multimodal perception module; wherein, the multimodal data includes: the user's voice signal, visual image and environmental context data.

[0038] Specifically, the multimodal perception module is the robot's information source. Its key design features low latency and information complementarity. Through a heterogeneous sensor array, it captures multi-dimensional data generated by the user during natural interactions in parallel, providing rich and comprehensive perceptual input for subsequent intent understanding. The module includes a voice interaction submodule, a visual perception submodule, and a context perception submodule.

[0039] Furthermore, in some preferred embodiments of the present invention, the multimodal perception module includes: a voice interaction submodule, a visual perception submodule, and a context perception submodule; the voice interaction submodule includes: an end-to-end speech recognition model based on a convolutional recurrent neural network and a lightweight acoustic feature analysis model; the speech recognition model is used to convert the user's speech signal into text; the acoustic feature analysis model is used to extract fundamental frequency, energy, and speech rate features based on the user's speech signal to quantify the user's state; the visual perception submodule is used to extract key points of the user's body in the visual image to analyze posture, and to use a preset convolutional neural network to classify facial images in the visual image to recognize expressions; the context perception submodule is used to process environmental context data to analyze the crowd density around the robot.

[0040] Specifically, the voice interaction submodule employs an end-to-end speech recognition model based on a convolutional recurrent neural network (CRNN) to convert audio streams into text in real time. The model's layered structure includes: a feature extraction layer consisting of three layers of a two-dimensional convolutional neural network (CNN) to extract robust local features layer-by-layer from the input audio spectrogram; a sequence modeling layer consisting of two layers of a bidirectional long short-term memory network (Bi-LSTM) to perform contextual modeling on the feature sequences output by the CNN, capturing long-term dependencies in the speech signal; and an output layer trained using a connection-time classification (CTC) loss function, directly aligning and decoding the network output into character sequences, eliminating the tedious phoneme alignment step and improving recognition efficiency and robustness. Simultaneously, a lightweight acoustic feature analysis model works in parallel to extract acoustic features such as fundamental frequency, energy, speech rate, and harmonic noise ratio from the original audio, inputting them into a support vector machine (SVM) classifier to quantify the user's urgency or emotional state.

[0041] The visual perception submodule aims to endow the robot with the ability to "read between the lines," extracting key information beyond language by analyzing the user's posture, facial expressions, and behavior. In the human pose estimation section, lightweight OpenPose or a YOLO-based variant is used to extract 2D or 3D human keypoints in real time at a frame rate of 30fps. This is used to analyze protective postures (such as clutching the abdomen), gait abnormalities (such as limping), and directional gestures. In the facial expression analysis section, a convolutional neural network (CNN) is used to classify facial images, identifying key expressions such as pain, neutrality, and confusion. To balance recognition sensitivity and false alarm rate, a dynamic confidence threshold (default value 0.75) is pre-set. Expression classification results are only passed to the decision center as valid input when the output probability is higher than this threshold, thus ensuring the reliability of emotion perception. In the scene understanding and OCR section, semantic segmentation of environmental images identifies scene signs such as "registration counter" and "pharmacies," and optical character recognition (OCR) reads documents or receipts that the user may present.

[0042] The context-aware submodule is responsible for perceiving the robot's macroscopic environment and its own state, providing a global context for decision-making. Using real-time point cloud data from the robot's 2D LiDAR as input, it is used not only for SLAM navigation but also for real-time analysis of surrounding crowd density through clustering algorithms to determine if the robot is in a crowded state, thus providing environmental context for the decision-making center.

[0043] Furthermore, in some preferred embodiments of the present invention, the context-aware submodule includes: an environment-aware model with a preset clustering algorithm; the context-aware submodule is used to set the neighborhood radius and minimum number of points parameters to distinguish pedestrian point cloud clusters, and calculate the crowd density value based on the number of clusters and the area of ​​the monitoring area based on the environment-aware model, and classify the current environmental state into sparse, medium or crowded levels based on the crowd density value.

[0044] Specifically, the context-aware submodule includes: an environment-aware model with a pre-defined clustering algorithm, i.e., a crowd density analysis unit; and the DBSCAN (density-based spatial clustering with noise) algorithm for real-time processing of the LiDAR point cloud to distinguish individual pedestrians. The key parameters of this algorithm are set as follows: a neighborhood radius (Eps) of 0.3 meters and a minimum number of points (MinPts) required to form a cluster of 3. This configuration effectively separates the point cloud clusters representing pedestrian legs from environmental noise such as walls and stationary obstacles. Based on the clustering results output by the DBSCAN algorithm, the crowd density in the area surrounding the robot is quantified and calculated using the following formula: Population density (ρ) = Number of clusters (N) / Area of ​​monitoring area (A) The monitoring area (A) is defined as a fan-shaped area with a radius of 5 meters centered on the robot (covering the robot's main interaction range). Based on the calculated density value (ρ), the environmental state is divided into three levels: "sparse" (ρ < 0.1 people / m²), "medium" (0.1 ≤ ρ < 0.3 people / m²), and "crowded" (ρ ≥ 0.3 people / m²). This level label is used as key environmental context information and is provided to the decision center in real time to support the dynamic adjustment of the interaction strategy.

[0045] Step S104: The intent understanding and decision center module performs information fusion and intent understanding on the collected multimodal data to generate decision instructions.

[0046] Specifically, the Intent Understanding and Decision-Making Center module is the core of the robot's intelligence, realizing the transformation from "data" to "decision-making". The module includes three sub-modules: a multimodal information fusion unit, an intent recognition and state evaluator, and a dynamic dialogue management and task planner.

[0047] Furthermore, in some preferred embodiments of the present invention, the intent understanding and decision center module includes: a multimodal information fusion unit, an intent recognition and state evaluator, and a dynamic dialogue management and task planner; the multimodal information fusion unit is used to dynamically weight and fuse feature vectors from different modalities using a preset attention-based feature fusion algorithm to generate a user state embedding vector; the intent recognition and state evaluator is used to perform semantic similarity matching and reasoning between the user state embedding vector and a pre-constructed medical knowledge graph to output a comprehensive state label; wherein, the comprehensive state label includes: confidence level, recommended department, urgency level, and whether assistance is needed; the dynamic dialogue management and task planner is used to dynamically generate or adjust task sequences based on the comprehensive state label using a hierarchical task network planner.

[0048] Specifically, the multimodal information fusion engine employs a feature-based fusion strategy, the core of which lies in utilizing an attention mechanism to achieve adaptive weighting of cross-modal information. First, heterogeneous features from each sub-module (text embedding vectors, facial expression classification probabilities, pose keypoint coordinates, acoustic feature vectors, etc.) are mapped to the same high-dimensional semantic space through a fully connected network, forming an initial set of feature vectors V=[v voice v vision v context [...]. Subsequently, an attention mechanism layer is used to dynamically evaluate and weigh the importance of different modalities in the current context.

[0049] The intent recognition and state evaluator includes a medical knowledge graph query engine. This knowledge graph stores medical knowledge in triples of (head entity, relation, tail entity), constructing a vast semantic network.

[0050] The Dynamic Dialogue Management and Task Planner is a Hierarchical Task Network (HTN) planner. The planner employs a three-layer structure for task planning: Top layer (task element): Defines abstract, composite tasks, such as triage. Middle layer (method): Decomposes the composite task into a more specific sequence of subtasks.

[0051] Furthermore, in some preferred embodiments of the present invention, a multimodal information fusion processor is used to calculate the attention score of the feature vector of each modality, normalize the attention score using a preset function to obtain the weight of each modality, and perform a weighted summation of the feature vectors of all modalities according to the weights to generate a user state embedding vector.

[0052] Specifically, the process of generating user state embedding vectors is as follows: The eigenvector of the i-th mode Calculate an attention score : ; Where W is a pre-defined trainable weight matrix, b is the corresponding bias term, u is a pre-defined trainable background vector used to capture the importance of the modality, and tanh is the activation function.

[0053] The attention scores for all modalities are normalized using the Softmax function to obtain the final attention weights. This weight reflects the contribution of each mode: ; i,j=1,2,3,…,n; in, The attention score is calculated for the feature vector of the j-th modality, where n is the number of modalities; the feature vectors of all modalities are assigned their corresponding attention weights. Perform weighted summation to generate a final, unified, semantically rich patient state embedding vector s: ; This mechanism allows for the autonomous adjustment of information dependence based on specific contexts. For example, when visual perception detects a user's expression of pain, but the vocal modality shows a calm tone, the attention mechanism automatically assigns a higher weight α_vision to the visual feature v_vision. This results in a more accurate reflection of the user's potential pain state in the generated overall state vector s, providing a more reliable basis for subsequent decision-making.

[0054] For example, a semantic network content fragment is shown below: (Abdominal pain, relevant department: gastroenterology); (Chest pain, urgency level, high); (Persistent headache, related department: neurology); (Joint swelling and pain, related department: orthopedics); (Fever, possibly accompanied by sore throat); The core process of semantic similarity matching between the "patient state embedding vector" and symptom entities in the knowledge graph is as follows: ① Vectorization representation: First, using knowledge graph embedding technology, all symptom entities in the graph (such as "abdominal pain" and "chest pain") are also mapped to the same semantic vector space as the "patient state embedding vector".

[0055] ② Similarity calculation: Next, the cosine similarity between the "patient state embedding vector" and each symptom entity vector in the knowledge graph is calculated to measure their semantic relevance.

[0056] ③ Retrieval and Reasoning: Retrieve the most similar symptom entities and traverse and reason along the relationship paths defined in the knowledge graph (such as "related departments" and "urgency level").

[0057] ④ Output Generation: Finally, aggregate the inference results of all highly relevant symptoms to obtain a preliminary list of recommended departments and their confidence levels. Simultaneously, synthesize all information and output a structured comprehensive status label, in the form of: {Confidence: 0.7, Recommended Departments: [Neurology, Orthopedics], Urgency: Medium, Assistance Required: Yes}.

[0058] Furthermore, in some preferred embodiments of the present invention, a hierarchical task network planner is used to dynamically generate or adjust task sequences, including: when the confidence level is lower than a preset threshold, inserting multiple rounds of dialogue into the task sequence to clarify subtasks; when receiving an event with high urgency, immediately suspending the current task sequence and prioritizing the task corresponding to the event with high urgency for execution.

[0059] Specifically, the triage task can be broken down into different methods: Method 1 (Standard Procedure): [Greeting, inquiring about chief complaint, confirming department, providing guidance].

[0060] Method 2 (process needs clarification): [Greeting, inquiring about chief complaint, (clarifying symptoms), confirming department, providing guidance].

[0061] The (clarifying symptoms) subtask is an optional dynamic subtask triggered by intent confidence.

[0062] Low-level (atomic actions): Indivisible basic operation commands. For example: synthesizing speech ("What kind of headache do you have?"), displaying AR arrows, and updating user status.

[0063] The planner dynamically selects and executes decomposition methods based on the confidence level output by the "intent recognizer". When the confidence level output by the "intent recognizer" is lower than a preset threshold (such as 0.7), the planner automatically switches from "method one" to "method two" and inserts a symptom clarification subtask and its corresponding atomic action (such as asking "is it a persistent throbbing pain or a paroxysmal stabbing pain?").

[0064] When the planner receives a high-priority event (such as visually recognizing a user falling), the planner immediately executes interrupt handling, as follows: ① Clear task stack: Aborts all currently executing tasks and subtasks.

[0065] ②Highest priority: Prioritize the emergency response task and immediately break it down into an atomic action sequence: [Voice reassurance ("I have notified medical staff, please remain calm"), send an emergency alert to the nurses' station, and continuously monitor vital signs].

[0066] ③Recovery Mechanism: After the emergency is handled, the planner can decide whether to resume the previous task or start a new interactive session based on the context state before the interruption.

[0067] Step S106: Execute decision instructions through the behavior execution and guidance module to perform human-like interaction by controlling the joint movement of the intelligent triage robot and / or render and overlay augmented reality navigation arrows on the display screen of the intelligent triage robot to perform visual guidance.

[0068] Specifically, the behavior execution and guidance module is responsible for transforming the abstract instruction code generated by the upper-level decision center into embodied, perceptible, and natural behaviors that conform to human social habits. Through anthropomorphic interaction, immersive guidance, and precise system collaboration, the robot's services are seamlessly integrated into the real hospital environment, thereby providing users with an intuitive, efficient, and user-friendly experience.

[0069] The anthropomorphic interaction unit is responsible for transforming abstract instructions into natural, emotionally resonant, and embodied interactive behaviors. Employing the FastSpeech2 end-to-end speech synthesis model based on deep neural networks, it can directly generate highly natural-looking speech waveforms from text sequences.

[0070] Furthermore, in some preferred embodiments of the present invention, the behavior execution and guidance module includes: a humanoid interaction unit, an AR visual guidance unit, and a system collaboration interface; the humanoid interaction unit is used to generate control commands based on a preset semantic-action mapping library to drive the neck and waist joints of the intelligent triage robot to perform nodding, turning, or indicating gestures; the AR visual guidance unit is used to obtain the real-time pose of the intelligent triage robot based on the real-time positioning and mapping technology of LiDAR and vision fusion, and render dynamic 3D navigation arrows on the display screen according to the planned path, and update the AR registration information in real time according to the robot's movement; the system collaboration interface is used to exchange data with the hospital information system to send structured alarm messages.

[0071] Specifically, this module generates body movements closely related to the semantics of the dialogue by controlling the robot's neck and waist joints. Its core generation rules are based on a pre-defined semantic-action mapping library, which defines the correspondence between keyword semantics and specific actions.

[0072] The system collaboration interface provides several standardized RESTful API interfaces for secure and reliable data exchange with existing hospital systems (such as the HIS hospital information system and nurse call system) via HTTPS protocol. All data interactions are encapsulated in JSON format to ensure the structured and parsable nature of the information. For example, after the intent understanding and decision-making center confirms an emergency, this interface automatically constructs and sends a structured alarm message to the nurse station call system.

[0073] The following are examples of API requests and responses: API request example (bot → nurse station system): POST https: / / his-hospital.com / api / v1 / emergency-alert Headers: Content-Type: application / json Authorization: Bearer<access_token> Body: { "event_id": "ALERT_20250101_000000", "event_type": "PATIENT_SYNCOPE", / / Event type: suspected fainting "robot_id": "Robot_01", "location": "Outpatient Hall Area A" "timestamp": "2025-01-01 00:00:00", "confidence": 0.88, / / Identify confidence level "patient_status": { / / Patient status snapshot "facial_expression": "pain", "posture": "fallen" }, "suggested action": "PRIORITY_ASSISTANCE" / / Suggested action: Prioritize assistance } API response example (Nurse Station System → Robot): Status: 200 OK Body: { "code": 200, "message": "Alert received and assigned.", "data": { "alert_id": "HIS_ALERT_123456", "nurse_station": "Station_3", "estimated_response_time": 120 / / Estimated response time (seconds) } } Furthermore, in some preferred embodiments of the present invention, the anthropomorphic interaction unit is used to trigger a nodding action when the voice content contains affirmative semantics; to trigger the intelligent triage robot to make a slightly sideways posture when it is recognized that the intelligent triage robot is listening to the user's statement; and to simultaneously execute indicative gestures when providing directional guidance.

[0074] Specifically, when the voice content contains affirmative or agreeable semantics (such as "okay" or "no problem"), a nodding action is triggered; when listening to the user's statement, the robot will make a slight sideways posture to convey focus and respect; when providing directional guidance, it combines AR guidance with simultaneous execution of directional gestures, with the robot's arm extending to point in the target direction.

[0075] Furthermore, in some preferred embodiments of the present invention, the AR visual guidance unit is used to achieve positioning in a pre-constructed hospital map by fusing lidar point cloud data and visual data, and to calculate the optimal path based on the navigation target point.

[0076] Specifically, the AR visual guidance unit is tightly coupled with the robot's localization system. First, using SLAM technology based on LiDAR and visual fusion, the robot's precise pose within a pre-built hospital map is acquired in real time, achieving an absolute positioning accuracy better than ±5 cm in a typical hospital indoor environment. Then, based on the navigation target point, the optimal path from the current position to the target is calculated. Finally, on the robot's real-time video stream, a 3D arrow with a fade-out effect is accurately superimposed onto the real-world passageway at a refresh rate of 30 fps using a graphics rendering engine, creating an immersive guidance effect.

[0077] The robot-based triage method based on multimodal data provided in this invention has the following outstanding advantages: 1. More natural and user-friendly interaction, bridging the digital divide. By using human-like voice, facial expressions, and gestures for interaction, this invention replaces traditional touchscreen operation, making it easy for elderly people and users unfamiliar with electronic devices to use, significantly lowering the barrier to entry. In simulated tests involving users of different age groups, the first-time operation success rate of the human-like multimodal interaction provided by this invention reached over 97%, while the first-time operation success rate of traditional touchscreen self-service machines was only about 75%.

[0078] 2. More accurate and reliable triage, reducing the risk of misdiagnosis. Employing multimodal information fusion and intelligent dialogue management, this system integrates multi-dimensional information such as voice, facial expressions, and body language, and clarifies ambiguous descriptions through proactive questioning. In simulation tests based on a real case database, the overall triage accuracy of this invention reached 95.2%, significantly higher than the 78.5% of traditional self-service machine systems. For atypical symptoms with vague descriptions, the triage accuracy remains above 90% through multiple rounds of dialogue clarification, effectively avoiding misdiagnosis caused by unclear patient descriptions.

[0079] 3. Possess context awareness capabilities to enable proactive service. Through visual and contextual awareness, it can proactively identify user emergency states (such as sudden collapse or painful expressions) and environmental conditions, and promptly trigger emergency response within 3 seconds of an emergency to provide differentiated services.

[0080] 4. More intuitive and efficient guidance, optimizing the medical experience. By using AR visual guidance technology, dynamic navigation arrows are overlaid in real-world scenes to provide immersive guidance that is "what you see is what you go", effectively solving the problem of traditional map navigation being unintuitive.

[0081] 5. More flexible task planning, adaptable to complex scenarios Employing a dynamic task planning mechanism, it can intelligently handle complex situations such as multi-user inquiries and emergency interruptions, demonstrating strong scenario adaptability and system robustness.

[0082] 6. Enhanced system synergy and improved operational efficiency. By deeply integrating with the hospital information system through standardized interfaces, data interoperability and business linkage are achieved, and the robot is organically integrated into the hospital's digital workflow.

[0083] In summary, the robot triage method based on multimodal data provided in this invention effectively solves the core problems of existing technologies, such as unfriendly interaction, one-sided understanding, and rigid planning, through systematic innovation in multimodal perception, deep understanding, dynamic decision-making, and human-like execution. It has achieved significant progress in triage accuracy, user experience, and scenario adaptability.

[0084] This invention provides a robot-based triage method based on multimodal data, applied to an intelligent triage robot. The intelligent triage robot includes a multimodal perception module, an intent understanding and decision-making center module, and a behavior execution and guidance module. The method includes: collecting multimodal data through the multimodal perception module; wherein, the multimodal data includes: the user's voice signal, visual images, and environmental context data; performing information fusion and intent understanding on the collected multimodal data through the intent understanding and decision-making center module to generate decision instructions; and executing the decision instructions through the behavior execution and guidance module to perform human-like interaction by controlling the joint movement of the intelligent triage robot and / or rendering and overlaying augmented reality navigation arrows on the display screen of the intelligent triage robot to perform visual guidance. This method overcomes the problems of unfriendly interaction, single perception, and rigid planning in existing technologies, significantly improving triage accuracy, user experience, and adaptability in complex medical scenarios.

[0085] Example 2 Based on the above embodiments, this invention provides an intelligent triage robot for executing the robot triage method based on multimodal data provided in the above embodiments.

[0086] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the intelligent triage robot described above can be referred to the corresponding process in the aforementioned embodiments of the robot triage method based on multimodal data, and will not be repeated here.

[0087] Example 3 This invention also provides an electronic device for running a robot-based triage method based on multimodal data; see [link to related documentation]. Figure 3 The schematic diagram of an electronic device provided by the embodiment of the present invention shown above includes a memory 400 and a processor 401. The memory 400 is used to store one or more computer instructions, which are executed by the processor 401 to implement the above-mentioned robot-guided diagnosis method based on multimodal data.

[0088] Furthermore, Figure 3 The electronic device shown also includes a bus 402 and a communication interface 403. The processor 401, the communication interface 403 and the memory 400 are connected via the bus 402.

[0089] The memory 400 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 403 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 402 can be an ISA bus, PCI bus, or EISA bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0090] Processor 401 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 401 or by instructions in software form. Processor 401 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this invention can be directly manifested as execution by a hardware decoding processor, or execution by a combination of hardware and software modules in the decoding processor. The software module can reside in a readily available storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 400, and processor 401 reads information from memory 400 and, in conjunction with its hardware, completes the steps of the method described in the foregoing embodiments.

[0091] This invention also provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are called and executed by a processor, they cause the processor to implement the aforementioned robot-guided diagnosis method based on multimodal data. For specific implementation details, please refer to the method embodiments, which will not be repeated here.

[0092] The computer program products of the robot-guided diagnosis method, device, and electronic device based on multimodal data provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0093] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and / or device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0094] Furthermore, in the description of the embodiments of the present invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0095] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A robot-based triage method based on multimodal data, characterized in that, The method is applied to an intelligent triage robot, which includes: a multimodal perception module, an intent understanding and decision-making center module, and a behavior execution and guidance module; the method includes: Multimodal data is acquired through the multimodal perception module; wherein, the multimodal data includes: the user's voice signal, visual image, and environmental context data; The intent understanding and decision center module performs information fusion and intent understanding on the collected multimodal data to generate decision instructions; The decision-making instructions are executed by the behavior execution and guidance module to perform human-like interaction by controlling the joint movements of the intelligent triage robot and / or to render and overlay augmented reality navigation arrows on the display screen of the intelligent triage robot to perform visual guidance.

2. The robot-guided medical service method based on multimodal data according to claim 1, characterized in that, The multimodal perception module includes: a voice interaction submodule, a visual perception submodule, and a context perception submodule; the voice interaction submodule includes: an end-to-end speech recognition model based on a convolutional recurrent neural network and a lightweight acoustic feature analysis model; The speech recognition model is used to convert the user's speech signal into text; The acoustic feature analysis model is used to extract fundamental frequency, energy, and speech rate features based on the user's speech signal to quantify the user's state. The visual perception submodule is used to extract the user's key human body points in the visual image to analyze posture, and to use a preset convolutional neural network to classify the facial images in the visual image to recognize expressions. The context-aware submodule is used to process the environmental context data to analyze the crowd density around the robot.

3. The robot-guided medical service method based on multimodal data according to claim 2, characterized in that, The context-aware submodule includes: an environment-aware model with a pre-set clustering algorithm; the context-aware submodule is used to set the neighborhood radius and minimum number of points parameters to distinguish pedestrian point cloud clusters, and calculate the crowd density value based on the number of clusters and the area of ​​the monitoring area based on the environment-aware model, and classify the current environmental state into sparse, medium or crowded levels based on the crowd density value.

4. The robot-guided medical service method based on multimodal data according to claim 1, characterized in that, The intent understanding and decision-making center module includes: a multimodal information fusion unit, an intent recognition and state evaluator, and a dynamic dialogue management and task planner; The multimodal information fusion unit is used to dynamically weight and fuse feature vectors from different modalities using a preset attention-based feature fusion algorithm to generate a user state embedding vector. The intent recognition and state evaluator is used to perform semantic similarity matching and reasoning between the user state embedding vector and a pre-built medical knowledge graph, and output a comprehensive state label; wherein, the comprehensive state label includes: confidence level, recommended department, urgency level, and whether assistance is needed. The dynamic dialogue management and task planner is used to dynamically generate or adjust task sequences based on the comprehensive status labels using a hierarchical task network planner.

5. The robot-guided medical service method based on multimodal data according to claim 4, characterized in that, The multimodal information fusion unit is used to calculate the attention score of the feature vector of each modality, normalize the attention score using a preset function to obtain the weight of each modality, and perform a weighted summation of the feature vectors of all modalities based on the weights to generate the user state embedding vector.

6. The robot-guided medical service method based on multimodal data according to claim 4, characterized in that, A hierarchical task network planner is used to dynamically generate or adjust task sequences, including: When the confidence level is lower than a preset threshold, multiple rounds of dialogue are inserted into the task sequence to clarify the sub-tasks; When an event with high urgency is received, the current task sequence is immediately terminated, and the task corresponding to the event with high urgency is put on top for execution.

7. The robot-guided medical service method based on multimodal data according to claim 1, characterized in that, The behavior execution and guidance module includes: a human-like interaction unit, an AR visual guidance unit, and a system collaboration interface; the human-like interaction unit is used to generate control commands based on a preset semantic-action mapping library to drive the neck and waist joints of the intelligent triage robot to perform nodding, turning to the side, or indicating gestures; The AR visual guidance unit is used to obtain the real-time pose of the intelligent triage robot based on the real-time positioning and mapping technology of LiDAR and vision fusion, and to render dynamic 3D navigation arrows on the display screen according to the planned path. The system collaboration interface is used to exchange data with the hospital information system in order to send structured alarm messages.

8. The robot-guided medical service method based on multimodal data according to claim 7, characterized in that, The anthropomorphic interaction unit is used to trigger a nodding action when the voice content contains affirmative semantics; when it is recognized that the intelligent triage robot is listening to the user's statement, it triggers the control of the intelligent triage robot's torso to make a slightly sideways posture; and when providing directional guidance, it simultaneously executes indicative gestures.

9. The robot-guided medical service method based on multimodal data according to claim 7, characterized in that, The AR visual guidance unit is used to achieve positioning in a pre-constructed hospital map by fusing lidar point cloud data and visual data, and to calculate the optimal path based on the navigation target point.

10. An intelligent triage robot, characterized in that, Used to perform the robot-guided diagnosis method based on multimodal data as described in any one of claims 1 to 9.

Images