Integrated mobile digital human interactive collaborative system and method

Abstract

The present invention discloses an integrated mobile digital human interactive collaborative system and method thereof, belonging to the field of human-computer interaction technology, including a mobile platform module, a central control module, an identification and perception module, an information sharing and access module, a crowd identification module, and a user interaction module. The mobile platform module includes a chassis system for receiving relevant instructions and providing power for movement through the chassis system. The information sharing and access module accesses the hospital's internal information network via the Internet, receives relevant information within the hospital, and transmits it to the central control module and the crowd identification module. The integrated mobile digital human interactive collaborative system and method thereof of the present invention can realize user interaction, medical guidance and consultation, active patient identification, and suspicious person marking. It integrates mobile platform, intelligent identification, information sharing, and user interaction functions, can improve emergency response efficiency and patient experience, and enhances practicality and functionality.

Description

Technical Field

[0001] The present invention belongs to the field of human-computer interaction technology, and specifically relates to an integrated movable digital human interactive collaborative system and method thereof. Background Art

[0002] With the popularization of artificial intelligence technology, robots equipped with human-computer interaction systems are being used in more and more places, including hotels, banks, hospitals, etc., to provide guidance and services to consumers or patients.

[0003] Most of the existing hospital human-computer interaction collaborative systems are fixed and have many obvious shortcomings. First, since patients need to actively use the system before the service can be provided, in a crowded and mixed environment such as the guidance hall, the system cannot actively provide guidance services to patients in urgent need of help, which greatly reduces the practicality of the system. In addition, this type of system cannot actively contact patrol personnel based on the hospital's emergency information to plan and clear routes. In an emergency, such as when a large number of emergency patients rush in at the same time, the system cannot effectively coordinate resources and plan the best route, which may cause problems such as channel congestion and delayed rescue. In addition, the lack of the function of actively contacting patrol personnel makes it impossible to quickly organize manpower for effective response and processing when an emergency occurs, resulting in low functionality.

[0004] In response to the above, this case proposes an integrated mobile digital human interactive collaborative system and method to solve the above technical problems. Summary of the Invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes an integrated mobile digital human interactive collaborative system and method thereof, which solves the above technical problems by improving detection and processing methods.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] An integrated mobile digital human interactive collaborative system and method thereof, comprising a mobile platform module, a central control module, an identification and perception module, an information sharing and access module, a crowd identification module, and a user interaction module;

[0008] The mobile platform module includes a chassis system, which is used to receive relevant instructions and move by providing power through the chassis system;

[0009] The information sharing access module accesses the hospital's internal information network through the Internet, receives relevant information from the hospital, and transmits it to the central control module and the crowd identification module;

[0010] The central control module generates a virtual image based on artificial intelligence technology and semantic recognition technology, reviews the received movement instructions, and transmits the control instructions to the mobile platform module. Based on the received emergency information and the hospital floor plan, it generates a predicted route and transmits it to the mobile platform module and the user interaction module;

[0011] The recognition and perception module includes a camera, radar, and gyroscope, which are used to sense the surrounding environment, collect surrounding image information, determine position, direction, and distance information, transmit relevant environmental information to the central control module, and transmit image information to the crowd recognition module;

[0012] The crowd recognition module identifies the crowd in the guidance hall by using the image data recognized by the camera of the recognition perception module, and classifies the medical urgency of the medical crowd based on their movements and expressions. According to the urgency of the medical crowd, the module transmits instructions to the central control module and actively moves to the emergency crowd to provide guidance services.

[0013] The user interaction module includes a control screen, which is used to identify the patient's relevant problems and generate relevant information to interact with the patient through the control screen, including medical recommendations, payment information, and registration information. At the same time, based on the emergency data of the information sharing access module and the predictive route planning transmitted by the central control module, the control screen generates interactive buttons.

[0014] Furthermore, the central control module generates a virtual image based on artificial intelligence technology and semantic recognition technology, reviews the received movement instructions, and transmits the control instructions to the mobile platform module. Based on the received emergency information and the hospital floor plan, it generates a predicted route and transmits it to the mobile platform module and the user interaction module, including the following steps:

[0015] Through artificial intelligence and semantic recognition technology, a virtual image is built to achieve user interaction and central control. Based on the surrounding environment information transmitted by the recognition and perception module, movement instructions are generated and transmitted to the mobile platform module for movement.

[0016] Based on the emergency information in the information sharing access module and combined with the current hospital floor plan, an emergency route is generated and the relevant information data is transmitted to the user interaction module. At the same time, combined with the surrounding environment image of the recognition and perception module, movement instructions are generated and transmitted to the mobile platform module, which automatically searches for patrol personnel in the guidance hall for user interaction.

[0017] Furthermore, the first aid information in the information sharing access module is combined with the current hospital floor plan to generate an emergency route, and the relevant information data is transmitted to the user interaction module. At the same time, combined with the surrounding environment picture of the recognition and perception module, a movement instruction is generated, and the movement instruction is transmitted to the mobile platform module to automatically find the patrol personnel in the guidance hall for user interaction, including the following steps:

[0018] Through the information sharing access module, emergency information is obtained, including the estimated time of return of the ambulance, the ambulance return location, and the patient's relevant department. The relevant emergency route is generated based on the current hospital floor plan. The specific steps are as follows:

[0019] Obtain data related to the hospital floor plan, including the hospital's floor layout, the locations of various departments, elevator locations, and major intersections. Combined with the ambulance's return location, build a graph structure:

[0020] Each department room is regarded as a department node, the elevator location and main intersection are marked as key location nodes, and the ambulance return location is marked as the starting node, where the department node, starting node and key location node represent the nodes of the graph structure;

[0021] Set up corridor connections, which represent direct connections between different departments on a floor, and elevator connections, which represent connections between floors. Corridor connections and elevator connections are edges in the graph structure.

[0022] Path planning is performed using the Dijkstra algorithm, and its algorithm formula is:

[0023] For each node A in the graph, initialize the shortest path estimate d(A) from node A to the starting point Q:

[0024]

[0025] Among them, Q represents the starting point, and the distances to the remaining nodes are infinite. Insert the starting point Q into the priority queue with a priority of 0. Take the node B with the smallest current distance from the priority queue. For each node A adjacent to B, if the path to A through B is shorter than the currently known shortest path, update the distance of A:

[0026] d(A)=min(d(A),d(B)+ω(B,A));

[0027] Where d(A) represents the currently known shortest distance from node A to the starting point, d(B) represents the currently known shortest distance from node B to the starting point, and ω(B, A) represents the actual distance from node B to node A. When node B taken out of the priority queue is the target node C, path planning is complete. At this time, d(C) is the shortest distance from the starting point Q to the target node C. The actual path is obtained by tracing back the predecessor node.

[0028] The actual path information is transmitted into the user interaction module, and the countdown of the estimated return time of the ambulance is generated and entered into the user interaction module. Combined with the surrounding environment picture of the recognition and perception module, the movement instructions are generated and transmitted to the mobile platform module, which automatically searches for patrol personnel in the guidance hall for user interaction.

[0029] Furthermore, the crowd recognition module identifies the crowd in the guidance hall by using the image data recognized by the camera of the recognition perception module, and divides the medical urgency of the medical crowd based on their movements and expressions. According to the urgency of the medical crowd, the module transmits instructions to the central control module, and actively moves to the emergency crowd to provide guidance services, including the following steps:

[0030] By identifying the image data recognized by the camera of the recognition module, the gait frequency and facial expressions of the people in the guidance hall are identified. Based on the speed of the gait frequency and the emotional score of the facial expression, the urgency level of the medical patients in the image data is classified;

[0031] According to the emotional urgency and cadence urgency of the guided group, instructions are transmitted to the central control module, and the module actively moves to the emergency group to provide guidance services.

[0032] Furthermore, the cadence and facial expressions of the people in the consultation hall are recognized, and based on the cadence speed and the emotional score of the facial expressions, the urgency level of the medical crowd in the screen data is divided. The specific steps are:

[0033] Combined with real-time image data, the motion characteristics of relevant personnel are extracted and the step frequency v0 is calculated. The algorithm formula is:

[0034]

[0035] Where v0 represents the cadence, p represents the number of steps per unit time, and Δt represents the unit time. Based on the low cadence threshold v1 and the high cadence threshold v2, the cadence v0 is determined as follows:

[0036] When v0 < v1, it means that the relevant personnel have a slow cadence, indicating that the cadence urgency is low;

[0037] When v1≤v0<v2, it means that the cadence of the relevant personnel is medium, indicating that the cadence urgency is medium;

[0038] When v0 ≥ v2, it means that the relevant personnel have a rapid cadence, indicating a high cadence urgency;

[0039] The facial expression recognition algorithm is used to identify and analyze the facial expressions of relevant personnel, including anxiety, pain, and calmness, and further score the emotions. The algorithm formula is:

[0040]

[0041] Among them, O0 represents the current emotion score, ω i Represents the weight of each emotion, e i Indicates the corresponding emotional intensity. The emotional score is the weighted value of the emotional intensity of the facial expression of the relevant person in unit time. Based on the low threshold O1 and the high threshold O2 of the emotional score, the emotions of the relevant people are graded:

[0042] When O0 < O1, it means that the emotions of the relevant personnel are stable, indicating that the emotional urgency is low;

[0043] When O1≤O0<O2, it means that the relevant personnel are in a moderate mood, indicating a medium level of emotional urgency;

[0044] When O0 ≥ O2, it means that the relevant personnel are nervous, indicating a high level of emotional urgency.

[0045] Furthermore, according to the emotional urgency and cadence urgency of the guided group, the instructions are transmitted to the central control module, and the guide actively moves to the emergency group to provide guidance services. The specific steps are as follows:

[0046] According to the emotional urgency and cadence urgency of the guided group, set the priority to high, medium, or low. The specific steps are as follows:

[0047] High priority: high cadence urgency + high emotional urgency, high cadence urgency + medium emotional urgency, high cadence urgency + low emotional urgency, medium cadence urgency + high emotional urgency, low cadence urgency + high emotional urgency;

[0048] Medium priority: medium cadence urgency + medium emotional urgency, medium cadence urgency + low emotional urgency, low cadence urgency + medium emotional urgency;

[0049] Low priority: low cadence urgency + low emotional urgency;

[0050] According to the priority of the guided group, the instructions are transmitted to the central control module, and based on the priority of high, medium or low, the module actively moves to the emergency group to provide guidance services.

[0051] Furthermore, the crowd recognition module is also provided with a monitoring system, which recognizes and marks suspicious actions of people in the guidance hall by identifying the image information transmitted by the perception module, and further determines suspicious persons in combination with the internal camera images of the hospital in the information sharing access module, and transmits instructions to the central control module based on the determination results. The specific steps are as follows:

[0052] The recognition and perception module obtains video clips of people's behavior inside the screen, and builds a recognition model based on the neural network model. The steps are as follows:

[0053] Collect video clips containing various suspicious and normal behaviors, divide the collected video clips into training sets, validation sets, and test sets, and select appropriate features for feature extraction based on the characteristics of suspicious behaviors, including hand movement trajectories and changes in people's postures;

[0054] Select a convolutional neural network (CNN) as the neural network model. Train the convolutional neural network model using the training set. Verify the trained recognition model using the validation set to adjust the model's parameters or structure. Test the recognition model using the test set to evaluate its performance in identifying suspicious behavior.

[0055] The recognition model is used to identify the real-time video images transmitted by the recognition perception module. The identified suspicious persons are marked as suspicious persons - pending. The YOLO algorithm is used to extract clothing features of the relevant suspicious persons - pending and obtain a timestamp.

[0056] Through the information sharing access module, the internal camera footage of the hospital is synchronized with the clothing feature data of the suspicious person based on the timestamp. The clothing features of the suspicious person are compared with the person in the internal camera footage through image processing technology. The match degree is determined by Euclidean distance, and the suspicious person is marked as "pending" in the internal camera footage data.

[0057] The recognition model is used to perform secondary behavior recognition on suspicious persons in the internal camera image data. When the recognition result is confirmed, the relevant person is marked as a suspicious person, and the mobile signal is transmitted to the central control module and moved to the patrol personnel. At the same time, the picture of the suspicious person is captured and transmitted to the user interaction module.

[0058] Furthermore, the user interaction module includes a control screen for identifying relevant questions of the patient and generating relevant information for interaction with the patient through the control screen, including medical recommendations, payment information, and registration information. At the same time, based on the emergency data of the information sharing access module and the predicted route planning transmitted by the central control module, the control screen generates interactive buttons, including the following steps:

[0059] It contains a hospital information database, which includes a basic introduction to the hospital and information about departments. The virtual image generated by the central control module identifies patients' relevant questions and answers them based on the relevant data in the hospital information database, including medical recommendations, payment information, and registration information. At the same time, virtual buttons are generated on the control screen to assist patients in interaction, including navigation routes, online appointments, and fee information inquiries. At the same time, patients interact with the system through touch operations on the control screen to inquire about medical recommendations, check payment information, and register;

[0060] For emergency medical information, after it reaches the patrol officer, a virtual button for route planning is generated on the control screen. The patrol officer clicks the virtual button to obtain the actual route information and generates the estimated time for the ambulance to return on the control screen.

[0061] Proactively provide guidance services. When moving to an emergency group, generate voice announcements and inquire with the relevant emergency group.

[0062] After a suspicious person is marked, when he moves to the patrol officer, a red virtual button is generated through the control screen. The patrol officer clicks on the virtual button to view the suspicious person's picture and related video clips of the suspicious person's suspicious behavior. At the same time, the control screen generates a confirmation button. The patrol officer ends the monitoring by clicking the confirmation button.

[0063] Furthermore, the mobile platform module includes a chassis system, including an electric motor, drive wheels and a battery, which is used to realize the movement of the mobile platform, and the movement is carried out by providing power through the chassis based on receiving relevant instructions;

[0064] The information sharing access module accesses the hospital's internal information network via the Internet and receives various data from the hospital in real time, including camera image data, hospital floor plans, and emergency information. It transmits the real-time camera image data to the crowd recognition module and transmits the hospital floor plans and emergency information to the central control module.

[0065] The recognition and perception module includes a camera, a radar, and a gyroscope, which are used to perceive the surrounding environment, determine the position, direction, and distance information, obtain the surrounding picture data information through the camera, transmit the relevant environmental information into the central control module for auxiliary movement, transmit the surrounding picture data information into the crowd recognition module for auxiliary judgment, and formulate a movement plan into the central control module.

[0066] The integrated mobile digital human interactive collaborative system includes the following steps:

[0067] S1. Generate a virtual avatar using artificial intelligence and semantic recognition technology, aggregate patient inquiries based on the hospital's information database, review received movement instructions, generate a predicted route based on the hospital's internal information sharing and the hospital's floor plan for emergency information, and then move to the patrol personnel;

[0068] S2. Interact with patrol personnel through the control screen to ensure that the predicted emergency route is unobstructed. Analyze the gait and demeanor of people in the reception hall to classify the urgency of the crowd, proactively move to the most urgent patients, and provide proactive inquiries.

[0069] S3. Combined with the camera recognition image data, suspicious actions of people in the guidance hall are identified and marked. Combined with the internal camera images of the hospital, further judgment is made on suspicious persons. After the judgment is confirmed, the person is moved to the patrol personnel and interacts with the patrol personnel through the control screen to effectively respond and handle the situation.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] 1. In the present invention, by setting up a user interaction module, it is possible to identify the inquiries of patients in the guidance hall, combine the hospital information database, and answer the relevant questions of patients in the guidance hall through the control screen to provide help to the patients in the guidance hall;

[0072] 2. In the present invention, by accessing the hospital's internal information network, when an emergency situation occurs, an emergency route is generated by combining the emergency patient information with the hospital floor plan, and a movement instruction is generated to actively move to the patrol personnel. The user interaction module interacts with the patrol personnel through the control screen to ensure the smooth flow of the emergency route, optimize the dispatch of patrol personnel and emergency equipment, and improve the overall rescue efficiency.

[0073] 3. In the present invention, by analyzing the camera image, the gait frequency and facial expressions of the people in the guidance hall, the patients in the image are classified according to their urgency, and the patient is actively moved to the high-priority patient. Through the user interaction module, voice inquiries are made, and help is actively provided. The situation in the guidance hall is analyzed and responded to in real time. The service strategy is dynamically adjusted according to the patient's urgency, thereby enhancing functionality.

[0074] 4. In the present invention, by analyzing the suspicious behavior of people in the guidance hall and combining it with the internal camera data for secondary judgment, it is easy to find suspicious people in the guidance hall. At the same time, patrol personnel are actively contacted for suspicious situations, and possible security threats are effectively identified, thereby improving the safety of the hospital, avoiding the occurrence of adverse events, and reducing the negative impact of safety accidents.

[0075] The entire integrated mobile digital human interactive collaborative system and its method can realize user interaction, medical guidance and consultation, active patient identification, and suspicious person marking. It integrates mobile platform, intelligent identification, information sharing and user interaction functions, can improve emergency response efficiency and patient experience, and enhance practicality and functionality. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a block diagram of the integrated mobile digital human interactive collaborative system and method of the present invention. DETAILED DESCRIPTION

[0077] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0078] Example 1:

[0079] like Figure 1 As shown, the integrated mobile digital human interactive collaborative system and method thereof include a mobile platform module, a central control module, an identification and perception module, an information sharing and access module, a crowd identification module, and a user interaction module;

[0080] The mobile platform module includes a chassis system, including an electric motor, drive wheels and batteries, which are used to realize the movement of the mobile platform and move the platform by providing power through the chassis based on receiving relevant instructions;

[0081] The information sharing access module is connected to the hospital's internal information network through the Internet, and receives various data from the hospital in real time, including camera image data, hospital floor plans, and emergency information. It transmits real-time camera image data to the crowd recognition module and transmits hospital floor plans and emergency information to the central control module.

[0082] The recognition and perception module, including cameras, radars, and gyroscopes, is used to sense the surrounding environment and determine location, direction, and distance. It obtains surrounding image data through the camera and transmits relevant environmental information to the central control module for auxiliary movement. It also transmits surrounding image data to the crowd recognition module for auxiliary judgment and formulates a movement plan and sends it to the central control module.

[0083] The central control module generates a virtual image based on artificial intelligence and semantic recognition technology, reviews the received movement instructions, and transmits the control instructions to the mobile platform module. Based on the received emergency information and the hospital floor plan, it generates a predicted route and transmits it to the mobile platform module and the user interaction module. The following steps are included:

[0084] Through artificial intelligence and semantic recognition technology, a virtual image is built to achieve user interaction and central control. Based on the surrounding environment information transmitted by the recognition and perception module, movement instructions are generated and transmitted to the mobile platform module for movement.

[0085] Based on the emergency information in the information sharing access module and the current hospital floor plan, an emergency route is generated and the relevant information data is transmitted to the user interaction module. At the same time, combined with the surrounding environment image of the recognition and perception module, a movement instruction is generated and transmitted to the mobile platform module, which automatically searches for patrol personnel in the guidance hall for user interaction, including the following steps:

[0086] Through the information sharing access module, emergency information is obtained, including the estimated time of return of the ambulance, the ambulance return location, and the patient's relevant department. The relevant emergency route is generated based on the current hospital floor plan. The specific steps are as follows:

[0087] Obtain data related to the hospital floor plan, including the hospital's floor layout, the locations of various departments, elevator locations, and major intersections. Combined with the ambulance's return location, build a graph structure:

[0088] Each department room is regarded as a department node, the elevator location and main intersection are marked as key location nodes, and the ambulance return location is marked as the starting node, where the department node, starting node and key location node represent the nodes of the graph structure;

[0089] Set up corridor connections, which represent direct connections between different departments on a floor, and elevator connections, which represent connections between floors. Corridor connections and elevator connections are edges in the graph structure.

[0090] Path planning is performed using the Dijkstra algorithm, and its algorithm formula is:

[0091] For each node A in the graph, initialize the shortest path estimate d(A) from node A to the starting point Q:

[0092]

[0093] Among them, Q represents the starting point, and the distances to the remaining nodes are infinite. Insert the starting point Q into the priority queue with a priority of 0. Take the node B with the smallest current distance from the priority queue. For each node A adjacent to B, if the path to A through B is shorter than the currently known shortest path, update the distance of A:

[0094] d(A)=min(d(A),d(B)+ω(B,A));

[0095] Where d(A) represents the currently known shortest distance from node A to the starting point, d(B) represents the currently known shortest distance from node B to the starting point, and ω(B, A) represents the actual distance from node B to node A. When node B taken out of the priority queue is the target node C, path planning is complete. At this time, d(C) is the shortest distance from the starting point Q to the target node C. The actual path is obtained by tracing back the predecessor node.

[0096] The actual path information is transmitted into the user interaction module, and the countdown of the estimated return time of the ambulance is generated and entered into the user interaction module. Combined with the surrounding environment picture of the recognition and perception module, the movement instructions are generated and transmitted to the mobile platform module, which automatically searches for patrol personnel in the guidance hall for user interaction.

[0097] It should be noted that d(B)+ω(B,A) represents the length of the path from node B to node A. If the distance of node A is updated, the priority of A needs to be updated in the priority queue, that is, the new distance value. d(A) represents the shortest path length from the starting point to node A up to the current moment, and d(B) represents the shortest path length from the starting point to node B up to the current moment. If the distance from node B to node A is shorter, that is, d(B)+ω(B,A) is less than the known d(A), then the distance of node A is updated to d(B)+ω(B,A).

[0098] Example 2:

[0099] The crowd recognition module uses the image data recognized by the recognition perception module camera to identify the crowd in the guidance hall. Based on the movements and expressions of the medical crowd, it divides the medical crowd into different levels of urgency. According to the urgency of the medical crowd, it transmits instructions to the central control module and actively moves to the emergency crowd to provide guidance services. The following steps are included:

[0100] By using the image data recognized by the perception module camera, the cadence and facial expressions of the people in the reception hall are identified. Based on the cadence speed and the emotional score of the facial expression, the urgency level of the medical personnel in the image data is classified. The specific steps are as follows:

[0101] Combined with real-time image data, the motion characteristics of relevant personnel are extracted and the step frequency v0 is calculated. The algorithm formula is:

[0102]

[0103] Where v0 represents the cadence, p represents the number of steps per unit time, and Δt represents the unit time. Based on the low cadence threshold v1 and the high cadence threshold v2, the cadence v0 is determined as follows:

[0104] When v0 < v1, it means that the relevant personnel have a slow cadence, indicating that the cadence urgency is low;

[0105] When v1≤v0<v2, it means that the cadence of the relevant personnel is medium, indicating that the cadence urgency is medium;

[0106] When v0 ≥ v2, it means that the relevant personnel have a rapid cadence, indicating a high cadence urgency;

[0107] The facial expression recognition algorithm is used to identify and analyze the facial expressions of relevant personnel, including anxiety, pain, and calmness, and further score the emotions. The algorithm formula is:

[0108]

[0109] Among them, O0 represents the current emotion score, ω i Represents the weight of each emotion, e i Indicates the corresponding emotional intensity. The emotional score is the weighted value of the emotional intensity of the facial expression of the relevant person in unit time. Based on the low threshold O1 and the high threshold O2 of the emotional score, the emotions of the relevant people are graded:

[0110] When O0 < O1, it means that the emotions of the relevant personnel are stable, indicating that the emotional urgency is low;

[0111] When O1≤O0<O2, it means that the relevant personnel are in a moderate mood, indicating a medium level of emotional urgency;

[0112] When O0 ≥ O2, it means that the relevant personnel are nervous, indicating a high level of emotional urgency.

[0113] It should be noted that when identifying and analyzing the facial expressions of relevant personnel, the relevant facial expressions are first mapped, including anxiety, pain, and calmness, and then mapped to emotion categories. For example, emotions are judged by facial features such as raised eyebrows and drooping corners of the mouth. The facial action unit model FACS is used to define facial action units (AUs) by the movement intensity of facial muscles. The intensity values of multiple action units are combined to estimate the overall intensity of each emotion. Each emotion may be composed of different action units, and the intensity of these action units contributes differently to the emotion. A weight is assigned to each emotion and adjusted according to the degree of emotional significance and the correlation between facial expressions, that is, the emotional intensity under different calm, painful, and anxious emotions is obtained:

[0114]

[0115] Among them, e i is the corresponding emotional intensity, i represents the type of emotion, including calm, pain, anxiety, Represents the weights of different action units under different types of emotions, Represents the values of different action units under different emotion types.

[0116] According to the emotional urgency and cadence urgency of the guided crowd, the instructions are transmitted to the central control module, and the system actively moves to the emergency crowd to provide guidance services. The specific steps are as follows:

[0117] According to the emotional urgency and cadence urgency of the guided group, set the priority to high, medium, or low. The specific steps are as follows:

[0118] High priority: high cadence urgency + high emotional urgency, high cadence urgency + medium emotional urgency, high cadence urgency + low emotional urgency, medium cadence urgency + high emotional urgency, low cadence urgency + high emotional urgency;

[0119] Medium priority: medium cadence urgency + medium emotional urgency, medium cadence urgency + low emotional urgency, low cadence urgency + medium emotional urgency;

[0120] Low priority: low cadence urgency + low emotional urgency;

[0121] According to the priority of the guided group, the instructions are transmitted to the central control module, and based on the priority of high, medium or low, the module actively moves to the emergency group to provide guidance services.

[0122] It should be noted that the priority of the target person is determined based on the high, medium and low priority of the guide group, combined with the priority determination table:

[0123] Table 1 Priority determination table

[0124]

[0125] Example 4:

[0126] The crowd recognition module is also equipped with a monitoring system. It uses the image information transmitted by the recognition and perception module to identify and mark suspicious movements of people in the guidance hall. It also combines the images of the hospital's internal cameras in the information sharing access module to further determine the suspicious persons. Based on the determination results, the system transmits instructions to the central control module. The specific steps are as follows:

[0127] The recognition and perception module obtains video clips of people's behavior inside the screen, and builds a recognition model based on the neural network model. The steps are as follows:

[0128] Collect video clips containing various suspicious and normal behaviors, divide the collected video clips into training sets, validation sets, and test sets, and select appropriate features for feature extraction based on the characteristics of suspicious behaviors, including hand movement trajectories and changes in people's postures;

[0129] Select a convolutional neural network (CNN) as the neural network model. Train the convolutional neural network model using the training set. Verify the trained recognition model using the validation set to adjust the model's parameters or structure. Test the recognition model using the test set to evaluate its performance in identifying suspicious behavior.

[0130] The recognition model is used to identify the real-time video images transmitted by the recognition perception module. The identified suspicious persons are marked as suspicious persons - pending. The YOLO algorithm is used to extract clothing features of the relevant suspicious persons - pending and obtain a timestamp.

[0131] Through the information sharing access module, the internal camera footage of the hospital is synchronized with the clothing feature data of the suspicious person based on the timestamp. The clothing features of the suspicious person are compared with the person in the internal camera footage through image processing technology. The match degree is determined by Euclidean distance, and the suspicious person is marked as "pending" in the internal camera footage data.

[0132] The recognition model is used to perform secondary behavior recognition on suspicious persons in the internal camera image data. When the recognition result is confirmed, the relevant person is marked as a suspicious person, and the mobile signal is transmitted to the central control module and moved to the patrol personnel. At the same time, the picture of the suspicious person is captured and transmitted to the user interaction module.

[0133] Example 5:

[0134] The user interaction module includes a control screen that identifies patients' related questions and generates relevant information for interaction with patients through the control screen, including treatment recommendations, payment information, and registration information. At the same time, based on the emergency data from the information sharing access module and the predicted route planning transmitted by the central control module, the control screen generates interactive buttons. The steps are as follows:

[0135] It contains a hospital information database, which includes a basic introduction to the hospital and information about departments. The virtual image generated by the central control module identifies patients' relevant questions and answers them based on the relevant data in the hospital information database, including medical recommendations, payment information, and registration information. At the same time, virtual buttons are generated on the control screen to assist patients in interaction, including navigation routes, online appointments, and fee information inquiries. At the same time, patients interact with the system through touch operations on the control screen to inquire about medical recommendations, check payment information, and register;

[0136] For emergency medical information, after it reaches the patrol officer, a virtual button for route planning is generated on the control screen. The patrol officer clicks the virtual button to obtain the actual route information and generates the estimated time for the ambulance to return on the control screen.

[0137] It should be noted that in the actual path information, a confirmation button is generated for each key node, including elevators and related paths. After the patrol personnel control an elevator or related path, they need to click the confirmation button to make the relevant position in the actual path information appear green, indicating that the relevant position is unblocked.

[0138] Proactively provide guidance services. When moving to an emergency group, generate voice announcements and inquire with the relevant emergency group.

[0139] It should be noted that the preset inquiry voices include: "Hello, I noticed that you may need help. May I ask if you or someone around you has encountered an emergency medical situation?"

[0140] “Please tell me exactly what happened and how you are feeling now”;

[0141] “Do you have any pain, discomfort, or other specific symptoms”;

[0142] "If you need to go to the emergency room, I can direct you there."

[0143] After a suspicious person is marked, when he moves to the patrol officer, a red virtual button is generated through the control screen. The patrol officer clicks on the virtual button to view the suspicious person's picture and related video clips of the suspicious person's suspicious behavior. At the same time, the control screen generates a confirmation button. The patrol officer ends the monitoring by clicking the confirmation button.

[0144] The integrated mobile digital human interactive collaborative system includes the following steps:

[0145] S1. Generate a virtual avatar using artificial intelligence and semantic recognition technology, aggregate patient inquiries based on the hospital's information database, review received movement instructions, generate a predicted route based on the hospital's internal information sharing and the hospital's floor plan for emergency information, and then move to the patrol personnel;

[0146] S2. Interact with patrol personnel through the control screen to ensure that the predicted emergency route is unobstructed. Analyze the gait and demeanor of people in the reception hall to classify the urgency of the crowd, proactively move to the most urgent patients, and provide proactive inquiries.

[0147] S3. Combined with the camera recognition image data, suspicious actions of people in the guidance hall are identified and marked. Combined with the internal camera images of the hospital, further judgment is made on suspicious persons. After the judgment is confirmed, the person is moved to the patrol personnel and interacts with the patrol personnel through the control screen to effectively respond and handle the situation.

[0148] In the embodiments provided herein, it should be understood that the disclosed devices, apparatuses, and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the module division is merely a logical functional division, and actual implementation may employ other division methods. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, i.e., they may be located in one location or distributed across multiple network elements. Some or all of the modules may be selected to achieve the objectives of the methods of the present embodiments, depending on actual needs.

[0149] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. An integrated mobile digital human interactive collaborative system, characterized by: It includes mobile platform module, central control module, recognition and perception module, information sharing and access module, crowd recognition module, and user interaction module; The mobile platform module includes a chassis system, which is used to receive relevant instructions and move by providing power through the chassis system; The information sharing access module accesses the hospital's internal information network through the Internet, receives relevant information from the hospital, and transmits it to the central control module and the crowd identification module; The central control module generates a virtual image based on artificial intelligence technology and semantic recognition technology, reviews the received movement instructions, and transmits the control instructions to the mobile platform module. Based on the received emergency information and the hospital floor plan, it generates a predicted route and transmits it to the mobile platform module and the user interaction module, including the following steps: Through artificial intelligence and semantic recognition technology, a virtual image is built to achieve user interaction and central control. Based on the surrounding environment information transmitted by the recognition and perception module, movement instructions are generated and transmitted to the mobile platform module for movement. Based on the emergency information in the information sharing access module, including the estimated time for the ambulance to return, the location where the ambulance returns, and the patient's relevant departments, a graph structure is established in combination with the current hospital floor plan. The Dijkstra algorithm is used to perform path planning to generate an emergency route, and the relevant information data is transmitted to the user interaction module to generate a countdown for the estimated time for the ambulance to return. At the same time, combined with the surrounding environment image of the recognition and perception module, a movement instruction is generated and transmitted to the mobile platform module, which automatically searches for patrol personnel in the guidance hall for user interaction. The recognition and perception module includes a camera, radar, and gyroscope, which are used to sense the surrounding environment, collect surrounding image information, determine position, direction, and distance information, transmit relevant environmental information to the central control module, and transmit image information to the crowd recognition module; The crowd recognition module identifies the crowd in the guidance hall by using the image data recognized by the camera of the recognition perception module, and classifies the medical urgency of the medical crowd based on their movements and expressions. According to the urgency of the medical crowd, the module transmits instructions to the central control module and actively moves to the emergency crowd to provide guidance services, including the following steps: By identifying the image data recognized by the camera of the recognition perception module, the cadence and facial expressions of the people in the guidance hall are identified. Based on the speed of the cadence and the emotional score of the facial expression, the urgency level of the medical personnel in the image data is divided. This includes judging the cadence v0 of the relevant personnel based on the low cadence threshold v1 and the high cadence threshold v2, and identifying and analyzing the facial expressions of the relevant personnel through the facial expression recognition algorithm, including anxiety, pain, and calmness, and further scoring the emotions; Among them, when identifying and analyzing the facial expressions of relevant personnel, the relevant facial expressions are first mapped, including anxiety, pain, and calmness, and then mapped to emotion categories. Emotions are judged by expression features such as raised eyebrows and drooping mouth corners. The facial action unit model FACS is used to define facial action units (AUs) by the movement intensity of facial muscles. The intensity values of multiple action units are combined to estimate the overall intensity of each emotion. Each emotion is composed of different action units, and the intensity of these action units contributes differently to the emotion. A weight is assigned to each emotion and adjusted according to the degree of emotional significance and the correlation between facial expressions, that is, the emotional intensity under different calm, painful, and anxious emotions is obtained: Among them, e i is the corresponding emotional intensity, i represents the type of emotion, including calm, pain, anxiety, Represents the weights of different action units under different types of emotions, Represents the values of different action units under different emotion types; Set the priority of high, medium, or low based on the emotional urgency and cadence urgency of the guided group, transmit the instructions to the central control module, and proactively move to the emergency group to provide guidance services based on the high, medium, or low priority; The crowd recognition module is also provided with a monitoring system. By recognizing the image information transmitted by the perception module, a recognition model is built based on a neural network model to identify and mark suspicious actions of the crowd in the guidance hall, and combined with the internal camera images of the hospital in the information sharing access module, further judgment is made on suspicious persons, clothing features are extracted for relevant suspicious persons-to-be-determined by the YOLO algorithm, the matching degree is judged by the Euclidean distance, and the suspicious persons-to-be-determined are marked in the internal camera image data. Secondary behavior recognition is performed on the suspicious persons-to-be-determined in the internal camera image data by the recognition model, and instructions are transmitted to the central control module according to the judgment results; The user interaction module includes a control screen, which is used to identify the patient's relevant problems and generate relevant information to interact with the patient through the control screen, including medical recommendations, payment information, and registration information. At the same time, based on the emergency data of the information sharing access module and the predictive route planning transmitted by the central control module, the control screen generates interactive buttons.

2. The integrated mobile digital human interactive collaborative system according to claim 1, characterized in that: The first aid information in the information sharing access module is combined with the current hospital floor plan to generate an first aid route, and the relevant information data is transmitted to the user interaction module. At the same time, combined with the surrounding environment image of the recognition and perception module, a movement instruction is generated, and the movement instruction is transmitted to the mobile platform module to automatically find the patrol personnel in the guidance hall for user interaction, including the following steps: Through the information sharing access module, emergency information is obtained, including the estimated time of return of the ambulance, the ambulance return location, and the patient's relevant department. The relevant emergency route is generated based on the current hospital floor plan. The specific steps are as follows: Obtain data related to the hospital floor plan, including the hospital's floor layout, the locations of various departments, elevator locations, and major intersections. Combined with the ambulance's return location, build a graph structure: Each department room is regarded as a department node, the elevator location and main intersection are marked as key location nodes, and the ambulance return location is marked as the starting node, where the department node, starting node and key location node represent the nodes of the graph structure; Set up corridor connections, which represent direct connections between different departments on a floor, and elevator connections, which represent connections between floors. Corridor connections and elevator connections are edges in the graph structure. Path planning is performed using the Dijkstra algorithm, and its algorithm formula is: For each node A in the graph, initialize the shortest path estimate d(A) from node A to the starting point Q: Among them, Q represents the starting point, and the distances to the remaining nodes are infinite. Insert the starting point Q into the priority queue with a priority of 0. Take the node B with the smallest current distance from the priority queue. For each node A adjacent to B, if the path to A through B is shorter than the currently known shortest path, update the distance of A: d(A)=min(d(A),d(B)+ω(B,A)); Where d(A) represents the currently known shortest distance from node A to the starting point, d(B) represents the currently known shortest distance from node B to the starting point, and ω(B, A) represents the actual distance from node B to node A. When node B taken out of the priority queue is the target node C, path planning is complete. At this time, d(C) is the shortest distance from the starting point Q to the target node C. The actual path is obtained by tracing back the predecessor node. The actual path information is transmitted into the user interaction module, and the countdown of the estimated return time of the ambulance is generated and entered into the user interaction module. Combined with the surrounding environment picture of the recognition and perception module, the movement instructions are generated and transmitted to the mobile platform module, which automatically searches for patrol personnel in the guidance hall for user interaction.

3. The integrated mobile digital human interactive collaborative system according to claim 1, characterized in that: The steps of identifying the cadence and facial expressions of the people in the consultation hall and classifying the urgency level of the medical crowd in the image data based on the cadence and the emotional score of the facial expressions are as follows: Combined with real-time image data, the motion characteristics of relevant personnel are extracted and the step frequency v0 is calculated. The algorithm formula is: Where v0 represents the cadence, p represents the number of steps per unit time, and Δt represents the unit time. Based on the low cadence threshold v1 and the high cadence threshold v2, the cadence v0 is determined as follows: When v0 < v1, it means that the relevant personnel have a slow cadence, indicating that the cadence urgency is low; When v1≤v0<v2, it means that the cadence of the relevant personnel is medium, indicating that the cadence urgency is medium; When v0 ≥ v2, it means that the relevant personnel have a rapid cadence, indicating a high cadence urgency; The facial expression recognition algorithm is used to identify and analyze the facial expressions of relevant personnel, including anxiety, pain, and calmness, and further score the emotions. The algorithm formula is: Among them, O0 represents the current emotion score, ω i Represents the weight of each emotion, e i Indicates the corresponding emotional intensity. The emotional score is the weighted value of the emotional intensity of the facial expression of the relevant person in unit time. Based on the low threshold O1 and the high threshold O2 of the emotional score, the emotions of the relevant people are graded: When O0 < O1, it means that the emotions of the relevant personnel are stable, indicating that the emotional urgency is low; When O1≤O0<O2, it means that the relevant personnel are in a moderate mood, indicating a medium level of emotional urgency; When O0 ≥ O2, it means that the relevant personnel are nervous, indicating a high level of emotional urgency.

4. The integrated mobile digital human interactive collaborative system according to claim 3, characterized in that: According to the emotional urgency and cadence urgency of the guided group, the instructions are transmitted to the central control module, and the module actively moves to the emergency group to provide guidance services. The specific steps are as follows: According to the emotional urgency and cadence urgency of the guided group, set the priority to high, medium, or low. The specific steps are as follows: High priority: high cadence urgency + high emotional urgency, high cadence urgency + medium emotional urgency, high cadence urgency + low emotional urgency, medium cadence urgency + high emotional urgency, low cadence urgency + high emotional urgency; Medium priority: medium cadence urgency + medium emotional urgency, medium cadence urgency + low emotional urgency, low cadence urgency + medium emotional urgency; Low priority: low cadence urgency + low emotional urgency; According to the priority of the guided group, the instructions are transmitted to the central control module, and based on the priority of high, medium or low, the module actively moves to the emergency group to provide guidance services.

5. The integrated mobile digital human interactive collaborative system according to claim 1, characterized in that: The crowd recognition module is also equipped with a monitoring system. By identifying and marking suspicious movements of people in the reception hall through the image information transmitted by the recognition and perception module, the system further determines the suspicious persons by combining the images of the internal cameras of the hospital in the information sharing and access module. The system transmits instructions to the central control module based on the determination results. The specific steps are as follows: The recognition and perception module obtains video clips of people's behavior inside the screen, and builds a recognition model based on the neural network model. The steps are as follows: Collect video clips containing various suspicious and normal behaviors, divide the collected video clips into training sets, validation sets, and test sets, and select appropriate features for feature extraction based on the characteristics of suspicious behaviors, including hand movement trajectories and changes in people's postures; Select a convolutional neural network (CNN) as the neural network model. Train the convolutional neural network model using the training set. Verify the trained recognition model using the validation set to adjust the model's parameters or structure. Test the recognition model using the test set to evaluate its performance in identifying suspicious behavior. The recognition model is used to identify the real-time video images transmitted by the recognition perception module. The identified suspicious persons are marked as suspicious persons - pending. The YOLO algorithm is used to extract clothing features of the relevant suspicious persons - pending and obtain a timestamp. Through the information sharing access module, the internal camera footage of the hospital is synchronized with the clothing feature data of the suspicious person based on the timestamp. The clothing features of the suspicious person are compared with the person in the internal camera footage through image processing technology. The match degree is determined by Euclidean distance, and the suspicious person is marked as "pending" in the internal camera footage data. The recognition model is used to perform secondary behavior recognition on suspicious persons in the internal camera image data. When the recognition result is confirmed, the relevant person is marked as a suspicious person, and the mobile signal is transmitted to the central control module and moved to the patrol personnel. At the same time, the picture of the suspicious person is captured and transmitted to the user interaction module.

6. The integrated mobile digital human interactive collaborative system according to claim 5, characterized in that: The user interaction module includes a control screen for identifying relevant patient questions and generating relevant information for interaction with the patient through the control screen, including medical recommendations, payment information, and registration information. At the same time, based on the emergency data of the information sharing access module and the predicted route planning transmitted by the central control module, the control screen generates interactive buttons, including the following steps: It contains a hospital information database, which includes a basic introduction to the hospital and information about departments. The virtual image generated by the central control module identifies patients' relevant questions and answers them based on the relevant data in the hospital information database, including medical recommendations, payment information, and registration information. At the same time, virtual buttons are generated on the control screen to assist patients in interaction, including navigation routes, online appointments, and fee information inquiries. At the same time, patients interact with the system through touch operations on the control screen to inquire about medical recommendations, check payment information, and register; For emergency medical information, after it reaches the patrol officer, a virtual button for route planning is generated on the control screen. The patrol officer clicks the virtual button to obtain the actual route information and generates the estimated time for the ambulance to return on the control screen. Proactively provide guidance services. When moving to an emergency group, generate voice announcements and inquire with the relevant emergency group. After a suspicious person is marked, when he moves to the patrol officer, a red virtual button is generated through the control screen. The patrol officer clicks on the virtual button to view the suspicious person's picture and related video clips of the suspicious person's suspicious behavior. At the same time, the control screen generates a confirmation button. The patrol officer ends the monitoring by clicking the confirmation button.

7. The integrated mobile digital human interactive collaborative system according to claim 1, characterized in that: The mobile platform module includes a chassis system, including an electric motor, drive wheels and batteries, which are used to realize the movement of the mobile platform and move the platform by providing power through the chassis based on receiving relevant instructions; The information sharing access module accesses the hospital's internal information network via the Internet and receives various data from the hospital in real time, including camera image data, hospital floor plans, and emergency information. It transmits the real-time camera image data to the crowd recognition module and transmits the hospital floor plans and emergency information to the central control module. The recognition and perception module includes a camera, a radar, and a gyroscope, which are used to perceive the surrounding environment, determine the position, direction, and distance information, obtain the surrounding picture data information through the camera, transmit the relevant environmental information into the central control module for auxiliary movement, transmit the surrounding picture data information into the crowd recognition module for auxiliary judgment, and formulate a movement plan into the central control module.

8. An integrated mobile digital human interactive collaboration method, characterized in that: The method adopts the integrated movable digital human interactive collaborative system according to any one of claims 1 to 7, comprising the following steps: S1. Generate a virtual avatar using artificial intelligence and semantic recognition technology, aggregate patient inquiries based on the hospital's information database, review received movement instructions, generate a predicted route based on the hospital's internal information sharing and the hospital's floor plan for emergency information, and then move to the patrol personnel; S2. Interact with patrol personnel through the control screen to ensure that the predicted emergency route is unobstructed. Analyze the gait and demeanor of people in the reception hall to classify the urgency of the crowd, proactively move to the most urgent patients, and provide proactive inquiries. S3. Combined with the camera recognition image data, suspicious actions of people in the guidance hall are identified and marked. Combined with the internal camera images of the hospital, further judgment is made on suspicious persons. After the judgment is confirmed, the person is moved to the patrol personnel and interacts with the patrol personnel through the control screen to effectively respond and handle the situation.

Images