Integrated mobile digital human interaction cooperation system and method thereof

Abstract

The invention discloses an integrated movable digital human interaction cooperation system and a method thereof, and belongs to the technical field of man-machine interaction, the integrated movable digital human interaction cooperation system comprises a mobile platform module, a central control module, an identification perception module, an information sharing access module, a crowd identification module and a user interaction module, the chassis system provides power to move, and the information sharing access module is accessed to an internal information network of a hospital through the Internet, receives internal related information of the hospital and transmits the information to the central control module and the crowd recognition module. According to the integrated movable digital human interaction coordination system and the method thereof, the functions of user interaction, hospital guide consultation, active patient identification, suspicious person marking, mobile platform integration, intelligent identification, information sharing and user interaction can be realized, the emergency response efficiency and the patient experience can be improved, and the practicability and the functionality are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-computer interaction, and specifically relates to an integrated movable digital human interactive collaborative system and method thereof. Background Art

[0002] With the popularization of artificial intelligence technology, robots equipped with human-computer interaction systems are being used in more and more places, including hotels, banks, hospitals, etc., to provide guidance and services to consumers or patients who come.

[0003] Most of the existing hospital human-computer interaction collaborative systems are fixed, and there are many obvious deficiencies. First, since patients need to actively use the system to provide services, in a crowded and complex environment such as the guidance hall, the system cannot actively provide guidance services to patients in urgent need of help, which greatly reduces the practicality of the system. In addition, this type of system cannot actively contact patrol personnel based on the hospital's emergency information to plan and clear routes. In an emergency, such as a large number of emergency patients pouring in at the same time, the system cannot effectively coordinate resources and plan the best route, which may cause channel congestion, untimely rescue and other problems. In addition, the system lacks the function of actively contacting patrol personnel, making it impossible to quickly organize manpower for effective response and processing when an emergency occurs, resulting in low functionality.

[0004] In response to the above, this case proposes an integrated movable digital human interactive collaborative system and method to solve the above technical problems. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes an integrated mobile digital human interactive collaborative system and method thereof, which solves the above technical problems by improving the detection method and the processing method.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] An integrated mobile digital human interactive collaborative system and method thereof, comprising a mobile platform module, a central control module, an identification and perception module, an information sharing and access module, a crowd identification module, and a user interaction module;

[0008] The mobile platform module includes a chassis system, which is used to receive relevant instructions and provide power for movement through the chassis system;

[0009] The information sharing access module accesses the internal information network of the hospital through the Internet, receives relevant information within the hospital, and transmits it to the central control module and the crowd identification module;

[0010] The central control module generates a virtual image based on artificial intelligence technology and semantic recognition technology, reviews the received movement instructions, and transmits the control instructions to the mobile platform module. Based on the received emergency information and the hospital floor plan, it generates a predicted route and transmits it to the mobile platform module and the user interaction module;

[0011] The recognition and perception module includes a camera, a radar, and a gyroscope, which are used to sense the surrounding environment, collect surrounding image information, determine the position, direction, and distance information, transmit the relevant environmental information to the central control module, and transmit the image information to the crowd recognition module;

[0012] The crowd recognition module recognizes the crowd in the guidance hall by identifying the image data recognized by the camera of the perception module, and divides the medical urgency of the medical crowd based on their movements and expressions. According to the urgency of the medical crowd, the module transmits instructions to the central control module and actively moves to the emergency crowd to provide guidance services.

[0013] The user interaction module includes a control screen, which is used to identify relevant issues of patients and generate relevant information to interact with patients through the control screen, including medical recommendations, payment information, and registration information. At the same time, based on the emergency data of the information sharing access module and the predictive route planning transmitted by the central control module, an interactive button is generated through the control screen.

[0014] Furthermore, the central control module generates a virtual image based on artificial intelligence technology and semantic recognition technology, reviews the received movement instructions, and transmits the control instructions to the mobile platform module. Based on the received emergency information and the hospital floor plan, a predicted route is generated and transmitted to the mobile platform module and the user interaction module, including the following steps:

[0015] Through artificial intelligence and semantic recognition technology, a virtual image is built to achieve user interaction and central control. Based on the surrounding environment information of the recognition and perception module, a movement instruction is generated and transmitted to the mobile platform module for movement according to the movement plan transmitted by the recognition and perception module;

[0016] Based on the emergency information in the information sharing access module and combined with the current hospital floor plan, an emergency route is generated, and the relevant information data is transmitted to the user interaction module. At the same time, combined with the surrounding environment picture of the recognition and perception module, movement instructions are generated and transmitted to the mobile platform module, which automatically searches for patrol personnel in the guidance hall for user interaction.

[0017] Furthermore, the emergency information in the information sharing access module is combined with the current hospital floor plan to generate an emergency route, and the relevant information data is transmitted to the user interaction module. At the same time, combined with the surrounding environment picture of the recognition perception module, a movement instruction is generated, and the movement instruction is transmitted to the mobile platform module, and the patrol personnel in the guidance hall are automatically found for user interaction, including the following steps:

[0018] Through the information sharing access module, emergency information is obtained, including the estimated time of return of the ambulance, the location of the ambulance return, and the patient's relevant department. The relevant emergency routes are generated according to the current hospital floor plan. The specific steps are as follows:

[0019] Get the hospital floor plan data, including the hospital floor layout, the location of each department, the location of the elevator, and the main intersections, and build a graph structure based on the ambulance return location:

[0020] Each department room is regarded as a department node, the elevator location and the main intersection are marked as key location nodes, and the ambulance return location is marked as the starting node, where the department node, the starting node and the key location node represent the nodes of the graph structure;

[0021] Set up corridor connections, which represent direct connections between different departments on a floor, and elevator connections, which represent connections between floors. The corridor connections and elevator connections are the edges of the Graph structure.

[0022] Path planning is performed using the Dijkstra algorithm, and the algorithm formula is:

[0023] For each node A in the graph, initialize the shortest path estimate d(A) from node A to the starting point Q:

[0024]

[0025] Among them, Q represents the starting point, and the distances of the other nodes are infinite. Insert the starting point Q into the priority queue with a priority of 0. Take out the node B with the smallest current distance from the priority queue. For each node A adjacent to B, if the path to A through B is shorter than the currently known shortest path, update the distance of A:

[0026] d(A)=min(d(A),d(B)+ω(B,A));

[0027] Among them, d(A) represents the currently known shortest distance from node A to the starting point, d(B) represents the currently known shortest distance from node B to the starting point, and ω(B,A) represents the actual distance from node B to node A. When node B taken out of the priority queue is the target node C, the path planning is completed. At this time, d(C) is the shortest distance from the starting point Q to the target node C. The actual path is obtained by tracing back the predecessor node;

[0028] The actual path information is transmitted into the user interaction module, and the countdown of the estimated return time of the ambulance is generated and entered into the user interaction module. Combined with the surrounding environment picture of the recognition perception module, the movement command is generated and transmitted to the mobile platform module, which automatically searches for patrol personnel in the guidance hall for user interaction.

[0029] Furthermore, the crowd recognition module recognizes the crowd in the guidance hall by identifying the image data recognized by the camera of the perception module, divides the medical urgency of the medical crowd based on their movements and expressions, transmits instructions to the central control module according to the urgency of the medical crowd, and actively moves to the emergency crowd to provide guidance services, including the following steps:

[0030] By identifying the image data recognized by the camera of the perception module, the cadence and facial expressions of the people in the reception hall are identified. Based on the speed of the cadence and the emotional score of the facial expression, the urgency level of the medical crowd in the image data is divided;

[0031] According to the emotional urgency and cadence urgency of the guided group, the instructions are transmitted to the central control module, and the module actively moves to the emergency group to provide guidance services.

[0032] Furthermore, the step frequency and facial expressions of the people in the reception hall are identified, and based on the speed of the step frequency and the emotional score of the facial expression, the emergency level of the medical crowd in the screen data is divided. The specific steps are:

[0033] Combined with real-time image data, the motion features of relevant personnel are extracted, and the step frequency v0 is calculated. The algorithm formula is:

[0034]

[0035] Among them, v0 represents the step frequency, p represents the number of steps of the relevant personnel per unit time, Δt represents the unit time, and the step frequency v0 is judged based on the low step frequency threshold v1 and the high step frequency threshold v2:

[0036] When v0<v1, it means that the relevant personnel have a slow step frequency, indicating that the step frequency urgency is low;

[0037] When v1≤v0<v2, it means that the step frequency of the relevant personnel is medium, indicating that the step frequency urgency is medium;

[0038] When v0 ≥ v2, it means that the relevant personnel have a rapid step frequency, indicating that the step frequency urgency is high;

[0039] The facial expression recognition algorithm is used to identify and analyze the facial expressions of relevant personnel, including anxiety, pain, and calmness, and further score the emotions. The algorithm formula is:

[0040]

[0041] Among them, O0 represents the current emotion score, ω i represents the weight of each emotion, e i Indicates the corresponding emotional intensity. The emotional score is the weighted value of the emotional intensity of the facial expression of the relevant personnel per unit time. Based on the low threshold O1 and the high threshold O2 of the emotional score, the emotions of the relevant personnel are graded:

[0042] When O0<O1, it means that the emotions of the relevant personnel are stable, indicating that the emotional urgency is low;

[0043] When O1≤O0<O2, it means that the relevant personnel are in a moderate mood, indicating a medium level of emotional urgency;

[0044] When O0 ≥ O2, it means that the relevant personnel are nervous, indicating a high level of emotional urgency.

[0045] Furthermore, according to the emotional urgency and cadence urgency of the guided group, the instructions are transmitted to the central control module, and the guide service is provided to the emergency group. The specific steps are as follows:

[0046] According to the emotional urgency and cadence urgency of the guided group, set the priority to high, medium, and low. The specific steps are as follows:

[0047] High priority: high cadence urgency + high emotional urgency, high cadence urgency + medium emotional urgency, high cadence urgency + low emotional urgency, medium cadence urgency + high emotional urgency, low cadence urgency + high emotional urgency;

[0048] Medium priority: medium cadence urgency + medium emotional urgency, medium cadence urgency + low emotional urgency, low cadence urgency + medium emotional urgency;

[0049] Low priority: low cadence urgency + low emotional urgency;

[0050] According to the priority of the guided group, the instructions are transmitted to the central control module, and according to the priority of high, medium or low, it actively moves to the emergency group to provide guidance services.

[0051] Furthermore, a monitoring system is also provided in the crowd recognition module, which recognizes and marks suspicious actions of people in the reception hall by identifying the image information transmitted by the perception module, and further determines suspicious persons by combining the images of the internal cameras of the hospital in the information sharing access module, and transmits instructions to the central control module according to the determination results. The specific steps are as follows:

[0052] The recognition perception module is used to obtain the video clips of the people’s behavior inside the screen, and the recognition model is built based on the neural network model. The steps are as follows:

[0053] Collect video clips containing various suspicious behaviors and normal behaviors, and divide the collected video clips into training sets, verification sets, and test sets. According to the characteristics of suspicious behaviors, select appropriate features for feature extraction, including hand movement trajectories and changes in people's postures;

[0054] Select the convolutional neural network (CNN) as the neural network model, train the convolutional neural network model with the training set, and verify the trained recognition model with the verification set to adjust the parameters or structure of the model. Test the recognition model with the test set to evaluate the recognition performance of suspicious behaviors.

[0055] Through the recognition model, the real-time video images transmitted by the recognition perception module are recognized, and the identified suspicious persons are marked as suspicious persons-pending. The clothing features of the relevant suspicious persons-pending are extracted through the YOLO algorithm, and the timestamp is obtained;

[0056] Through the information sharing access module, the hospital's internal camera screen data is synchronized with the suspicious person's clothing feature data based on the timestamp. The suspicious person's clothing features are compared with the person in the internal camera screen through image processing technology. The matching degree is determined by Euclidean distance, and the suspicious person is marked as "pending" in the internal camera screen data;

[0057] The recognition model is used to perform secondary behavior recognition on suspicious persons in the internal camera image data. When the recognition result is confirmed, the relevant person is marked as a suspicious person, and the mobile signal is transmitted to the central control module and moved to the patrol personnel. At the same time, the picture of the suspicious person is captured and transmitted to the user interaction module.

[0058] Furthermore, the user interaction module includes a control screen for identifying relevant problems of the patient, and generating relevant information through the control screen to interact with the patient, including medical recommendation, payment information, and registration information. At the same time, based on the emergency data of the information sharing access module and the predicted route planning transmitted by the central control module, an interactive button is generated through the control screen, including the following steps:

[0059] It contains a hospital information database, which includes a basic introduction to the hospital and department information. The virtual image generated by the central control module identifies the patient's relevant questions and answers the patient's questions based on the relevant data in the hospital information database, including medical recommendation, payment information, and registration information. At the same time, virtual buttons are generated through the control screen to assist patients in interaction, including navigation routes, online appointments, and fee information inquiries. At the same time, patients interact with the system through touch operations on the control screen to inquire about medical recommendations, check payment information, and register;

[0060] For emergency emergency information, after it moves to the patrol personnel, a virtual button for route planning is generated through the control screen. The patrol personnel obtains the actual path information by clicking the virtual button and generates the estimated time for the ambulance to return through the control screen.

[0061] For proactive guidance services, when the system moves to the emergency group, it generates voice announcements and inquires the relevant emergency group;

[0062] After marking a suspicious person, when it moves to the patrol personnel, a red virtual button is generated through the control screen. The patrol personnel clicks on the virtual button to obtain the picture of the suspicious person and the video clip of the suspicious behavior of the suspicious person. At the same time, the control screen generates a confirmation button. The patrol personnel clicks the confirmation button to end the monitoring.

[0063] Further, the mobile platform module includes a chassis system, including an electric motor, a drive wheel and a battery, which is used to realize the movement of the mobile platform, and based on receiving relevant instructions, the chassis provides power for movement;

[0064] The information sharing access module accesses the internal information network of the hospital through the Internet, receives various data from the hospital in real time, including camera image data, hospital floor plan, and emergency information, transmits the real-time camera image data to the crowd recognition module, and transmits the hospital floor plan and emergency information to the central control module;

[0065] The recognition and perception module includes a camera, a radar, and a gyroscope, which are used to perceive the surrounding environment, determine the position, direction, and distance information, obtain the surrounding picture data information through the camera, transmit the relevant environmental information into the central control module for auxiliary movement, transmit the surrounding picture data information into the crowd recognition module for auxiliary judgment, and formulate a movement plan into the central control module.

[0066] The integrated mobile digital human interactive collaborative system includes the following steps:

[0067] S1. Generate a virtual image through artificial intelligence technology and semantic recognition technology, summarize the patient's inquiries in combination with the hospital information database, review the received movement instructions, share information within the hospital, generate a predicted route for sudden emergency information in combination with the hospital floor plan, and move to the patrol personnel;

[0068] S2. Interact with patrol personnel through the control screen to ensure that the emergency prediction route is unobstructed. Analyze the walking frequency and demeanor of the crowd in the reception hall, classify the urgency of the crowd, actively move to the patients with high urgency, and provide active inquiries;

[0069] S3. Combined with the camera recognition image data, suspicious actions of people in the reception hall are identified and marked, and combined with the internal camera images of the hospital, suspicious persons are further judged. After confirmation, they are moved to the patrol personnel and interacted with the patrol personnel through the control screen to effectively respond and handle the situation.

[0070] Compared with the prior art, the present invention has the following beneficial effects:

[0071] 1. In the present invention, by setting up a user interaction module, the inquiries of patients in the guidance hall can be identified, and in combination with the hospital information database, the relevant questions of patients in the guidance hall can be answered through the control screen to provide help for the patients in the guidance hall;

[0072] 2. In the present invention, by accessing the internal information network of the hospital, when an emergency situation occurs, an emergency route is generated by combining the emergency patient information with the hospital floor plan, and a movement instruction is generated to actively move to the patrol personnel, and the control screen of the user interaction module is used to interact with the patrol personnel to ensure the smooth flow of the emergency route, optimize the dispatch of patrol personnel and emergency equipment, and improve the overall rescue efficiency;

[0073] 3. In the present invention, by analyzing the camera screen, analyzing the walking frequency and facial expressions of the crowd in the guidance hall, dividing the patients in the screen into emergency levels, actively moving to high-priority patients, making voice inquiries through the user interaction module, actively providing help, real-time analysis and response to the situation in the guidance hall, dynamically adjusting the service strategy according to the patient's emergency level, and enhancing functionality;

[0074] 4. In the present invention, by analyzing the suspicious behaviors of people inside the guidance hall and combining them with the internal camera data for secondary judgment, it is easy to find suspicious people inside the guidance hall. At the same time, patrol personnel are actively contacted for suspicious situations to effectively identify possible security threats, improve the safety of the hospital, avoid the occurrence of adverse events, and reduce the negative impact of safety accidents.

[0075] The entire integrated mobile digital human interactive collaborative system and its method can realize user interaction, medical guidance and consultation, active patient identification, and suspicious person marking. It integrates mobile platform, intelligent identification, information sharing and user interaction functions, can improve emergency response efficiency and patient experience, and enhance practicality and functionality. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a block diagram of the integrated mobile digital human interactive collaborative system and method of the present invention. DETAILED DESCRIPTION

[0077] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0078] Embodiment 1:

[0079] like Figure 1 As shown, the integrated movable digital human interactive collaborative system and method thereof include a mobile platform module, a central control module, an identification and perception module, an information sharing and access module, a crowd identification module, and a user interaction module;

[0080] The mobile platform module includes a chassis system, including an electric motor, a drive wheel and a battery, which is used to realize the movement of the mobile platform, and the movement is carried out by providing power through the chassis based on receiving relevant instructions;

[0081] The information sharing access module accesses the hospital's internal information network through the Internet, receives various data from the hospital in real time, including camera image data, hospital floor plan, and emergency information, transmits real-time camera image data to the crowd recognition module, and transmits hospital floor plan and emergency information to the central control module;

[0082] The recognition and perception module includes a camera, a radar, and a gyroscope, which are used to perceive the surrounding environment, determine the position, direction, and distance information, obtain the surrounding image data information through the camera, transmit the relevant environmental information to the central control module for auxiliary movement, transmit the surrounding image data information to the crowd recognition module for auxiliary judgment, and formulate a movement plan to enter the central control module;

[0083] The central control module generates a virtual image based on artificial intelligence technology and semantic recognition technology, reviews the received mobile instructions, and transmits the control instructions to the mobile platform module. Based on the received emergency information and the hospital floor plan, it generates a predicted route and transmits it to the mobile platform module and the user interaction module, including the following steps:

[0084] Through artificial intelligence and semantic recognition technology, a virtual image is built to achieve user interaction and central control. Based on the surrounding environment information of the recognition and perception module, a movement instruction is generated and transmitted to the mobile platform module for movement according to the movement plan transmitted by the recognition and perception module;

[0085] Based on the emergency information in the information sharing access module, combined with the current hospital floor plan, an emergency route is generated, and the relevant information data is transmitted to the user interaction module. At the same time, combined with the surrounding environment picture of the recognition perception module, a movement instruction is generated, and the movement instruction is transmitted to the mobile platform module, and the patrol personnel in the guidance hall are automatically found for user interaction, including the following steps:

[0086] Through the information sharing access module, emergency information is obtained, including the estimated time of return of the ambulance, the location of the ambulance return, and the patient's relevant department. The relevant emergency routes are generated according to the current hospital floor plan. The specific steps are as follows:

[0087] Get the hospital floor plan data, including the hospital floor layout, the location of each department, the location of the elevator, and the main intersections, and build a graph structure based on the ambulance return location:

[0088] Each department room is regarded as a department node, the elevator location and the main intersection are marked as key location nodes, and the ambulance return location is marked as the starting node, where the department node, the starting node and the key location node represent the nodes of the graph structure;

[0089] Set up corridor connections, which represent direct connections between different departments on a floor, and elevator connections, which represent connections between floors. The corridor connections and elevator connections are the edges of the graph structure.

[0090] Path planning is performed using the Dijkstra algorithm, and the algorithm formula is:

[0091] For each node A in the graph, initialize the shortest path estimate d(A) from node A to the starting point Q:

[0092]

[0093] Among them, Q represents the starting point, and the distances of the other nodes are infinite. Insert the starting point Q into the priority queue with a priority of 0. Take out the node B with the smallest current distance from the priority queue. For each node A adjacent to B, if the path to A through B is shorter than the currently known shortest path, update the distance of A:

[0094] d(A)=min(d(A),d(B)+ω(B,A));

[0095] Among them, d(A) represents the currently known shortest distance from node A to the starting point, d(B) represents the currently known shortest distance from node B to the starting point, and ω(B,A) represents the actual distance from node B to node A. When node B taken out of the priority queue is the target node C, the path planning is completed. At this time, d(C) is the shortest distance from the starting point Q to the target node C. The actual path is obtained by tracing back the predecessor node;

[0096] The actual path information is transmitted into the user interaction module, and the countdown of the estimated return time of the ambulance is generated and entered into the user interaction module. Combined with the surrounding environment picture of the recognition perception module, the movement command is generated and transmitted to the mobile platform module, which automatically searches for patrol personnel in the guidance hall for user interaction.

[0097] It should be noted that d(B)+ω(B,A) represents the length of the path from node B to node A. If the distance of node A is updated, the priority of A, that is, the new distance value, needs to be updated in the priority queue. d(A) represents the shortest path length from the starting point to node A up to the current moment, and d(B) represents the shortest path length from the starting point to node B up to the current moment. If the distance from node B to node A is shorter, that is, d(B)+ω(B,A) is less than the known d(A), then the distance of node A is updated to d(B)+ω(B,A).

[0098] Embodiment 2:

[0099] The crowd recognition module recognizes the crowd in the guidance hall by using the image data recognized by the camera of the recognition perception module, and divides the medical urgency of the medical crowd based on their movements and expressions. According to the urgency of the medical crowd, the module transmits instructions to the central control module and actively moves to the emergency crowd to provide guidance services, including the following steps:

[0100] By identifying the image data recognized by the camera of the perception module, the cadence and facial expressions of the people in the reception hall are identified. Based on the speed of the cadence and the emotional score of the facial expression, the urgency level of the medical crowd in the image data is divided. The specific steps are as follows:

[0101] Combined with real-time image data, the motion features of relevant personnel are extracted, and the step frequency v0 is calculated. The algorithm formula is:

[0102]

[0103] Among them, v0 represents the step frequency, p represents the number of steps of the relevant personnel per unit time, Δt represents the unit time, and the step frequency v0 is judged based on the low step frequency threshold v1 and the high step frequency threshold v2:

[0104] When v0<v1, it means that the relevant personnel have a slow step frequency, indicating that the step frequency urgency is low;

[0105] When v1≤v0<v2, it means that the step frequency of the relevant personnel is medium, indicating that the step frequency urgency is medium;

[0106] When v0 ≥ v2, it means that the relevant personnel have a rapid step frequency, indicating that the step frequency urgency is high;

[0107] The facial expression recognition algorithm is used to identify and analyze the facial expressions of relevant personnel, including anxiety, pain, and calmness, and further score the emotions. The algorithm formula is:

[0108]

[0109] Among them, O0 represents the current emotion score, ω i represents the weight of each emotion, e i Indicates the corresponding emotional intensity. The emotional score is the weighted value of the emotional intensity of the facial expression of the relevant personnel per unit time. Based on the low threshold O1 and the high threshold O2 of the emotional score, the emotions of the relevant personnel are graded:

[0110] When O0<O1, it means that the emotions of the relevant personnel are stable, indicating that the emotional urgency is low;

[0111] When O1≤O0<O2, it means that the relevant personnel are in a moderate mood, indicating a medium level of emotional urgency;

[0112] When O0 ≥ O2, it means that the relevant personnel are nervous, indicating a high level of emotional urgency.

[0113] It should be noted that when identifying and analyzing the facial expressions of relevant personnel, the relevant facial expressions are first mapped, including anxiety, pain, and calmness, and the facial expressions are mapped to emotion categories. For example, emotions are judged by expression features such as raised eyebrows and drooping corners of the mouth. The facial action unit model FACS is used to define facial action units AUs by the movement intensity of facial muscles. The intensity values ​​of multiple action units are combined to estimate the overall intensity of each emotion. Each emotion may be composed of different action units, and the intensity of these action units contributes differently to the emotion. A weight is assigned to each emotion, and it is adjusted according to the degree of significance of the emotion and the relevance of the facial expression, that is, the emotional intensity under different calm, painful, and anxious emotions is obtained:

[0114]

[0115] Among them, e i is the corresponding emotional intensity, i represents the type of emotion, including calm, pain, anxiety, Represents the weights of different action units under different types of emotions, Represents the values ​​of different action units under different emotion types.

[0116] According to the emotional urgency and cadence urgency of the guided crowd, the command is transmitted to the central control module, and the guide service is provided to the emergency crowd. The specific steps are as follows:

[0117] According to the emotional urgency and cadence urgency of the guided group, set the priority to high, medium, and low. The specific steps are as follows:

[0118] High priority: high cadence urgency + high emotional urgency, high cadence urgency + medium emotional urgency, high cadence urgency + low emotional urgency, medium cadence urgency + high emotional urgency, low cadence urgency + high emotional urgency;

[0119] Medium priority: medium cadence urgency + medium emotional urgency, medium cadence urgency + low emotional urgency, low cadence urgency + medium emotional urgency;

[0120] Low priority: low cadence urgency + low emotional urgency;

[0121] According to the priority of the guided group, the instructions are transmitted to the central control module, and according to the priority of high, medium or low, it actively moves to the emergency group to provide guidance services.

[0122] It should be noted that the priority of the target person is determined based on the high, medium and low priority of the guided group, combined with the priority determination table:

[0123] Table 1 Priority determination table

[0124]

[0125] Embodiment 4:

[0126] The crowd recognition module is also equipped with a monitoring system. By identifying the image information transmitted by the perception module, the suspicious actions of the crowd in the reception hall are identified and marked. In combination with the internal camera images of the hospital in the information sharing access module, the suspicious persons are further judged. According to the judgment results, the instructions are transmitted to the central control module. The specific steps are as follows:

[0127] The recognition perception module is used to obtain the video clips of the people’s behavior inside the screen, and the recognition model is built based on the neural network model. The steps are as follows:

[0128] Collect video clips containing various suspicious behaviors and normal behaviors, and divide the collected video clips into training sets, verification sets, and test sets. According to the characteristics of suspicious behaviors, select appropriate features for feature extraction, including hand movement trajectories and changes in people's postures;

[0129] Select the convolutional neural network (CNN) as the neural network model, train the convolutional neural network model with the training set, and verify the trained recognition model with the verification set to adjust the parameters or structure of the model. Test the recognition model with the test set to evaluate the recognition performance of suspicious behaviors.

[0130] Through the recognition model, the real-time video images transmitted by the recognition perception module are recognized, and the identified suspicious persons are marked as suspicious persons-pending. The clothing features of the relevant suspicious persons-pending are extracted through the YOLO algorithm, and the timestamp is obtained;

[0131] Through the information sharing access module, the hospital's internal camera screen data is synchronized with the suspicious person's clothing feature data based on the timestamp. The suspicious person's clothing features are compared with the person in the internal camera screen through image processing technology. The matching degree is determined by Euclidean distance, and the suspicious person is marked as "pending" in the internal camera screen data;

[0132] The recognition model is used to perform secondary behavior recognition on suspicious persons in the internal camera image data. When the recognition result is confirmed, the relevant person is marked as a suspicious person, and the mobile signal is transmitted to the central control module and moved to the patrol personnel. At the same time, the picture of the suspicious person is captured and transmitted to the user interaction module.

[0133] Embodiment 5:

[0134] The user interaction module includes a control screen, which is used to identify relevant problems of patients and generate relevant information to interact with patients through the control screen, including medical recommendations, payment information, and registration information. At the same time, based on the emergency data of the information sharing access module and the predicted route planning transmitted by the central control module, the control screen generates interactive buttons, and the steps are as follows:

[0135] It contains a hospital information database, which includes a basic introduction to the hospital and department information. The virtual image generated by the central control module identifies the patient's relevant questions and answers the patient's questions based on the relevant data in the hospital information database, including medical recommendation, payment information, and registration information. At the same time, virtual buttons are generated through the control screen to assist patients in interaction, including navigation routes, online appointments, and fee information inquiries. At the same time, patients interact with the system through touch operations on the control screen to inquire about medical recommendations, check payment information, and register;

[0136] For emergency emergency information, after it moves to the patrol personnel, a virtual button for route planning is generated through the control screen. The patrol personnel obtains the actual path information by clicking the virtual button and generates the estimated time for the ambulance to return through the control screen.

[0137] It should be noted that in the actual path information, a confirmation button is generated for each key node, including elevators and related paths. After the patrol personnel control an elevator or related path, they need to click the confirmation button to make the relevant position in the actual path information appear green, indicating that the relevant position is unblocked.

[0138] For proactive guidance services, when the system moves to the emergency group, it generates voice announcements and inquires the relevant emergency group;

[0139] It should be noted that the preset inquiry voices include: "Hello, I noticed that you may need help. May I ask if you or someone around you has encountered an emergency medical situation";

[0140] “Please tell me exactly what happened and how you are feeling now”;

[0141] “Are you experiencing any pain, discomfort, or other specific symptoms”;

[0142] "If you need to go to the emergency room, I can direct you there".

[0143] After marking a suspicious person, when it moves to the patrol personnel, a red virtual button is generated through the control screen. The patrol personnel clicks on the virtual button to obtain the picture of the suspicious person and the video clip of the suspicious behavior of the suspicious person. At the same time, the control screen generates a confirmation button. The patrol personnel clicks the confirmation button to end the monitoring.

[0144] The integrated mobile digital human interactive collaborative system includes the following steps:

[0145] S1. Generate a virtual image through artificial intelligence technology and semantic recognition technology, summarize the patient's inquiries in combination with the hospital information database, review the received movement instructions, share information within the hospital, generate a predicted route for sudden emergency information in combination with the hospital floor plan, and move to the patrol personnel;

[0146] S2. Interact with patrol personnel through the control screen to ensure that the emergency prediction route is unobstructed. Analyze the walking frequency and demeanor of the crowd in the reception hall, classify the urgency of the crowd, actively move to the patients with high urgency, and provide active inquiries;

[0147] S3. Combined with the camera recognition image data, suspicious actions of people in the reception hall are identified and marked, and combined with the internal camera images of the hospital, suspicious persons are further judged. After confirmation, they are moved to the patrol personnel and interacted with the patrol personnel through the control screen to effectively respond and handle the situation.

[0148] In the embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules is only a logical function division, and there may be other division methods in actual implementation; the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the method of this embodiment.

[0149] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. Integrated mobile digital human interactive collaborative system, characterized by: It includes mobile platform module, central control module, recognition and perception module, information sharing and access module, crowd recognition module and user interaction module; The mobile platform module includes a chassis system, which is used to receive relevant instructions and provide power for movement through the chassis system; The information sharing access module accesses the internal information network of the hospital through the Internet, receives relevant information within the hospital, and transmits it to the central control module and the crowd identification module; The central control module generates a virtual image based on artificial intelligence technology and semantic recognition technology, reviews the received movement instructions, and transmits the control instructions to the mobile platform module. Based on the received emergency information and the hospital floor plan, it generates a predicted route and transmits it to the mobile platform module and the user interaction module; The recognition and perception module includes a camera, a radar, and a gyroscope, which are used to sense the surrounding environment, collect surrounding image information, determine the position, direction, and distance information, transmit the relevant environmental information to the central control module, and transmit the image information to the crowd recognition module; The crowd recognition module recognizes the crowd in the guidance hall by identifying the image data recognized by the camera of the perception module, and divides the medical urgency of the medical crowd based on their movements and expressions. According to the urgency of the medical crowd, the module transmits instructions to the central control module and actively moves to the emergency crowd to provide guidance services. The user interaction module includes a control screen, which is used to identify relevant issues of patients and generate relevant information to interact with patients through the control screen, including medical recommendations, payment information, and registration information. At the same time, based on the emergency data of the information sharing access module and the predictive route planning transmitted by the central control module, an interactive button is generated through the control screen.

2. The integrated mobile digital human interactive collaborative system according to claim 1, characterized in that: The central control module generates a virtual image based on artificial intelligence technology and semantic recognition technology, reviews the received movement instructions, and transmits the control instructions to the mobile platform module. Based on the received emergency information and the hospital floor plan, a predicted route is generated and transmitted to the mobile platform module and the user interaction module, including the following steps: Through artificial intelligence and semantic recognition technology, a virtual image is built to achieve user interaction and central control. Based on the surrounding environment information of the recognition and perception module, a movement instruction is generated and transmitted to the mobile platform module for movement according to the movement plan transmitted by the recognition and perception module; Based on the emergency information in the information sharing access module and combined with the current hospital floor plan, an emergency route is generated, and the relevant information data is transmitted to the user interaction module. At the same time, combined with the surrounding environment picture of the recognition and perception module, movement instructions are generated and transmitted to the mobile platform module, which automatically searches for patrol personnel in the guidance hall for user interaction.

3. The integrated mobile digital human interactive collaborative system according to claim 2, characterized in that: The first aid information in the information sharing access module is combined with the current hospital floor plan to generate a first aid route, and the relevant information data is transmitted to the user interaction module. At the same time, combined with the surrounding environment picture of the recognition perception module, a movement instruction is generated, and the movement instruction is transmitted to the mobile platform module, and the patrol personnel in the guidance hall are automatically found for user interaction, including the following steps: Through the information sharing access module, emergency information is obtained, including the estimated time of return of the ambulance, the location of the ambulance return, and the patient's relevant department. The relevant emergency routes are generated according to the current hospital floor plan. The specific steps are as follows: Get the data related to the hospital floor plan, including the floor layout of the hospital, the location of each department, the location of the elevator, and the main intersections, and build a graph structure based on the ambulance return location: Each department room is regarded as a department node, the elevator location and the main intersection are marked as key location nodes, and the ambulance return location is marked as the starting node, where the department node, the starting node and the key location node represent the nodes of the graph structure; Set up corridor connections, which represent direct connections between different departments on a floor, and elevator connections, which represent connections between floors. The corridor connections and elevator connections are the edges of the Graph structure. The path planning is performed through the Dijkstra algorithm, and its algorithm formula is: For each node A in the graph, initialize the shortest path estimate d(A) from node A to the starting point Q: Among them, Q represents the starting point, and the distances of the other nodes are infinite. Insert the starting point Q into the priority queue with a priority of 0. Take out the node B with the smallest current distance from the priority queue. For each node A adjacent to B, if the path to A through B is shorter than the currently known shortest path, update the distance of A: d(A)=min(d(A),d(B)+ω(B,A)); Among them, d(A) represents the currently known shortest distance from node A to the starting point, d(B) represents the currently known shortest distance from node B to the starting point, and ω(B,A) represents the actual distance from node B to node A. When node B taken out of the priority queue is the target node C, the path planning is completed. At this time, d(C) is the shortest distance from the starting point Q to the target node C. The actual path is obtained by tracing back the predecessor node; The actual path information is transmitted into the user interaction module, and the countdown of the estimated return time of the ambulance is generated and entered into the user interaction module. Combined with the surrounding environment picture of the recognition perception module, the movement command is generated and transmitted to the mobile platform module, which automatically searches for patrol personnel in the guidance hall for user interaction.

4. The integrated mobile digital human interactive collaborative system according to claim 1, characterized in that: The crowd recognition module recognizes the crowd in the guidance hall by identifying the image data recognized by the camera of the perception module, divides the medical urgency of the medical crowd based on their movements and expressions, transmits instructions to the central control module according to the urgency of the medical crowd, and actively moves to the emergency crowd to provide guidance services, including the following steps: By identifying the image data recognized by the camera of the perception module, the cadence and facial expressions of the people in the reception hall are identified. Based on the speed of the cadence and the emotional score of the facial expression, the urgency level of the medical crowd in the image data is divided; According to the emotional urgency and cadence urgency of the guided group, the instructions are transmitted to the central control module, and the module actively moves to the emergency group to provide guidance services.

5. The integrated mobile digital human interactive collaborative system according to claim 4, characterized in that: The step frequency and facial expressions of the people in the reception hall are identified, and based on the speed of the step frequency and the emotional score of the facial expression, the emergency level of the medical crowd in the screen data is divided. The specific steps are: Combined with real-time image data, the motion features of relevant personnel are extracted, and the step frequency v0 is calculated. The algorithm formula is: Among them, v0 represents the step frequency, p represents the number of steps of the relevant personnel per unit time, Δt represents the unit time, and the step frequency v0 is judged based on the low step frequency threshold v1 and the high step frequency threshold v2: When v0<v1, it means that the relevant personnel have a slow step frequency, indicating that the step frequency urgency is low; When v1≤v0<v2, it means that the step frequency of the relevant personnel is medium, indicating that the step frequency urgency is medium; When v0 ≥ v2, it means that the relevant personnel have a rapid step frequency, indicating that the step frequency urgency is high; The facial expression recognition algorithm is used to identify and analyze the facial expressions of relevant personnel, including anxiety, pain, and calmness, and further score the emotions. The algorithm formula is: Among them, O0 represents the current emotion score, ω i represents the weight of each emotion, e i Indicates the corresponding emotional intensity. The emotional score is the weighted value of the emotional intensity of the facial expression of the relevant personnel per unit time. Based on the low threshold O1 and the high threshold O2 of the emotional score, the emotions of the relevant personnel are graded: When O0<O1, it means that the emotions of the relevant personnel are stable, indicating that the emotional urgency is low; When O1≤O0<O2, it means that the relevant personnel are in a moderate mood, indicating a medium level of emotional urgency; When O0 ≥ O2, it means that the relevant personnel are nervous, indicating a high level of emotional urgency.

6. The integrated mobile digital human interactive collaborative system according to claim 4, characterized in that: According to the emotional urgency and cadence urgency of the guided group, the instructions are transmitted to the central control module, and the module actively moves to the emergency group to provide the guided service. The specific steps are as follows: According to the emotional urgency and cadence urgency of the guided group, set the priority to high, medium, and low. The specific steps are as follows: High priority: high cadence urgency + high emotional urgency, high cadence urgency + medium emotional urgency, high cadence urgency + low emotional urgency, medium cadence urgency + high emotional urgency, low cadence urgency + high emotional urgency; Medium priority: medium cadence urgency + medium emotional urgency, medium cadence urgency + low emotional urgency, low cadence urgency + medium emotional urgency; Low priority: low cadence urgency + low emotional urgency; According to the priority of the guided group, the instructions are transmitted to the central control module, and according to the priority of high, medium or low, it actively moves to the emergency group to provide guidance services.

7. The integrated mobile digital human interactive collaborative system according to claim 1, characterized in that: The crowd recognition module is also provided with a monitoring system, which recognizes and marks suspicious actions of people in the reception hall by identifying the image information transmitted by the perception module, and further determines suspicious persons by combining the images of the internal cameras of the hospital in the information sharing access module, and transmits instructions to the central control module according to the determination results. The specific steps are as follows: The recognition perception module is used to obtain the video clips of the people’s behavior inside the screen, and the recognition model is built based on the neural network model. The steps are as follows: Collect video clips containing various suspicious behaviors and normal behaviors, and divide the collected video clips into training sets, verification sets, and test sets. According to the characteristics of suspicious behaviors, select appropriate features for feature extraction, including hand movement trajectories and changes in people's postures; Select the convolutional neural network (CNN) as the neural network model, train the convolutional neural network model with the training set, and verify the trained recognition model with the verification set to adjust the parameters or structure of the model. Test the recognition model with the test set to evaluate the recognition performance of suspicious behaviors. Through the recognition model, the real-time video images transmitted by the recognition perception module are recognized, and the identified suspicious persons are marked as suspicious persons-pending. The clothing features of the relevant suspicious persons-pending are extracted through the YOLO algorithm, and the timestamp is obtained; Through the information sharing access module, the hospital's internal camera screen data is synchronized with the suspicious person's clothing feature data based on the timestamp. The suspicious person's clothing features are compared with the person in the internal camera screen through image processing technology. The matching degree is determined by Euclidean distance, and the suspicious person is marked as "pending" in the internal camera screen data; The recognition model is used to perform secondary behavior recognition on suspicious persons in the internal camera image data. When the recognition result is confirmed, the relevant person is marked as a suspicious person, and the mobile signal is transmitted to the central control module and moved to the patrol personnel. At the same time, the picture of the suspicious person is captured and transmitted to the user interaction module.

8. The integrated mobile digital human interactive collaborative system according to claim 7, characterized in that: The user interaction module includes a control screen, which is used to identify relevant problems of patients, and to interact with patients through the control screen by generating relevant information, including medical recommendation, payment information, and registration information. At the same time, based on the emergency data of the information sharing access module and the predicted route planning transmitted by the central control module, the control screen generates an interactive button, including the following steps: It contains a hospital information database, which includes a basic introduction to the hospital and department information. The virtual image generated by the central control module identifies the patient's relevant questions and answers the patient's questions based on the relevant data in the hospital information database, including medical recommendation, payment information, and registration information. At the same time, virtual buttons are generated through the control screen to assist patients in interaction, including navigation routes, online appointments, and fee information inquiries. At the same time, patients interact with the system through touch operations on the control screen to inquire about medical recommendations, check payment information, and register; For emergency emergency information, after it moves to the patrol personnel, a virtual button for route planning is generated through the control screen. The patrol personnel obtains the actual path information by clicking the virtual button and generates the estimated time for the ambulance to return through the control screen. For proactive guidance services, when the system moves to the emergency group, it generates voice broadcasts to inquire relevant emergency groups; After marking a suspicious person, when it moves to the patrol personnel, a red virtual button is generated through the control screen. The patrol personnel clicks on the virtual button to obtain the picture of the suspicious person and the video clip of the suspicious behavior of the suspicious person. At the same time, the control screen generates a confirmation button. The patrol personnel clicks on the confirmation button to end the monitoring.

9. The integrated mobile digital human interactive collaborative system according to claim 1, characterized in that: The mobile platform module includes a chassis system, including an electric motor, a drive wheel and a battery, which is used to realize the movement of the mobile platform, and the movement is performed by providing power through the chassis based on receiving relevant instructions; The information sharing access module accesses the internal information network of the hospital through the Internet, receives various data from the hospital in real time, including camera image data, hospital floor plan, and emergency information, transmits the real-time camera image data to the crowd recognition module, and transmits the hospital floor plan and emergency information to the central control module; The recognition and perception module includes a camera, a radar, and a gyroscope, which are used to perceive the surrounding environment, determine the position, direction, and distance information, obtain the surrounding picture data information through the camera, transmit the relevant environmental information into the central control module for auxiliary movement, transmit the surrounding picture data information into the crowd recognition module for auxiliary judgment, and formulate a movement plan into the central control module.

10. An integrated movable digital human interactive collaboration method, characterized in that: The method adopts the integrated movable digital human interactive collaborative system as described in any one of claims 1 to 9, comprising the following steps: S1. Generate a virtual image through artificial intelligence technology and semantic recognition technology, summarize the patient's inquiries in combination with the hospital information database, review the received movement instructions, share information within the hospital, generate a predicted route for sudden emergency information in combination with the hospital floor plan, and move to the patrol personnel; S2. Interact with patrol personnel through the control screen to ensure that the emergency prediction route is unobstructed. Analyze the walking frequency and demeanor of the crowd in the reception hall, classify the urgency of the crowd, actively move to the patients with high urgency, and provide active inquiries; S3. Combined with the camera recognition image data, suspicious actions of people in the reception hall are identified and marked, and combined with the internal camera images of the hospital, suspicious persons are further judged. After confirmation, they are moved to the patrol personnel and interacted with the patrol personnel through the control screen to effectively respond and handle the situation.

Images