An intelligent risk prediction method and system based on multimodal perception

Abstract

This invention discloses an intelligent risk prediction method and system based on multimodal perception, relating to the field of health risk prediction. The method includes: collecting images of the waiting area environment to construct a 3D map and extracting information to form a set of waiting personnel; using the built-in sensors of a carebot to collect facial expressions and voice information, performing emotion analysis to identify emotional states; calculating stress and risk indices, and sending information on personnel exceeding the risk index threshold to the nurses' station; combining emotional states to calculate health risk indicators and assess potential health risks; based on the health risk assessment, providing health education to low- and medium-risk personnel through the carebot, prioritizing reassurance for high-risk personnel and sending information alerts to the nurses' station. By accurately identifying high-risk individuals through multimodal perception technology, the method optimizes waiting area management, improves health early warning and intervention capabilities, reduces patient anxiety, and enhances patient safety and satisfaction.

Description

Technical Field

[0001] This invention relates to the field of health risk prediction, specifically to an intelligent risk prediction method and system based on multimodal perception. Background Technology

[0002] Faced with the increasing demand for nursing care, the traditional nursing model that relies on manual intervention is no longer able to provide sufficient service quality, and there is an urgent need to find new ways to solve this challenge.

[0003] The development of artificial intelligence technology offers a potential solution to this challenge. By combining affective computing with patient state perception technology, intelligent nursing robots can monitor patients' physiological and emotional states in real time, alleviating the stress on caregivers and providing psychological support through emotional interaction. Furthermore, health risk prediction technology based on big data and machine learning can identify potential health risks in patients and provide timely warnings. This intelligent, multi-dimensional nursing solution is particularly suitable for high-load medical scenarios such as waiting areas, helping to improve the efficiency of medical services and patient experience, and paving a new path for the development of smart healthcare.

[0004] The development of intelligent nursing robots is based on technological advancements in areas such as affective computing, patient state perception, and health risk prediction. In recent years, affective computing technologies, such as facial expression recognition and voice emotion analysis, have matured, enabling nursing robots to recognize patient emotions and provide personalized care. The combination of sensors and AI algorithms makes real-time monitoring of patient physiological states possible, providing precise data support for health management. Furthermore, machine learning-based intelligent prediction algorithms have shown promising applications in medical risk assessment, effectively predicting risks such as postoperative complications and chronic disease flare-ups, thus assisting medical decision-making. Intelligent robots can alleviate the stress on nursing staff, optimize service efficiency, and align with the trend of automation and intelligence in the healthcare industry. Combining market demand and technological innovation, the CareBot project has significant social and academic value. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide an intelligent risk prediction method and system based on multimodal perception to solve the above-mentioned technical problems.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an intelligent risk prediction method based on multimodal perception, comprising:

[0007] S1. Collect environmental images of the waiting area using the built-in sensors of carebot to construct a 3D map of the waiting area, extract information from the 3D map, construct a set of waiting personnel, and calculate the abnormality of body temperature and spatial abnormality indicators of the waiting personnel.

[0008] S2. Based on the facial expressions and voice information of waiting patients collected by the built-in sensors of carebot, emotion analysis is performed to identify the emotional state of waiting patients;

[0009] S3. Calculate the stress index and risk index based on the information set of waiting patients, and send the information of waiting patients with risk indices greater than the threshold to the nurse station for early warning;

[0010] S4. Calculate health risk indicators based on the emotional state of waiting patients and assess their potential health risks;

[0011] S5. Based on the assessment of potential health risks of waiting patients, health education is provided to those with medium and low risk through carebot, while those with high risk are given priority for reassurance and their information is sent to the nurses' station for early warning.

[0012] The present invention is further configured such that S1 specifically includes:

[0013] Carebot uses its built-in sensors to collect environmental images of the waiting area to build a 3D map of the waiting area. Data is then extracted from the 3D model to build a set of information about the waiting people, including: person ID, person location, person posture, waiting time, and real-time body temperature.

[0014] The degree of abnormality in the real-time body temperature of waiting patients is quantified to obtain the body temperature abnormality level. The pressure of the environment in which the waiting patients are located is quantified by combining the two dimensions of space crowding and waiting time to obtain the space abnormality index.

[0015] The present invention is further configured as follows: a group of people waiting for treatment: Where H represents the group of people waiting for treatment, N represents the total number of people in the waiting area, and h i The personnel information set contains: h i ={ID i ,x i ,y i Z i ,t i ,T i}, ID i For personnel ID, x i ,y i For personnel location, Z i For personnel posture, t i For waiting time, T i Real-time body temperature;

[0016] Logic for calculating abnormal body temperature: Among them, A i T represents the degree of body temperature abnormality, λ represents the sensitivity to control abnormality, and T represents the temperature abnormality. i Take the temperature of people waiting for treatment;

[0017] Calculation logic for spatial anomaly indicators: Where, ρ i γ is the spatial anomaly index, τ is the spatial decay coefficient, and t is the time smoothing parameter. i For waiting time, (x) i ,y i Let (x) be the coordinates of the location of person i in the waiting area. j ,y j Let ) represent the coordinates of the waiting person j, and ||·||2 represent the Euclidean norm.

[0018] The present invention is further configured such that S2 specifically includes:

[0019] The Carebot uses its built-in camera to collect facial images and voice information of people waiting for treatment. It uses facial expression recognition technology to obtain facial expression features and voice emotion recognition technology to obtain voice features. The acquired facial expression features and voice features are then fused in a multimodal manner to obtain joint features.

[0020] Modal feature transformation is performed on the joint features to obtain the fused hidden layer features;

[0021] The emotional state of waiting patients is obtained by calculating the classification probability based on the hidden layer features.

[0022] The present invention is further configured as follows: Modal feature transformation calculation logic: Where, m i Let g be the hidden layer feature, ⊙ be the gated probability vector, ⊙ be the successive multiplication operator, ReLU be the activation function, and W be the hidden layer feature. f For the face-dominated transformation matrix, z i For joint features, W v The transformation matrix is ​​speech-driven;

[0023] Classification probability calculation logic: Where s∈{anxiety, tension, calm}, where... For the classification probability, W p For the classification weight matrix, b p For classification bias, Softmax is the activation function;

[0024] Logic for determining emotional state: Among them, e i The emotional state of people waiting for treatment.

[0025] The present invention is further configured such that S3 specifically includes:

[0026] High-risk markers are made based on personnel information sets. If a person is lying down and their emotional state is anxious, a high-risk marker is made directly without calculation, immediately triggering an alarm at the nurses' station.

[0027] The stress index and risk index are calculated for waiting patients based on their real-time body temperature.

[0028] Information on patients waiting for treatment whose risk index exceeds the threshold will be sent to the nurses' station for early warning.

[0029] The present invention is further configured with the following pressure index calculation logic: Among them, P i As a stress index, A i For abnormal body temperature, ρ i This is a spatial anomaly indicator, with β1 representing the anxiety weight. For the anxiety state term, β2 represents the tension weight. For the stress term, θ T The set body temperature range;

[0030] Risk index calculation logic: Among them, R i Here, η is the risk index, η is the stress index weighting coefficient, and ω2 is the sitting posture weight. For sitting posture, ω1 represents the weight of lying posture. For personnel in a lying position;

[0031] If R i >θ R Send an alert to the nurses' station, where θ R The threshold for the set high-risk index.

[0032] The present invention is further configured such that S4 specifically includes:

[0033] Health risk indicators were calculated by combining the abnormal body temperature and emotional anxiety risk of waiting patients;

[0034] Health risk indicator calculation logic: Among them, Risk i As a health risk indicator, β represents the weight of emotional anxiety. This refers to the subjective state item.

[0035] The present invention is further configured such that S5 specifically includes: health risk determination logic: Among them, r1 and r2 are set thresholds. For high-risk waiting patients, the information of the waiting patients will be sent to the nurse station for early warning and the carebot will be dispatched first to comfort them and play health education videos. For medium and low-risk waiting patients, health education videos or texts will be played through the carebot.

[0036] The present invention also provides an intelligent risk prediction system based on multimodal perception, the system comprising:

[0037] Information collection module: The system collects environmental images of the waiting area through Carebot's built-in sensors to construct a 3D map of the waiting area, extracts information from the 3D map, constructs a set of waiting personnel, and calculates the abnormality of body temperature and spatial abnormality indicators of the waiting personnel.

[0038] Emotional state recognition module: Based on the facial expressions and voice information of waiting patients collected by the built-in sensors of carebot, emotion analysis is performed to identify the emotional state of waiting patients;

[0039] Risk calculation module: Calculates stress index and risk index based on the information set of waiting patients, and sends the information of waiting patients with risk indices exceeding the threshold to the nurse station for early warning;

[0040] Health risk assessment module: Combines the emotional state of waiting patients to calculate health risk indicators and assess the potential health risks of waiting patients;

[0041] Feedback module: Based on the assessment of potential health risks of waiting patients, health education is provided to those with medium and low risk through Carebot, while those with high risk are given priority for reassurance and their information is sent to the nurses' station for early warning.

[0042] This invention provides an intelligent risk prediction method and system based on multimodal perception. The method constructs a 3D map of the waiting area by collecting environmental images from the waiting area using the built-in sensors of a carebot. Information is extracted from the 3D map to construct a set of waiting personnel and calculate their abnormal body temperature and spatial anomaly indicators. Facial expressions and voice information of waiting personnel are collected by the carebot's built-in sensors for emotion analysis to identify their emotional states. Stress and risk indices are calculated based on the waiting personnel information set, and information on waiting personnel with risk indices exceeding a threshold is sent to the nurses' station for early warning. Health risk indicators are calculated based on the emotional states of waiting personnel to assess their potential health risks. Based on the assessed potential health risks, health education is provided to low- and medium-risk waiting personnel through the carebot, while high-risk waiting personnel are prioritized for reassurance, and their information is sent to the nurses' station for early warning. The beneficial effects include:

[0043] Improving the efficiency of personnel management in waiting areas: This method uses multimodal sensing technology to comprehensively assess the stress and risk of waiting personnel, accurately identify high-risk individuals, optimize personnel management and flow scheduling in waiting areas, and reduce the occurrence of sudden health events.

[0044] Enhance health early warning and intervention capabilities: By combining the abnormal body temperature of waiting patients with their emotional anxiety index, the system can provide early warnings for high-risk individuals and automatically take intervention measures, such as sending alerts to the nurses' station or providing health education through carebot, thereby reducing the waste of medical resources and improving patients' sense of security and satisfaction.

[0045] Enhancing patient experience and satisfaction: The emotional state and health risks of waiting patients can be monitored and intervened in a timely manner, reducing patients' anxiety during the waiting period, enhancing patients' trust and satisfaction with hospital services, and thus improving the overall service quality of the hospital.

[0046] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0048] Figure 1 A flowchart illustrating an intelligent risk prediction method based on multimodal perception, as an exemplary embodiment of the present invention;

[0049] Figure 2 This is a schematic diagram illustrating the structure of an intelligent risk prediction system based on multimodal perception, as an exemplary embodiment of the present invention. Detailed Implementation

[0050] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred embodiments. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0051] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0052] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0053] Example 1

[0054] A smart risk prediction method based on multimodal perception, such as Figure 1 Shown, including:

[0055] S1. Collect environmental images of the waiting area using the built-in sensors of carebot to construct a 3D map of the waiting area, extract information from the 3D map, construct a set of waiting personnel, and calculate the abnormality of body temperature and spatial abnormality indicators of the waiting personnel.

[0056] S2. Based on the facial expressions and voice information of waiting patients collected by the built-in sensors of carebot, emotion analysis is performed to identify the emotional state of waiting patients;

[0057] S3. Calculate the stress index and risk index based on the information set of waiting patients, and send the information of waiting patients with risk indices greater than the threshold to the nurse station for early warning;

[0058] S4. Calculate health risk indicators based on the emotional state of waiting patients and assess their potential health risks;

[0059] S5. Based on the assessment of potential health risks of waiting patients, health education is provided to those with medium and low risk through carebot, while those with high risk are given priority for reassurance and their information is sent to the nurses' station for early warning.

[0060] The present invention is further configured such that S1 specifically includes:

[0061] Carebot uses its built-in sensors to collect environmental images of the waiting area to build a 3D map of the waiting area. Data is then extracted from the 3D model to build a set of information about the waiting people, including: person ID, person location, person posture, waiting time, and real-time body temperature.

[0062] The degree of abnormality in the real-time body temperature of waiting patients is quantified to obtain the body temperature abnormality level. The stress level of the environment in which the waiting patients are located is quantified by combining the spatial crowding level and waiting time to obtain the spatial abnormality index. Specifically, Carebot's built-in sensors include a camera and an infrared sensor. The camera is used to collect images of the waiting area in real time, and the infrared sensor is used to collect the body temperature information of the people in the waiting area. The collected information is uploaded to the cloud for processing to construct a 3D map of the environment. Object detection and tracking are performed on the images of the waiting area to obtain the location information, posture, and waiting time of the waiting patients. This part is existing technology and will not be elaborated further. Emotional fluctuations can affect body temperature, and the body temperature abnormality level is used to assess the degree of deviation of the waiting patient's body temperature during the waiting period. Long waiting times or excessively high crowd density can affect emotions, and the spatial abnormality index is used to assess the current patient's waiting time and crowd density. The spatial crowding level specifically refers to the density of people around the current waiting patient.

[0063] The present invention is further configured as follows: a group of people waiting for treatment: Where H represents the group of people waiting for treatment, N represents the total number of people in the waiting area, and h i The personnel information set contains: h i ={ID i ,x i ,y i Z i ,t i ,T i}, ID i For personnel ID, x i ,y i For personnel location, Z i For personnel posture, t i For waiting time, T i Real-time body temperature;

[0064] Logic for calculating abnormal body temperature: Among them, A i T represents the degree of body temperature abnormality, λ represents the sensitivity to control abnormality, and T represents the temperature abnormality. i Take the temperature of people waiting for treatment;

[0065] Calculation logic for spatial anomaly indicators: Where, ρ i γ is the spatial anomaly index, τ is the spatial decay coefficient, and t is the time smoothing parameter. i For waiting time, (x) i ,y i Let (x) be the coordinates of the location of person i in the waiting area. j ,y jLet be the coordinates of the waiting person j, and ||·||2 be the Euclidean norm. Specifically, the waiting person set H contains the information of all waiting persons within the area collected by Carebot during the current time period, including: a unique person ID used to distinguish different waiting persons; a person's location (the specific spatial position of the person in the environment); and three posture categories: standing, sitting, and lying down, with standing posture Z being the most common. i =3, sitting posture Z i =2, lying position Z i =1, waiting time is the duration of the person's wait in this environment, obtained through object detection and tracking, which is existing technology. Real-time body temperature is collected by the infrared sensor built into Carebot; in the calculation of abnormal body temperature, the abnormal body temperature A... i It is represented by two exponential functions: one is the exponential increment when body temperature is elevated. Another is the index increment when body temperature is low. The difference between these two exponential functions calculates the degree of temperature abnormality among waiting personnel. λ is used to control the sensitivity to abnormalities; adjusting λ affects the system's sensitivity to temperature abnormalities. A value typically between 0.1 and 1 can be adjusted according to actual needs. -2 ensures the temperature abnormality is zero or negative when the temperature is within the normal range. max(·,0) ensures the temperature abnormality is zero when the temperature is within the normal range. In the calculation of spatial abnormality indicators, the spatial attenuation coefficient γ determines the degree of influence of spatial distance on the abnormality indicator. The larger the value, the smaller the influence of people at greater distances on the spatial abnormality indicator. The value ranges from 0.1 to 1, adjusted according to the density of people in the waiting area, with a default value of 0.5. The time smoothing parameter τ controls the smoothing effect of waiting time on the spatial abnormality indicator, typically set to 30 minutes or 1 hour to smooth out time effects. The Euclidean norm ||(x i ,y i )-(x j ,y j )||2 represents calculating the spatial distance between two people waiting for their appointment. This represents the summation of the spatial pressure contributions of all other waiting individuals (j) within the waiting area, excluding waiting individual i. This represents the spatial pressure contribution of waiting person j to waiting person i. The closer the distance, the greater the influence, and the closer the value is to 1. tanh(·) is the hyperbolic tangent function used to compress the summation result to [0,1) to avoid extreme values ​​interfering with subsequent calculations. The value represents the psychological impact of waiting time; the longer the waiting time, the closer the value is to 1.

[0066] The present invention is further configured such that S2 specifically includes:

[0067] The Carebot uses its built-in camera to collect facial images and voice information of people waiting for treatment. It uses facial expression recognition technology to obtain facial expression features and voice emotion recognition technology to obtain voice features. The acquired facial expression features and voice features are then fused in a multimodal manner to obtain joint features.

[0068] Modal feature transformation is performed on the joint features to obtain the fused hidden layer features;

[0069] Based on the hidden layer features, classification probability calculations are performed to obtain the emotional state of the waiting patients. Specifically, facial image acquisition: using the Carebot's built-in camera, the system captures facial images of the waiting patients in real time. Facial images contain a lot of important emotional information, such as smiles, frowns, and eye expressions, which can reflect the emotional state of the waiting patients. Voice information acquisition: using the Carebot's built-in microphone, the system collects the voice data of the waiting patients. Factors such as pitch, speech rate, and speech intensity in the voice can reflect emotional fluctuations. For example, when people are anxious or nervous, they usually speak faster and with a higher pitch. The acquired facial expression features and voice features are fused using multimodal methods to obtain joint features. The existing technology will not be elaborated upon here; modality feature transformation is a further processing of joint features, that is, mapping the joint features to a high-dimensional hidden feature space, which can capture the complex relationships between different modalities; classification probability calculation inputs the hidden features obtained after feature transformation into a classification model, and the classifier calculates the probability value of each emotion label, finally returning the most likely emotion state and its corresponding probability; in general, it is mainly divided into three steps: feature extraction of facial expression and voice data: through the camera and microphone of carebot, facial expression and voice information of waiting people are collected. Deep learning algorithms are used to perform emotion analysis on facial expression and voice data to extract meaningful features.

[0070] Feature fusion and processing: Facial expression features and speech features are fused, and the influence of one modality is selectively enhanced through a gating mechanism. The fused features are then transformed to obtain a hidden feature representation, which contains the emotional state of the waiting person.

[0071] Emotional state prediction: Based on hidden layer features, the Softmax classification algorithm is used to calculate the probability of waiting patients in different emotional states, such as anxiety, tension, and calmness. Based on these probabilities, the emotional state of waiting patients is determined, thus providing a basis for subsequent health interventions.

[0072] The present invention is further configured as follows: Modal feature transformation calculation logic: Where, m iLet g be the hidden layer feature, ⊙ be the gated probability vector, ⊙ be the successive multiplication operator, ReLU be the activation function, and W be the hidden layer feature. f For the face-dominated transformation matrix, z i For joint features, W v The transformation matrix is ​​speech-driven;

[0073] Classification probability calculation logic: Where s∈{anxiety, tension, calm}, where... For the classification probability, W p For the classification weight matrix, b p For classification bias, Softmax is the activation function;

[0074] Logic for determining emotional state: Among them, e i The emotional state of people waiting for treatment. Specifically, hidden layer features m i The gating probability vector is a feature vector representing the emotional state of waiting patients, obtained by gating and activation functions using facial expression and speech features. This vector contains a comprehensive description of the waiting patients' emotions. The gating probability vector g controls the weights of facial expression and speech features in the final fused features. The gating mechanism allows the system to select more useful modal information for fusion based on the characteristics of the input data. The value of g ranges from [0,1], and its value determines the contribution ratio of facial and speech features in the fused features. If it approaches 1, facial features dominate the fusion; if it approaches 0, speech features dominate. The specific formula for calculating the gating probability vector is as follows: W g Let b be the gate control weight matrix. g For gate control bias term, W g z i +b g The linear transformation maps the input to three-dimensional space, σ(W) g z i +b g ) is the activation function used to compress each dimension between (0,1), and the face-dominant transformation matrix W f Primarily used for extracting features related to cousins, the speech-dominated transformation matrix W v Primarily used for extracting speech-related features, W f and W v These are all used to map input features to a new space; they are obtained through training and are learnable parameters of the neural network; classification probabilities This represents the probability of a waiting person grabbing the counter under different gun handles. This needs to be calculated three times: the probability of anxiety, the probability of tension, and the probability of calmness. The sum of these three probabilities should ultimately equal 1. The classification weight matrix W... pThe probability used to transform a hidden layer feature into an emotional state is learned through training, and the classification bias term b is used. p It is used to adjust the output value; based on the obtained three classification probabilities, the largest probability is taken as the judgment result, which is set as the emotional state of the waiting person.

[0075] The present invention is further configured such that S3 specifically includes:

[0076] High-risk markers are made based on personnel information sets. If a person is lying down and their emotional state is anxious, a high-risk marker is made directly without calculation, immediately triggering an alarm at the nurses' station.

[0077] The stress index and risk index are calculated for waiting patients based on their real-time body temperature.

[0078] Information on waiting patients with risk indices exceeding a threshold is sent to the nurses' station for alert. Specifically, the high-risk labeling logic is as follows: if a waiting patient is lying down and exhibits anxiety, the probability of physical discomfort is very high. In this case, other external factors can only serve as a basis for medical staff to assess the patient's condition. Therefore, this patient is directly identified as high-risk and requires immediate medical intervention to prevent accidents and ensure their health and safety. The stress index is calculated based on a set temperature classification. If the temperature is within the set range, the patient's emotional state (anxiety or tension), the size of their waiting area (crowdedness), and any abnormal spatial indicators (high values) are considered. If the patient's temperature is outside the set range, it indicates an abnormal temperature; individuals with temperatures exceeding or below normal have a higher probability of risk. For these patients, other factors are not considered. Only the degree of body temperature abnormality and the anxiety level of the waiting patient are considered. The risk index is also calculated according to body temperature classification. If the body temperature is within the preset range, the posture of the waiting patient is considered, whether they are sitting or lying down. Standing posture is not considered because if the waiting patient can stand up on their own, it proves that they still have the ability to move. If the body temperature of the waiting patient is not within the set range, it proves that the body temperature is abnormal, exceeding or falling below the normal body temperature. Such people have a higher probability of risk. At this time, only the waiting time of the waiting patient is considered without considering other factors. The longer the time, the higher the risk. Different waiting thresholds are set according to different waiting environments to adjust the sensitivity of the procedure. If the waiting environment is low-risk, such as dental or dermatology, the threshold can be appropriately increased. If the waiting environment is high-risk, such as respiratory or fever clinic, the threshold can be appropriately decreased.

[0079] The present invention is further configured with the following pressure index calculation logic: Among them, P i As a stress index, A i For abnormal body temperature, ρ iThis is a spatial anomaly indicator, with β1 representing the anxiety weight. For the anxiety state term, β2 represents the tension weight. For the stress term, θ T The set body temperature range;

[0080] Risk index calculation logic: Among them, R i Here, η is the risk index, η is the pressure index weighting coefficient, and ω1 is the lying posture weight. For the lying posture, ω2 represents the weight of the sitting posture. For seated positions;

[0081] If R i >θ R Send an alert to the nurses' station, where θ R This is a set high-risk index threshold. Specifically, the stress index P... i To quantify the immediate stress experienced by waiting patients within a set temperature threshold range due to abnormal body temperature and overcrowding, different stress levels lead to different emotional states. This emotional state is incorporated into a stress index; a higher value indicates a higher level of stress. If a patient's body temperature exceeds the range, environmental stress and feelings of calm or tension are disregarded; only the degree of temperature abnormality and anxiety are calculated. This stress index allows the system to assess whether waiting patients are in a high-stress state, providing a basis for subsequent risk index calculations. The anxiety weight β1 has a default value of 0.3 and can be modified according to different waiting environments, ranging from 0.2 to 0.5. The tension weight β2 has a lower weight than anxiety, with a default value of 0.1 and can be modified according to different waiting environments, ranging from 0.05 to 0.2. (The anxiety state item...) This is an anxiety indicator function, which restricts the emotional state to anxiety. It takes a value of 1 if the emotional state is anxious, and a value of 0 if it is not anxious. The tension term... Similar to the logic of the anxiety state item mentioned above, the set body temperature range θ T A normal body temperature is typically between 36 and 38 degrees Celsius; a high-risk index (R) indicates a risk level of 36-38 degrees Celsius. iThe detection is mainly based on two standards: normal body temperature and extreme body temperature. For normal body temperature, a high-risk index is calculated by combining the person's posture, waiting time, and stress level. The risk index is considered because if the person is uncomfortable, it is difficult for them to stand, and they will adopt a sitting or lying position to relieve their discomfort. For extreme body temperature, since the temperature is already high enough to trigger an alarm, other privacy considerations are disregarded, and the focus is on time. The longer the waiting time, the greater the risk of danger for the person. The weight of lying posture ω1 is 0.8, ranging from 0.5 to 1.0, and is adjusted according to different waiting environments. The weight of sitting posture ω2 is slightly lower than that of lying posture at 0.2, ranging from 0.1 to 0.3, and is also adjusted according to different waiting environments. This is a function indicating the posture of a person, restricting their posture to lying down. The value is 1 if the person is lying down, and 0 otherwise. (This is related to the "person sitting posture" function.) The logic is the same as the anxiety state item mentioned above; if the high-risk index R... i If the threshold is exceeded, the corresponding waiting patient data will be transmitted to the nurses' station to generate an alert, reminding nurses to pay close attention to that patient. The default threshold is θ. R The value is 5.0, which can be modified according to the needs of different departments. The suggested range is between 3.0 and 7.0. The specific value is related to the sensitivity of the overall program.

[0082] The present invention is further configured such that S4 specifically includes:

[0083] Health risk indicators were calculated by combining the abnormal body temperature and emotional anxiety risk of waiting patients;

[0084] Health risk indicator calculation logic: Among them, Risk i As a health risk indicator, β represents the weight of emotional anxiety. This is the subjective state item. Specifically, the health risk index assesses the health risk of waiting patients by integrating multiple factors such as body temperature and emotional anxiety. Especially in cases of anxiety and abnormal body temperature, it can promptly identify potentially high-risk individuals, facilitating rapid emergency intervention. The emotional anxiety weight β controls the contribution of emotional anxiety to health risk; anxiety increases health risk, and its value ranges from [0,1]. The larger the value, the greater the impact of anxiety on health risk. Subjective state item Indicates the emotional state of people waiting for treatment, e i =1 indicates that the person waiting for treatment is in an anxious state. The design logic is: when the emotional state is anxious, the subjective state item takes the value of 1; otherwise, when the emotional state is tense or calm, the subjective state item is 0. This item is used to indicate that the emotional state is restricted to anxiety.

[0085] The present invention is further configured such that S5 specifically includes: health risk determination logic: Here, r1 and r2 are the set thresholds. Specifically, based on the calculated health risk indicators, waiting patients are divided into three categories according to their health risk: low risk, medium risk, and high risk, according to the set thresholds r1 and r2. The set thresholds r1 and r2 are usually set by historical data and clinical experience, with the default settings being r1=1.0 and r2=2.0, which are adjusted according to different background environments. For high-risk waiting patients, their information needs to be sent to the nursing station to notify the nursing station to pay special attention to them. The data sent to the nursing station includes: a collection of waiting patient information, the waiting patient's stress index, the waiting patient's emotional state, the waiting patient's health risk indicators, and the waiting patient's health risk. For medium-risk and low-risk waiting patients, health science videos or texts are played through Carebot to convey health knowledge and alleviate waiting stress.

[0086] Example 2

[0087] Please see Figure 2 This exemplary intelligent risk prediction system based on multimodal perception includes:

[0088] Information collection module: The system collects environmental images of the waiting area through Carebot's built-in sensors to construct a 3D map of the waiting area, extracts information from the 3D map, constructs a set of waiting personnel, and calculates the abnormality of body temperature and spatial abnormality indicators of the waiting personnel.

[0089] Emotional state recognition module: Based on the facial expressions and voice information of waiting patients collected by the built-in sensors of carebot, emotion analysis is performed to identify the emotional state of waiting patients;

[0090] Risk calculation module: Calculates stress index and risk index based on the information set of waiting patients, and sends the information of waiting patients with risk indices exceeding the threshold to the nurse station for early warning;

[0091] Health risk assessment module: Combines the emotional state of waiting patients to calculate health risk indicators and assess the potential health risks of waiting patients;

[0092] Feedback module: Based on the assessment of potential health risks of waiting patients, health education is provided to those with medium and low risk through Carebot, while those with high risk are given priority for reassurance and their information is sent to the nurses' station for early warning.

[0093] It should be noted that the intelligent risk prediction system based on multimodal perception provided in the above embodiments and the intelligent risk prediction method based on multimodal perception provided in the above embodiments belong to the same concept. The specific ways in which each module and unit performs operations have been described in detail in the method embodiments, and will not be repeated here. In practical applications, the intelligent risk prediction system based on multimodal perception provided in the above embodiments can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above, and this is not a limitation here.

[0094] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0095] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. A and B can be singular or plural. Additionally, the character " / " in this article generally indicates an "or" relationship between the preceding and following related objects, but it can also represent an "and / or" relationship. Please refer to the context for a more accurate understanding.

[0096] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0097] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0098] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0103] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A method for intelligent risk prediction based on multimodal perception, characterized in that, include: S1. Collect environmental images of the waiting area using the built-in sensors of carebot to construct a 3D map of the waiting area, extract information from the 3D map, construct a set of waiting personnel, and calculate the abnormality of body temperature and spatial abnormality indicators of the waiting personnel. S2. Based on the facial expressions and voice information of waiting patients collected by the built-in sensors of carebot, emotion analysis is performed to identify the emotional state of waiting patients. Specifically, this includes: collecting facial images and voice information of waiting patients through the built-in camera of carebot; using facial expression recognition technology to obtain facial expression features; using voice emotion recognition technology to obtain voice features; performing multimodal fusion of the obtained facial expression features and voice features to obtain joint features; and performing modal feature transformation on the joint features to obtain the fused hidden layer features. Modal feature transformation calculation logic: ,in, Hidden layer features This is the gated probability vector. For the symbol of sequential multiplication, For activation function, The transformation matrix is ​​dominated by the face. For joint features, The transformation matrix is ​​speech-driven; classification probability is calculated based on hidden layer features to obtain the emotional state of the waiting patient; the classification probability calculation logic is as follows: ,in, For classification probability, This is the classification weight matrix. For classification bias terms, Activation function; Emotional state determination logic: ,in, For the emotional state of people waiting for treatment; S3. Calculate the stress index and risk index based on the waiting patient information set, and send the information of waiting patients with risk indices exceeding the threshold to the nurse station for early warning; specifically including: marking high-risk individuals based on the personnel information set; if a person is lying down and their emotional state is anxious, a high-risk label is automatically generated without calculation, immediately triggering an alarm at the nurse station; calculating the stress index and risk index for waiting patients based on their real-time body temperature classification; stress index calculation logic: ,in, As a stress index, For abnormal body temperature, As a spatial anomaly indicator, For anxiety weighting, For anxiety state items, For the sake of tension weight, For the stress state item, For the set body temperature range, Real-time body temperature; Risk index calculation logic: ,in, As a risk index, For lying position weight, For personnel in a lying position, For sitting posture weight, For seated positions; if Send an alert to the nurses' station, including A high-risk index threshold is set; information on waiting patients whose risk index exceeds the threshold is sent to the nurses' station for early warning. Waiting time; S4. Calculate health risk indicators based on the emotional state of waiting patients to assess their potential health risks; calculate health risk indicators by combining the patients' abnormal body temperature and emotional anxiety risk; the calculation logic for health risk indicators is as follows: ,in, As a health risk indicator, Weighting for emotional anxiety This refers to the subjective state item; S5. Based on the assessment of the potential health risks of waiting patients, health education is provided to those with medium and low risk through Carebot, while those with high risk are given priority for reassurance and their information is sent to the nurses' station for early warning.

2. The intelligent risk prediction method based on multimodal perception according to claim 1, characterized in that, S1 specifically includes: Carebot uses its built-in sensors to collect environmental images of the waiting area to build a 3D map of the waiting area. Data is then extracted from the 3D model to build a set of information about the waiting people, including: person ID, person location, person posture, waiting time, and real-time body temperature. The degree of abnormality in the real-time body temperature of waiting patients is quantified to obtain the body temperature abnormality level. The pressure of the environment in which the waiting patients are located is quantified by combining the two dimensions of space crowding and waiting time to obtain the space abnormality index.

3. The intelligent risk prediction method based on multimodal perception according to claim 2, characterized in that, Group of people waiting for medical treatment: ,in, For those waiting for treatment, The total number of people in the waiting area. This is a personnel information set, which includes: , For personnel ID, For personnel location, For personnel posture, For waiting time, Real-time body temperature; Logic for calculating abnormal body temperature: ,in, For abnormal body temperature, To control abnormal sensitivity, Take the temperature of people waiting for treatment; Calculation logic for spatial anomaly indicators: ,in, As a spatial anomaly indicator, The spatial attenuation coefficient, For time smoothing parameters, For waiting time, For waiting patients Location coordinates For waiting patients Location coordinates It is the Euclidean norm.

4. The intelligent risk prediction method based on multimodal perception according to claim 1, characterized in that, S5 specifically includes: Health Risk Assessment Logic: ,in, and Based on the set threshold, the information of high-risk waiting patients is sent to the nurse station for early warning, and the carebot is dispatched first to comfort them and play health education videos. For medium- and low-risk waiting patients, health education videos or texts are played through the carebot.

5. A multimodal perception-based intelligent risk prediction system, used to implement the multimodal perception-based intelligent risk prediction method according to any one of claims 1-4, characterized in that, include: Information collection module: The system collects environmental images of the waiting area through Carebot's built-in sensors to construct a 3D map of the waiting area, extracts information from the 3D map, constructs a set of waiting personnel, and calculates the abnormality of body temperature and spatial abnormality indicators of the waiting personnel. Emotional state recognition module: Based on the facial expressions and voice information of waiting patients collected by the built-in sensors of carebot, emotion analysis is performed to identify the emotional state of waiting patients; Risk calculation module: Calculates stress index and risk index based on the information set of waiting patients, and sends the information of waiting patients with risk indices exceeding the threshold to the nurse station for early warning; Health risk assessment module: Combines the emotional state of waiting patients to calculate health risk indicators and assess the potential health risks of waiting patients; Feedback module: Based on the assessment of potential health risks of waiting patients, health education is provided to those with medium and low risk through Carebot, while those with high risk are given priority for reassurance and their information is sent to the nurses' station for early warning.

Images