Intelligent risk prediction method and system based on multi-modal perception

Abstract

The invention discloses an intelligent risk prediction method and system based on multi-modal perception, and relates to the field of health risk prediction. The method comprises the following steps: acquiring an environment image of a waiting area to construct a three-dimensional map, and extracting information to construct a waiting personnel set; acquiring facial expressions and voice information by using a carebox built-in sensor, and performing emotion analysis to recognize an emotion state; calculating a pressure index and a risk index, and sending personnel information exceeding a risk index threshold to a nurse station; health risk indexes are calculated in combination with the emotional state, and potential health risks are evaluated; on the basis of health risk assessment, health science popularization is provided for middle and low-risk personnel through carebox, high-risk personnel are pacified preferentially, and information early warning is sent to a nurse station. High-risk individuals are accurately identified through a multi-mode perception technology, waiting area management is optimized, health early warning and intervention capabilities are improved, anxiety of patients is reduced, and the sense of security and satisfaction of the patients are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of health risk prediction, and specifically to an intelligent risk prediction method and system based on multimodal perception. Background Art

[0002] Faced with the growing demand for care, the traditional care model that relies on manual intervention can no longer provide sufficient service quality, and there is an urgent need to find new ways to address this challenge.

[0003] The development of artificial intelligence (AI) technology offers a potential solution to this challenge. By combining affective computing with patient status perception, intelligent nursing robots can monitor patients' physical and emotional states in real time, alleviating stress on caregivers while providing psychological support through emotional interaction. Furthermore, health risk prediction technology based on big data and machine learning can identify potential health risks and provide timely warnings. This intelligent, multi-dimensional nursing solution is particularly suitable for high-load medical scenarios such as waiting areas, helping to improve medical service efficiency and patient experience, and opening up new paths for the development of smart healthcare.

[0004] The development of intelligent nursing robots is based on technological advances in fields such as affective computing, patient status perception, and health risk prediction. In recent years, affective computing technologies such as facial expression recognition and voice emotion analysis have continued to mature, enabling nursing robots to identify patients' emotions and provide personalized care. The combination of sensors and AI algorithms enables real-time monitoring of patients' physiological status, providing precise data support for health management. Furthermore, intelligent predictive algorithms based on machine learning have shown promising application prospects in medical risk assessment, effectively predicting risks such as postoperative complications and the onset of chronic diseases, and assisting in medical decision-making. Intelligent robots can alleviate the stress of caregivers and optimize service efficiency, aligning with the trend of automation and intelligentization in the medical industry. Combining market demand with technological innovation, the research in the CareBot project has significant social value and academic significance. Summary of the Invention

[0005] Based on the above-mentioned shortcomings of the prior art, the purpose of the present invention is to provide an intelligent risk prediction method and system based on multimodal perception to solve the above-mentioned technical problems.

[0006] To achieve the above objectives, the present invention provides the following technical solution: an intelligent risk prediction method based on multimodal perception, comprising:

[0007] S1. The carebot's built-in sensors collect images of the waiting area to construct a three-dimensional map of the waiting area. Information is extracted from the three-dimensional map to construct a set of waiting personnel and calculate their temperature abnormality and spatial abnormality indicators.

[0008] S2. Carebot's built-in sensors collect facial expressions and voice information from patients waiting for treatment, perform sentiment analysis, and identify their emotional state.

[0009] S3. Calculate the stress index and risk index based on the waiting patient information set, and send the waiting patient information corresponding to the risk index greater than the threshold to the nurse station for early warning;

[0010] S4. Calculate health risk indicators based on the emotional state of the waiting person and assess their potential health risks;

[0011] S5. Based on the assessed potential health risks of waiting patients, health education will be provided to those with medium and low risks through carebot. For those with high risks, priority will be given to comforting them and their information will be sent to the nurse station for early warning.

[0012] The present invention is further configured such that S1 specifically includes:

[0013] Carebot's built-in sensors collect images of the waiting area to build a 3D map of the area. Data is extracted from the 3D model to create a waiting person information set, which includes their ID, location, posture, waiting time, and real-time body temperature.

[0014] The abnormality of the real-time body temperature of waiting personnel is quantified to obtain the temperature abnormality degree. The pressure of the waiting personnel's environment is quantified by combining the two dimensions of spatial crowding and waiting time to obtain the spatial abnormality index.

[0015] The present invention is further configured such that the waiting personnel set is: Among them, H is the set of waiting personnel, N is the total number of people in the waiting area, and h i It is a personnel information set, which includes: i ={ID i ,x i ,y i ,Z i ,t i ,T i}, ID i is the personnel ID, x i ,y i is the personnel position, Z i For personnel posture, t i is the waiting time, T i Real-time body temperature;

[0016] Calculation logic of abnormal body temperature: Among them, A i is the abnormality of body temperature, λ is the sensitivity of control abnormality, T i Take the temperature of people waiting for treatment;

[0017] Spatial anomaly indicator calculation logic: Among them, ρ i is the spatial anomaly index, γ is the spatial attenuation coefficient, τ is the time smoothing parameter, t i is the waiting time, (x i ,y i ) is the position coordinate of patient i, (x j ,y j ) is the position coordinate of waiting patient j, and ||·||2 is the Euclidean norm.

[0018] The present invention is further configured such that S2 specifically includes:

[0019] Carebot uses its built-in camera to collect facial images and voice information from patients waiting for treatment. It then uses facial expression recognition technology to obtain facial expression features, and uses voice emotion recognition technology to obtain voice features. The obtained facial expression features and voice features are then fused multimodally to obtain joint features.

[0020] Perform modal feature transformation on the joint features to obtain the fused hidden features;

[0021] The classification probability is calculated based on the hidden layer features to obtain the emotional state of the waiting personnel.

[0022] The present invention is further configured such that the modal feature transformation calculation logic is: Among them, m i is the hidden layer feature, g is the gate probability vector, ⊙ is the element-by-element multiplication symbol, ReLU is the activation function, W f is the facial dominant transformation matrix, z i is the joint feature, W v is the speech-dominated transformation matrix;

[0023] Classification probability calculation logic: Where s∈{anxiety, tension, calm}, is the classification probability, W p is the classification weight matrix, b p is the classification bias term, Softmax is the activation function;

[0024] Emotional state judgment logic: Among them, e i The emotional state of the waiting people.

[0025] The present invention is further configured such that S3 specifically includes:

[0026] High-risk tagging is performed based on the personnel information set. If the person is lying down and their emotional state is anxious, the high-risk tag is directly assigned without further calculation, and an alarm is immediately triggered at the nurse station.

[0027] Calculate the stress index and risk index of waiting patients based on real-time temperature classification;

[0028] The information of waiting patients whose risk index is greater than the threshold is sent to the nurse station for early warning.

[0029] The present invention is further configured such that the pressure index calculation logic is: Among them, P i is the pressure index, A i is the abnormality of body temperature, ρ i is the spatial anomaly index, β1 is the anxiety weight, is the anxiety state item, β2 is the tension weight, is the tension state term, θ T The set body temperature range;

[0030] Risk index calculation logic: Among them, R i is the risk index, η is the stress index weight coefficient, ω2 is the sitting posture weight, is the personnel sitting posture item, ω1 is the lying posture weight, For personnel lying position items;

[0031] If R i >θ R , send an early warning to the nurse station, where θ R The high risk index threshold is set.

[0032] The present invention is further configured such that S4 specifically includes:

[0033] The health risk index is calculated by combining the abnormal body temperature and emotional anxiety risk of the waiting personnel;

[0034] Health risk indicator calculation logic: Among them, Risk i is the health risk index, β is the emotional anxiety weight, It is a subjective state item.

[0035] The present invention is further configured such that S5 specifically includes: health risk determination logic: Among them, r1 and r2 are the set thresholds. The information of high-risk waiting patients will be sent to the nurse station for early warning, and carebot will be dispatched first to comfort them and play health science videos. Health science videos or texts will be played by carebot for medium and low-risk waiting patients.

[0036] The present invention also provides an intelligent risk prediction system based on multimodal perception, the system comprising:

[0037] Information collection module: The Carebot's built-in sensors collect environmental images of the waiting area to construct a three-dimensional map of the waiting area. Information is extracted from the three-dimensional map to construct a set of waiting personnel and calculate their temperature abnormality and spatial abnormality indicators.

[0038] Emotional state recognition module: Carebot's built-in sensors collect facial expressions and voice information from patients waiting for treatment, perform sentiment analysis, and identify their emotional state.

[0039] Risk calculation module: Calculates stress index and risk index based on the waiting patient information set, and sends waiting patient information corresponding to risk index greater than the threshold to the nurse station for early warning;

[0040] Health risk assessment module: Combined with the emotional state of patients waiting for treatment, calculate health risk indicators and assess their potential health risks;

[0041] Feedback module: Based on the assessed potential health risks of waiting patients, carebot is used to provide health education to waiting patients with medium and low risks. High-risk waiting patients are given priority for comfort and their information is sent to the nurse station for early warning.

[0042] The present invention provides an intelligent risk prediction method and system based on multimodal perception. The method collects environmental images of the waiting area through the built-in sensors of a carebot to construct a three-dimensional map of the waiting area, extracts information from the three-dimensional map, constructs a set of waiting personnel, and calculates the body temperature abnormality and spatial abnormality index of the waiting personnel; collects facial expressions and voice information of the waiting personnel based on the built-in sensors of the carebot, performs emotion analysis, and identifies the emotional state of the waiting personnel; calculates a stress index and a risk index based on the waiting personnel information set, and sends the waiting personnel information corresponding to a risk index greater than a threshold to a nurse station for early warning; calculates a health risk index based on the emotional state of the waiting personnel, and assesses the potential health risks of the waiting personnel; based on the assessed potential health risks of the waiting personnel, provides health education to waiting personnel with medium and low risks through the carebot, and gives priority to comforting waiting personnel with high risks and sending the waiting personnel information to the nurse station for early warning. The beneficial effects produced include:

[0043] Improve the efficiency of personnel management in waiting areas: This method uses multimodal sensing technology to comprehensively assess the stress and risks of waiting personnel. It can accurately identify high-risk individuals, optimize personnel management and flow scheduling in waiting areas, and reduce the occurrence of sudden health events.

[0044] Enhanced health warning and intervention capabilities: By combining the abnormal body temperature and emotional anxiety index of waiting patients, the system can provide early warning of high-risk individuals and automatically take intervention measures, such as sending alerts to the nurse station or providing health education through carebot, reducing the waste of medical resources and improving patients' sense of security and satisfaction.

[0045] Improve patient experience and satisfaction: The emotional state and health risks of waiting patients can be monitored and intervened in a timely manner, reducing patients' anxiety while waiting, enhancing patients' trust and satisfaction with hospital services, and thus improving the overall service quality of the hospital.

[0046] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without inventive efforts. In the drawings:

[0048] Figure 1 This is a flowchart of an intelligent risk prediction method based on multimodal perception, illustrating an exemplary embodiment of the present invention;

[0049] Figure 2 The figure is a schematic diagram of the structure of an intelligent risk prediction system based on multimodal perception, showing an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present invention with reference to the accompanying drawings and preferred embodiments. Those skilled in the art will readily appreciate the other advantages and benefits of the present invention from the disclosure herein. The present invention may also be implemented or applied through various other specific embodiments, and the various details in this specification may be modified or altered based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred embodiments are intended only to illustrate the present invention and are not intended to limit the scope of protection of the present invention.

[0051] It should be noted that the illustrations provided in the following embodiments are merely schematic illustrations of the basic concept of the present invention. Therefore, the illustrations only show components related to the present invention and are not drawn according to the number, shape, and size of components in actual implementation. In actual implementation, the type, quantity, and proportion of each component may be changed arbitrarily, and the component layout may also be more complex.

[0052] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it will be apparent to those skilled in the art that the embodiments of the present invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring the embodiments of the present invention.

[0053] Example 1

[0054] An intelligent risk prediction method based on multimodal perception, such as Figure 1 As shown, including:

[0055] S1. The carebot's built-in sensors collect images of the waiting area to construct a three-dimensional map of the waiting area. Information is extracted from the three-dimensional map to construct a set of waiting personnel and calculate their temperature abnormality and spatial abnormality indicators.

[0056] S2. Carebot's built-in sensors collect facial expressions and voice information from patients waiting for treatment, perform sentiment analysis, and identify their emotional state.

[0057] S3. Calculate the stress index and risk index based on the waiting patient information set, and send the waiting patient information corresponding to the risk index greater than the threshold to the nurse station for early warning;

[0058] S4. Calculate health risk indicators based on the emotional state of the waiting person and assess their potential health risks;

[0059] S5. Based on the assessed potential health risks of waiting patients, health education will be provided to those with medium and low risks through carebot. For those with high risks, priority will be given to comforting them and their information will be sent to the nurse station for early warning.

[0060] The present invention is further configured such that S1 specifically includes:

[0061] Carebot's built-in sensors collect images of the waiting area to build a 3D map of the area. Data is extracted from the 3D model to create a waiting person information set, which includes their ID, location, posture, waiting time, and real-time body temperature.

[0062] The degree of abnormality in the waiting person's real-time body temperature is quantified to obtain the temperature abnormality degree. The spatial abnormality index is then used to quantify the stress of the waiting person's environment, combining the two dimensions of spatial crowding and waiting time. Specifically, the carebot's built-in sensors include a camera and an infrared sensor. The camera is used to capture real-time images of the waiting area, while the infrared sensor is used to collect body temperature information from people in the waiting area. This collected information is uploaded to the cloud for processing, and a three-dimensional map of the environment is constructed. Object detection and tracking are performed on the waiting area images to obtain the waiting person's location information, posture, and waiting time. This is existing technology and will not be elaborated on here. Emotional fluctuations can affect body temperature. The temperature abnormality degree is used to assess the degree of temperature deviation of waiting patients while waiting. Long wait times or high crowd density can also affect emotions. The spatial abnormality index is used to evaluate the current patient's performance in terms of waiting time and crowd density. Spatial crowding specifically refers to the density of people around the current waiting person.

[0063] The present invention is further configured such that the waiting personnel set is: Among them, H is the set of waiting personnel, N is the total number of people in the waiting area, and h i It is a personnel information set, which includes: i ={ID i ,x i ,y i ,Z i ,t i ,T i}, ID i is the personnel ID, x i ,y i is the personnel position, Z i For personnel posture, t i is the waiting time, T i Real-time body temperature;

[0064] Calculation logic of abnormal body temperature: Among them, A i is the abnormality of body temperature, λ is the sensitivity of control abnormality, T i Take the temperature of people waiting for treatment;

[0065] Spatial anomaly indicator calculation logic: Among them, ρ i is the spatial anomaly index, γ is the spatial attenuation coefficient, τ is the time smoothing parameter, t i is the waiting time, (x i ,y i ) is the position coordinate of patient i, (x j ,y j) is the position coordinate of waiting person j, and ||·||2 is the Euclidean norm. Specifically, the waiting person set H contains the information of all waiting people in the area collected by the carebot during the current time period, including: the person ID is used to distinguish different waiting people, the person ID is unique, the person position is the specific spatial location of the person in the environment, and the person posture is divided into three categories: standing, sitting, and lying. Among them, the standing posture Z i =3, sitting posture Z i =2, lying position Z i =1, the waiting time is the length of time the patient has been waiting in this environment, which is obtained through object detection and tracking, which is an existing technology. The real-time body temperature is the real-time body temperature of the patient collected by the infrared sensor built into the carebot; in the calculation of body temperature abnormality, the body temperature abnormality A i It is represented by two exponential functions: one is the exponential increment when the body temperature is high The other is the exponential increment when the body temperature is low The difference between these two exponential functions calculates the temperature abnormality of the waiting personnel. λ is used to control the abnormal sensitivity. By adjusting λ, the sensitivity of the system to temperature abnormality is affected. The value is generally between 0.1 and 1 and can be adjusted according to actual needs. -2 is to make the temperature abnormality zero or negative when the temperature is within the normal range. Max(·,0) is to make the temperature abnormality zero when the temperature is within the normal range. In the calculation of spatial abnormality index, the spatial attenuation coefficient γ determines the degree of influence of spatial distance on the abnormality index. The larger the value, the smaller the influence of people farther away on the spatial abnormality index. The value range is between 0.1-1 and is adjusted according to the crowd density of the waiting area. The default value is 0.5. The time smoothing parameter τ is used to control the smoothing effect of waiting time on the spatial abnormality index. It is usually taken as 30 minutes or 1 hour to smooth the time effect. The Euclidean norm ||(x i ,y i )-(x j ,y j )||2 means calculating the spatial distance between two waiting patients. represents the sum of the spatial pressure contributions of all other waiting personnel j in the waiting area except waiting personnel i, It represents the spatial pressure contribution of waiting person j to waiting person i. The closer the distance, the greater the impact, and the closer the value is to 1. Tanh(·) is the hyperbolic tangent function, which is used to compress the summation result to [0,1) to avoid extreme values interfering with subsequent calculations. The impact of waiting time is used to simulate the psychological impact of waiting. The longer the waiting time, the closer the value is to 1.

[0066] The present invention is further configured such that S2 specifically includes:

[0067] Carebot uses its built-in camera to collect facial images and voice information from patients waiting for treatment. It then uses facial expression recognition technology to obtain facial expression features, and uses voice emotion recognition technology to obtain voice features. The obtained facial expression features and voice features are then fused multimodally to obtain joint features.

[0068] Perform modal feature transformation on the joint features to obtain the fused hidden features;

[0069] The classification probability is calculated based on the hidden features to obtain the emotional state of the waiting person. Specifically, facial image acquisition: through the built-in camera of carebot, the system captures the facial images of the waiting person in real time. Facial images contain a lot of important emotional information, such as: smiles, frowns, eyes and other expressions. These expressions can reflect the emotional state of the waiting person; voice information acquisition: through the built-in microphone of carebot, the system collects the voice data of the waiting person. The factors such as tone, speaking speed, and voice intensity contained in the voice can reflect emotional fluctuations. For example, when people are anxious or nervous, they usually speak faster and have a higher pitch; the acquired facial expression features and voice features are multimodally fused to obtain joint features. The feature extraction process is already well-known and will not be elaborated on here. Modal feature transformation further processes the joint features, mapping them into a high-dimensional latent feature space that can capture the complex relationships between different modalities. Classification probability calculation uses the transformed latent features as input into a classification model. The classifier calculates the probability of each emotion label, ultimately returning the most likely emotion state and its corresponding probability. The process generally consists of three steps: Feature extraction of facial expression and voice data: The carebot's camera and microphone collect facial expressions and voice information from patients waiting for treatment. Deep learning algorithms are then used to perform sentiment analysis on these facial expression and voice data to extract meaningful features.

[0070] Feature fusion and processing: Facial expression features and speech features are fused, and a gating mechanism is used to selectively enhance the influence of one modality. The fused features are then transformed to obtain a latent feature representation that captures the patient's emotional state.

[0071] Emotional state prediction: Based on latent features, the Softmax classification algorithm is used to calculate the probability of a patient being in different emotional states, such as anxiety, tension, and calmness. Based on this probability, the patient's emotional state is determined, providing a basis for subsequent health interventions.

[0072] The present invention is further configured such that the modal feature transformation calculation logic is: Among them, m iis the hidden layer feature, g is the gate probability vector, ⊙ is the element-by-element multiplication symbol, ReLU is the activation function, W f is the facial dominant transformation matrix, z i is the joint feature, W v is the speech-dominated transformation matrix;

[0073] Classification probability calculation logic: Where s∈{anxiety, tension, calm}, is the classification probability, W p is the classification weight matrix, b p is the classification bias term, Softmax is the activation function;

[0074] Emotional state judgment logic: Among them, e i is the emotional state of the waiting staff. Specifically, the hidden feature m i It is a feature vector representing the emotional state of the waiting person obtained by passing facial expression and voice features through a gating mechanism and an activation function. This vector contains a comprehensive description of the waiting person's emotions. The gated probability vector g controls the weights of facial expression features and voice features in the final fusion features. The gating mechanism allows the system to select more useful modal information for fusion based on the characteristics of the input data. The value range of g is [0,1]. Its value determines the contribution ratio of facial and voice features in the fusion features. If it approaches 1, facial features dominate the fusion; if it approaches 0, voice features dominate. The specific calculation formula of the gated probability vector is: W g is the gate control weight matrix, b g is the gate control bias term, W g z i +b g Used for linear transformation to map the input to three-dimensional space, σ(W g z i +b g ) is an activation function used to compress each dimension between (0,1), and the facial dominant transformation matrix W f Mainly used to extract cousin-related features, the voice-dominated transformation matrix W v Mainly used to extract speech-related features, W f and W v They are used to map input features to a new space. They are obtained through training and are learnable parameters of the neural network; classification probability It represents the classification probability of the waiting staff under different guns grabbing the counter. It needs to be calculated three times here, namely the probability of anxiety, the probability of tension, and the probability of calmness. The sum of the three probabilities finally obtained should be equal to 1. The classification weight matrix W pThe probability of converting hidden features into emotional states is obtained through training, and the classification bias term b p It is used to adjust the output value; based on the three classification probabilities obtained, the largest probability is taken as the judgment result and set as the emotional state of the waiting person.

[0075] The present invention is further configured such that S3 specifically includes:

[0076] High-risk tagging is performed based on the personnel information set. If the person is lying down and their emotional state is anxious, the high-risk tag is directly assigned without further calculation, and an alarm is immediately triggered at the nurse station.

[0077] Calculate the stress index and risk index of waiting patients based on real-time temperature classification;

[0078] The information of waiting personnel whose risk index is greater than the threshold is sent to the nurse station for early warning. Specifically, the logic of high-risk marking is: if the waiting person satisfies both the lying posture and the emotional state of anxiety, it proves that the probability of physical discomfort is very high. At this time, other external factors can only be used as the basis for medical staff to judge the illness of the waiting person. Therefore, the waiting person at this time does not participate in the subsequent judgment and is directly identified as a high-risk person. Medical staff need to intervene urgently to prevent accidents and ensure the health and safety of the waiting person; the stress index is calculated according to the set body temperature classification. If the body temperature is within the set range, consider whether the emotional state is anxious or nervous, whether the space where the waiting person is located is crowded, and whether the space abnormality index is a high value. If the waiting person's temperature is not within the set range, it proves that the body temperature is abnormal. The probability of such people being at risk if the temperature exceeds or is lower than the normal body temperature is greater. Such waiting personnel are not considered for other factors. Only the degree of abnormal body temperature and whether the waiting person is anxious are considered; the risk index is also calculated according to the body temperature classification. If the body temperature is within the preset range, the posture of the waiting person is considered, whether sitting or lying. The standing posture is not considered because if the waiting person can stand independently, it proves that he is still able to move. If the waiting person's temperature is not within the set range, it proves that the body temperature is abnormal, exceeding or lower than the normal body temperature. The probability of such people being at risk will be greater. At this time, other factors are not considered. Only the waiting time of the waiting person is considered. The longer the time, the higher the risk; different waiting thresholds are set according to different waiting environments to adjust the sensitivity of the program. If you are in a low-risk waiting environment such as dentistry or dermatology, the threshold can be appropriately raised. If you are in a high-risk waiting environment such as respiratory department or fever clinic, the threshold can be appropriately lowered.

[0079] The present invention is further configured such that the pressure index calculation logic is: Among them, P i is the pressure index, A i is the abnormality of body temperature, ρ iis the spatial anomaly index, β1 is the anxiety weight, is the anxiety state item, β2 is the tension weight, is the tension state term, θ T The set body temperature range;

[0080] Risk index calculation logic: Among them, R i is the risk index, η is the stress index weight coefficient, ω1 is the lying posture weight, is the personnel lying posture item, ω2 is the sitting posture weight, It is the personnel sitting posture item;

[0081] If R i >θ R , send an early warning to the nurse station, where θ R is the high risk index threshold. Specifically, the stress index P i In order to quantify the immediate stress of waiting personnel due to abnormal body temperature and crowded space within the set body temperature threshold range, different stress values will lead to different emotional states of waiting personnel. The emotional state is introduced into the stress index. The higher the value, the higher the stress state of the waiting personnel. If the waiting personnel's body temperature exceeds the range, the environmental pressure and calm and nervous emotions are not considered, and only the abnormal body temperature and anxious emotions are calculated. Through this stress index, the system can evaluate whether the waiting personnel are in a high stress state, providing a basis for the subsequent calculation of the risk index. The default value of the anxiety weight β1 is 0.3, which can be modified according to different waiting environments, ranging from 0.2 to 0.5. The tension weight β2 has a lower weight value than anxiety, and the default value is 0.1. It can be modified according to different waiting environments, ranging from 0.05 to 0.2. The anxiety state item Anxiety indicator function, which means limiting the emotional state to anxiety. If the emotional state is anxiety, the value is 1, if not, the value is 0. The tension state item The same logic as the anxiety state item above, the set body temperature range θ T The temperature usually ranges from 36 to 38, indicating a normal body temperature range; the high-risk index R iThere are mainly two detection standards: normal body temperature and extreme body temperature. For normal body temperature, the high-risk index is calculated in combination with the personnel posture, waiting time and stress indication. The calculation of the risk index takes into account the personnel posture because if the waiting personnel are in an uncomfortable state and it is difficult to remain standing, they will take postures such as sitting or lying down to relieve physical discomfort. For extreme body temperature, because the body temperature is enough to alarm, other privacy is not considered and the time is emphasized. The longer the waiting time, the greater the risk of danger for the waiting personnel. The lying posture weight ω1 is 0.8, ranging from 0.5 to 1.0, and is modified according to different waiting environments. The sitting posture weight ω2 is slightly lower than the lying posture, which is 0.2, ranging from 0.1 to 0.3, and is modified according to different waiting environments. The personnel lying posture item It is the lying position indicator function, which means that the personnel posture is restricted to lying position. If the personnel posture is lying position, the value is 1, if not, the value is 0. The personnel sitting position item The logic is the same as the above anxiety state item; if the high risk index R i When the threshold exceeds the set value, the corresponding waiting person data will be transmitted to the nurse station to generate an early warning prompt, reminding the nurse to pay special attention to the waiting person. The default threshold value is θ R The recommended range is 5.0, which can be modified according to the needs of different departments. The recommended range is between 3.0 and 7.0. The specific value is related to the sensitivity of the overall program.

[0082] The present invention is further configured such that S4 specifically includes:

[0083] The health risk index is calculated by combining the abnormal body temperature and emotional anxiety risk of the waiting personnel;

[0084] Health risk indicator calculation logic: Among them, Risk i is the health risk index, β is the emotional anxiety weight, is a subjective state item. Specifically, the health risk index evaluates the health risk of waiting patients by integrating multiple factors such as body temperature and emotional anxiety. Especially in the case of anxiety and abnormal body temperature, it can timely identify potential high-risk individuals so that emergency intervention measures can be taken quickly; the emotional anxiety weight β is used to control the contribution of emotional anxiety to health risk. Anxiety will increase health risk. The value range is [0,1]. The larger the value, the greater the impact of anxiety on health risk. Indicates the emotional state of the waiting person, e i =1 indicates that the patient is in an anxious state. The design logic is: when the emotional state is anxious, the subjective state item takes the value of 1; otherwise, when the emotional state is nervous or calm, the subjective state item is 0. This item is used to indicate that the emotional state is limited to anxiety.

[0085] The present invention is further configured such that S5 specifically includes: health risk determination logic: Among them, r1 and r2 are set thresholds. Specifically, based on the calculated health risk index, waiting patients are divided into three categories according to their health risks based on the set thresholds r1 and r2: low risk, medium risk, and high risk. The set thresholds r1 and r2 are usually set based on historical data and clinical experience, with the default settings of r1 = 1.0 and r2 = 2.0. The thresholds are adjusted according to different background environments. For high-risk waiting patients, the waiting patient information needs to be sent to the nurse station to notify the nurse station to pay close attention. The data sent to the nurse station includes: waiting patient information collection, waiting patient stress index, waiting patient emotional state, waiting patient health risk index, and waiting patient health risk. For medium-risk and low-risk waiting patients, carebot plays health science videos or text to convey health knowledge and alleviate waiting stress.

[0086] Example 2

[0087] See also Figure 2 , the exemplary intelligent risk prediction system based on multimodal perception includes:

[0088] Information collection module: The Carebot's built-in sensors collect environmental images of the waiting area to construct a three-dimensional map of the waiting area. Information is extracted from the three-dimensional map to construct a set of waiting personnel and calculate their temperature abnormality and spatial abnormality indicators.

[0089] Emotional state recognition module: Carebot's built-in sensors collect facial expressions and voice information from patients waiting for treatment, perform sentiment analysis, and identify their emotional state.

[0090] Risk calculation module: Calculates stress index and risk index based on the waiting patient information set, and sends waiting patient information corresponding to risk index greater than the threshold to the nurse station for early warning;

[0091] Health risk assessment module: Combined with the emotional state of patients waiting for treatment, calculate health risk indicators and assess their potential health risks;

[0092] Feedback module: Based on the assessed potential health risks of waiting patients, carebot is used to provide health education to waiting patients with medium and low risks. High-risk waiting patients are given priority for comfort and their information is sent to the nurse station for early warning.

[0093] It should be noted that the intelligent risk prediction system based on multimodal perception provided by the above embodiment and the intelligent risk prediction method based on multimodal perception provided by the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs operations has been described in detail in the method embodiment and will not be repeated here. In actual applications, the intelligent risk prediction system based on multimodal perception provided by the above embodiment can allocate the above functions to different functional modules as needed, that is, divide the internal structure of the system into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0094] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer program are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state drive.

[0095] It should be understood that the term "and / or" as used herein simply describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A alone, A and B together, or B alone. A and B can be singular or plural. Furthermore, the character " / " as used herein generally indicates an "or" relationship between the associated objects, but it may also indicate an "and / or" relationship. For specific understanding, please refer to the context.

[0096] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or plural.

[0097] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0098] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed system can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0101] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0102] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0103] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0104] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An intelligent risk prediction method based on multimodal perception, characterized in that: include: S1. The carebot's built-in sensors collect images of the waiting area to construct a three-dimensional map of the waiting area. Information is extracted from the three-dimensional map to construct a set of waiting personnel and calculate their temperature abnormality and spatial abnormality indicators. S2. Carebot's built-in sensors collect facial expressions and voice information from patients waiting for treatment, perform sentiment analysis, and identify their emotional state. S3. Calculate the stress index and risk index based on the waiting patient information set, and send the waiting patient information corresponding to the risk index greater than the threshold to the nurse station for early warning; S4. Calculate health risk indicators based on the emotional state of the waiting person and assess their potential health risks; S5. Based on the assessed potential health risks of waiting patients, health education will be provided to those with medium and low risks through carebot. For those with high risks, priority will be given to comforting them and their information will be sent to the nurse station for early warning.

2. The intelligent risk prediction method based on multimodal perception according to claim 1 is characterized in that: S1 specifically includes: Carebot's built-in sensors collect images of the waiting area to build a 3D map of the area. Data is extracted from the 3D model to create a waiting person information set, which includes their ID, location, posture, waiting time, and real-time body temperature. The abnormality of the real-time body temperature of waiting personnel is quantified to obtain the temperature abnormality degree. The pressure of the waiting personnel's environment is quantified by combining the two dimensions of spatial crowding and waiting time to obtain the spatial abnormality index.

3. The intelligent risk prediction method based on multimodal perception according to claim 2 is characterized in that: Waiting staff set: Among them, H is the set of waiting personnel, N is the total number of people in the waiting area, and h i It is a personnel information set, which includes: i ={ID i ,x i ,y i ,Z i ,t i ,T i }, ID i is the personnel ID, x i ,y i is the personnel position, Z i For personnel posture, t i is the waiting time, T i Real-time body temperature; Calculation logic of abnormal body temperature: Among them, A i is the abnormality of body temperature, λ is the sensitivity of control abnormality, T i Take the temperature of people waiting for treatment; Spatial anomaly indicator calculation logic: Among them, ρ i is the spatial anomaly index, γ is the spatial attenuation coefficient, τ is the time smoothing parameter, t i is the waiting time, (x i ,y i ) is the position coordinate of patient i, (x j ,y j ) is the position coordinate of waiting patient j, and ||·||2 is the Euclidean norm.

4. The intelligent risk prediction method based on multimodal perception according to claim 1 is characterized in that: S2 specifically includes: Carebot uses its built-in camera to collect facial images and voice information from patients waiting for treatment. It then uses facial expression recognition technology to obtain facial expression features, and uses voice emotion recognition technology to obtain voice features. The obtained facial expression features and voice features are then fused multimodally to obtain joint features. Perform modal feature transformation on the joint features to obtain the fused hidden features; The classification probability is calculated based on the hidden layer features to obtain the emotional state of the waiting personnel.

5. The intelligent risk prediction method based on multimodal perception according to claim 4 is characterized in that: Modal feature transformation calculation logic: Among them, m i is the hidden layer feature, g is the gate probability vector, ⊙ is the element-by-element multiplication symbol, ReLU is the activation function, W f is the facial dominant transformation matrix, z i is the joint feature, W v is the speech-dominated transformation matrix; Classification probability calculation logic: Where s∈{anxiety, tension, calm}, is the classification probability, W p is the classification weight matrix, b p is the classification bias term, Softmax is the activation function; Emotional state judgment logic: Among them, e i The emotional state of the waiting people.

6. The intelligent risk prediction method based on multimodal perception according to claim 1 is characterized in that: S3 specifically includes: High-risk tagging is performed based on the personnel information set. If the person is lying down and their emotional state is anxious, the high-risk tag is directly assigned without further calculation, and an alarm is immediately triggered at the nurse station. Calculate the stress index and risk index of waiting patients based on real-time temperature classification; The information of waiting patients whose risk index is greater than the threshold is sent to the nurse station for early warning.

7. The intelligent risk prediction method based on multimodal perception according to claim 6 is characterized in that: Pressure index calculation logic: Among them, P i is the pressure index, A i is the abnormality of body temperature, ρ i is the spatial anomaly index, β1 is the anxiety weight, is the anxiety state item, β2 is the tension weight, is the tension state term, θ T The set body temperature range; Risk index calculation logic: Among them, R i is the risk index, ω1 is the lying posture weight, is the personnel lying posture item, ω2 is the sitting posture weight, It is the personnel sitting posture item; If R i >θ R , send an early warning to the nurse station, where θ R The high risk index threshold is set.

8. The intelligent risk prediction method based on multimodal perception according to claim 1 is characterized in that: S4 specifically includes: The health risk index is calculated by combining the abnormal body temperature and emotional anxiety risk of the waiting personnel; Health risk indicator calculation logic: Among them, Risk i is the health risk index, β is the emotional anxiety weight, It is a subjective state item.

9. The intelligent risk prediction method based on multimodal perception according to claim 8, characterized in that: S5 specifically includes: Health risk determination logic: Among them, r1 and r2 are the set thresholds. The information of high-risk waiting patients will be sent to the nurse station for early warning, and carebot will be dispatched first to comfort them and play health science videos. Health science videos or texts will be played by carebot for medium and low-risk waiting patients.

10. An intelligent risk prediction system based on multimodal perception, used to implement the intelligent risk prediction method based on multimodal perception according to any one of claims 1 to 9, characterized in that: include: Information collection module: The Carebot's built-in sensors collect environmental images of the waiting area to construct a three-dimensional map of the waiting area. Information is extracted from the three-dimensional map to construct a set of waiting personnel and calculate their temperature abnormality and spatial abnormality indicators. Emotional state recognition module: Carebot's built-in sensors collect facial expressions and voice information from patients waiting for treatment, perform sentiment analysis, and identify their emotional state. Risk calculation module: Calculates stress index and risk index based on the waiting patient information set, and sends waiting patient information corresponding to risk index greater than the threshold to the nurse station for early warning; Health risk assessment module: Combined with the emotional state of patients waiting for treatment, calculate health risk indicators and assess their potential health risks; Feedback module: Based on the assessed potential health risks of waiting patients, carebot is used to provide health education to waiting patients with medium and low risks. High-risk waiting patients are given priority for comfort and their information is sent to the nurse station for early warning.

Images