Multifunctional health monitoring and early warning system based on radar and vision fusion

Abstract

The application provides a multifunctional health monitoring and early warning system based on radar and vision fusion, and relates to the technical field of intelligent early warning.The system comprises a feedback receiving module, a radar feedback signal and an image feedback signal are received; a monitoring and identification module, a monitoring data is determined by traversing a target monitoring area, and monitoring and identification is performed; an abnormality determination module, a first monitoring data set and a second monitoring data set are fused and analyzed, bidirectional abnormality determination is performed, and a warning instruction is generated; a risk analysis module, an abnormality determination result is uploaded to a cloud platform for risk analysis, and warning information is sent to a remote terminal according to a risk level. Through the application, the technical problem that in the prior art, health monitoring relies on a single sensor, is easily affected by the environment, and has a high false alarm rate, thereby affecting the accuracy of the warning information, can be solved. Through risk assessment by combining radar and vision fusion, the accuracy of abnormality identification is improved, and the real-time performance and accuracy of early warning are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent early warning, and in particular to a multifunctional health monitoring early warning system based on radar and vision fusion. BACKGROUND

[0002] With the arrival of an aging society, the health and safety problems of groups such as the elderly living alone are increasingly prominent. Traditional health monitoring methods rely on single sensors for data collection and wearable devices. Single sensor technology (such as vision or radar) is effective in some specific application scenarios. However, in complex environmental conditions, there are problems such as inconvenience of use and limited monitoring range. Contact devices may be affected by human motion, sweat or other factors, and cannot provide continuous monitoring without contact, making it difficult to identify abnormal behavior in a timely manner, resulting in a high false alarm rate, which in turn affects the accuracy of early warning information.

[0003] In summary, the prior art has the technical problem that health monitoring relies on a single sensor, is easily affected by the environment, has a high false alarm rate, and thus affects the accuracy of early warning information. SUMMARY

[0004] The purpose of the present application is to provide a multifunctional health monitoring early warning system based on radar and vision fusion to solve the technical problem in the prior art that health monitoring relies on a single sensor, is easily affected by the environment, has a high false alarm rate, and thus affects the accuracy of early warning information.

[0005] In view of the above problems, the present application provides a multifunctional health monitoring early warning system based on radar and vision fusion, which comprises: a feedback receiving module for emitting a radar signal through a radar module, receiving a radar feedback signal, recording a target monitoring area in real time through a camera module, and receiving an image feedback signal; a monitoring and identification module for determining target monitoring human data by traversing the target monitoring area, monitoring and identifying the target monitoring human data based on the radar feedback signal and the image feedback signal respectively, and generating a first monitoring data set and a second monitoring data set; an abnormality determination module for fusion analysis of the first monitoring data set and the second monitoring data set, generating a health monitoring result for bidirectional abnormality determination, generating an early warning instruction according to the abnormality determination result, and the early warning instruction containing early warning information; and a risk analysis module for uploading the abnormality determination result to a cloud platform for risk analysis by triggering an early warning mechanism through the early warning instruction, and sending early warning information to a remote terminal according to the abnormality risk level.

[0006] Optionally, a radar signal receiving unit is configured to emit a continuous frequency modulation wave radar signal to a target monitoring area through a radar module in the host, receive a reflected signal in the target area through a receiving antenna, and obtain an initial radar echo signal; a frequency shift screening unit is configured to perform frequency shift screening based on the initial radar echo signal, and obtain the radar feedback signal; and a multi-view acquisition unit is configured to adjust a camera module in the host to collect an angle at a preset interval period, determine a data collection angle, collect multi-view images through the camera module at the data collection angle, and obtain the image feedback signal.

[0007] Optionally, a grid determination unit is configured to divide the target monitoring area into NXM grid units, and determine a plurality of grid units, where N and M are positive integers greater than 1, and N and M can be equal; a scanning result determination unit is configured to control the radar module and the camera module to traverse the plurality of grid units to perform human body recognition scanning according to a preset path, and obtain target monitoring human body data, where the target monitoring human body data includes radar scanning results and image scanning results; a feature extraction unit is configured to perform feature extraction on the radar scanning results based on the radar feedback signal, determine human body micro-motion feature data, perform change monitoring according to the human body micro-motion feature data, and generate the first monitoring data set; and a trajectory monitoring unit is configured to perform feature extraction on the image scanning results based on the image feedback signal, determine human body contour feature data, perform trajectory monitoring according to the human body contour feature data, and generate the second monitoring data set.

[0008] Optionally, a multi-feature extraction subunit is configured to perform short-time Fourier transform on the radar scanning results based on the radar feedback signal, extract human body micro-motion feature data, and the human body micro-motion feature data includes a breathing fundamental frequency feature, a heartbeat harmonic feature, and a body motion acceleration feature; and a state monitoring subunit is configured to perform change state monitoring based on the breathing fundamental frequency feature, the heartbeat harmonic feature, and the body motion acceleration feature, obtain a plurality of body motion change events, and add the plurality of body motion change events to the first monitoring data set.

[0009] Optionally, a frame difference processing subunit is configured to perform frame difference processing on the image scanning results based on the image feedback signal, and construct a human body bounding box parameter; an identifier allocation subunit is configured to allocate an identity identifier to the human body bounding box parameter, and determine the human body contour feature data; and a vector calculation subunit is configured to perform motion vector calculation based on the human body contour feature data, obtain human body motion trajectory data, and add the human body motion trajectory data to the second monitoring data set.

[0010] Optionally, a matrix construction unit is configured to perform wavelet denoising on the first monitoring data set, construct a radar feature matrix, perform key frame extraction on the second monitoring data set, and construct a visual feature matrix; a regularization fusion unit is configured to perform dynamic time warping fusion on the radar feature matrix and the visual feature matrix to generate a joint feature vector; a feature division unit is configured to divide the joint feature vector into a physiological feature subset and a behavior feature subset; a correlation analysis unit is configured to perform time series correlation analysis based on the physiological feature subset to obtain a physiological health monitoring result; a motion analysis unit is configured to perform skeletal point motion analysis based on the behavior feature subset to obtain a behavior health monitoring result; and a result generation unit is configured to associate and integrate the physiological health monitoring result and the behavior health monitoring result to generate the health monitoring result.

[0011] Optionally, a physiological abnormality determination unit is configured to perform physiological abnormality determination based on the physiological health monitoring result to obtain a physiological abnormality determination result; a behavior abnormality determination unit is configured to perform behavior abnormality determination based on the behavior health monitoring result to obtain a behavior abnormality determination result; a physiological early warning unit is configured to generate physiological early warning information when only the physiological abnormality determination result exists, perform confidence analysis according to the physiological early warning information, and trigger a first early warning instruction according to a first confidence level; a behavior early warning unit is configured to generate behavior early warning information when only the behavior abnormality determination result exists, perform confidence analysis according to the behavior early warning information, and trigger a second early warning instruction according to a second confidence level; and a confidence analysis unit is configured to generate bidirectional early warning information when the physiological abnormality determination result and the behavior abnormality determination result coexist, perform confidence analysis according to the bidirectional early warning information, and trigger a third early warning instruction according to a third confidence level.

[0012] Optionally, a first risk analysis unit is configured to upload the physiological abnormality determination result to a cloud platform to perform risk analysis and generate a first abnormality risk level when the first early warning instruction triggers the early warning mechanism; a first early warning unit is configured to send the physiological early warning information to a remote terminal according to the first abnormality risk level; a second risk analysis unit is configured to upload the behavior abnormality determination result to the cloud platform to perform risk analysis and generate a second abnormality risk level when the second early warning instruction triggers the early warning mechanism; a second early warning unit is configured to send the behavior early warning information to the remote terminal according to the second abnormality risk level; a third risk analysis unit is configured to upload the physiological abnormality determination result and the behavior abnormality determination result to the cloud platform to perform risk analysis and generate a third abnormality risk level when the third early warning instruction triggers the early warning mechanism; and a third early warning unit is configured to send the bidirectional early warning information to the remote terminal according to the third abnormality risk level.

[0013] Optionally, a level determination unit is configured to call a set of historical abnormal events in a target monitoring area, perform a causal analysis based on the set of historical abnormal events, identify risk level labels for the set of historical abnormal events according to an analysis result, and determine an abnormal risk level list; a weight optimization unit is configured to perform weight optimization according to the abnormal risk level list, and construct a gradient boosting decision tree; and a decision tree embedding unit is configured to embed the gradient boosting decision tree into a cloud platform for risk analysis.

[0014] Optionally, a response recording unit is configured to record a response process of the remote terminal to the early warning information, and generate a response record report; a processing efficiency determination unit is configured to analyze a processing result of the remote terminal by traversing the response record report, and generate a terminal processing efficiency; a mapping relationship determination unit is configured to perform mapping analysis on the abnormal risk level and the early warning information, and determine a risk-early warning mapping relationship; and an early warning mode determination unit is configured to dynamically adjust the risk-early warning mapping relationship according to the terminal processing efficiency, generate a risk level score, and change an early warning mode for the remote terminal according to the risk level score.

[0015] The technical solutions provided in the present application have at least the following beneficial effects:

[0016] The feedback receiving module is configured to emit a radar signal through the radar module, receive a radar feedback signal, and receive an image feedback signal by recording the target monitoring area in real time through the camera module. The monitoring and identifying module is configured to determine target monitoring human body data by traversing the target monitoring area, and perform monitoring and identification on the target monitoring human body data based on the radar feedback signal and the image feedback signal, respectively, to generate a first monitoring data set and a second monitoring data set. The abnormality determination module is configured to perform fusion analysis on the first monitoring data set and the second monitoring data set, generate a health monitoring result for bidirectional abnormality determination, generate an early warning instruction according to an abnormality determination result, and determine that the early warning instruction includes early warning information. The risk analysis module is configured to upload the abnormality determination result to a cloud platform for risk analysis by triggering an early warning mechanism through the early warning instruction, and send early warning information to a remote terminal according to an abnormal risk level. That is, the radar module and visual fusion are used for monitoring, the monitoring result is subjected to bidirectional abnormality determination, abnormal risk is quickly identified and uploaded to the cloud platform for risk assessment, early warning information is sent, false positives and false negatives are reduced, the accuracy and timeliness of abnormal situation identification are improved, and the real-time and accuracy of early warning are improved.

[0017] The above description is only a summary of the technical solutions of the present application. In order to more clearly understand the technical means of the present application, the specific embodiments of the present application can be implemented according to the content of the specification, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only exemplary, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0019] Figure 1 The structure schematic diagram of the multi-functional health monitoring and early warning system based on radar and vision fusion of the present application.

[0020] Figure 2 The structure schematic diagram of the monitoring and identification module in the multi-functional health monitoring and early warning system based on radar and vision fusion of the present application.

[0021] Explanation of reference numerals: feedback receiving module 11, monitoring and identification module 12, abnormality determination module 13, risk analysis module 14, grid determination unit 21, scanning result determination unit 22, feature extraction unit 23, trajectory monitoring unit 24. DETAILED DESCRIPTION

[0022] The present application provides a multi-functional health monitoring and early warning system based on radar and vision fusion, which solves the technical problem in the prior art that the health monitoring relies on a single sensor, which is easily affected by the environment, resulting in a high false alarm rate, thereby affecting the accuracy of the early warning information. Through radar module and vision fusion monitoring, two-way abnormality determination is performed on the monitoring results, abnormal risks are quickly identified and uploaded to the cloud platform for risk assessment, early warning information is sent, false positives and false negatives are reduced, the accuracy and timeliness of abnormal situation identification are improved, and the real-time and accuracy of early warning are improved.

[0023] Below, the technical solutions in the present application will be described clearly and completely with reference to the drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. It should be understood that the present application is not limited by the example embodiments described herein. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application. In addition, it should be noted that, for the convenience of description, only parts related to the present application are shown in the drawings, not all.

[0024] Embodiments, please refer to the accompanying Figure 1 The present application provides a multi-functional health monitoring and early warning system based on radar and vision fusion, wherein the multi-functional health monitoring and early warning system based on radar and vision fusion comprises:

[0025] The feedback receiving module 11 is configured to emit a radar signal through the radar module, receive a radar feedback signal, and record a target monitoring area in real time through the camera module, and receive an image feedback signal.

[0026] Further, the feedback receiving module 11 in the multi-functional health monitoring and early warning system based on radar and vision fusion is further configured to:

[0027] The radar signal receiving unit is configured to emit a continuous frequency modulation wave radar signal to the target monitoring area through the radar module in the host computer, receive the reflected signal in the target area through the receiving antenna, and obtain an initial radar echo signal; the frequency shift screening unit is configured to perform frequency shift screening based on the initial radar echo signal, and obtain the radar feedback signal; and the multi-view acquisition unit is configured to adjust the camera module in the host computer according to a preset interval period to determine a data acquisition angle, and perform multi-view acquisition through the camera module according to the data acquisition angle to obtain the image feedback signal.

[0028] Specifically, the host computer and the slave computer are registered through the cloud platform to complete device binding and network configuration, efficient communication between the host computer and the slave computer is ensured through an intranet communication module (such as supporting WiFi or Bluetooth), and the real-time and stability of data transmission are ensured through an extranet communication module (such as supporting 4G / 5G transmission). The intranet communication module is used to transmit the radar signal collected by the slave computer to the host computer through the intranet to ensure that the data can be quickly and stably transmitted. The extranet communication module is used for transmission between the cloud platform to ensure the real-time and stability of data transmission.

[0029] The host is installed at the core position that can maximize the coverage of the target monitoring area, ensuring that the radar and camera can cover the entire monitoring range. The radar module and camera module are both installed on the host, the radar module in the host continuously transmits and receives radar signals, and the camera module collects video signals in real time, transmits the radar signals and video signals to the computing module, and performs human body recognition and posture analysis through the preset AI data model.

[0030] The host includes both a radar module and a camera module, which transmits and receives radar signals through the radar module for monitoring human posture, breathing, and heartbeat; the camera module includes a binocular camera and a 270-degree rotating base for collecting video signals to assist the radar module in human body recognition and posture analysis. When there are areas where cameras are not allowed to be installed, such as bedrooms, bathrooms, etc., only radar monitoring is used, and millimeter wave radar is emitted by the fall monitoring slave (which focuses on monitoring human posture and identifying fall status) and the breathing and heart rate monitoring slave (which focuses on monitoring human breathing and heartbeat status), the reflected millimeter wave is transmitted to the host through the internal network communication module, the host judges whether there is a person in the target area according to the radar signal uploaded by the slave, when it is judged that there is a person in the target area, the posture of the person is analyzed, and the breathing and heartbeat of the person are monitored in real time through the fall monitoring slave and the breathing and heart rate monitoring slave.

[0031] The frequency of the millimeter wave radar signal will shift with the movement of the object, and the frequency shift in the radar echo reflects the movement speed of the target object. The movement of the human body is different from other objects (such as furniture, pets, etc.), usually showing different speed patterns and frequency shift characteristics. The movement of the human body usually has a relatively continuous and specific movement pattern (such as walking, falling, standing still, etc.), while the movement of other objects is relatively simple or irregular. For example, fixed objects such as walls and tables have no movement, and the frequency change of the reflection signal of static objects is small, while the human body shows a continuously changing frequency shift signal. The movement frequency range of pets (such as dogs and cats) is relatively wide, showing rapid and jumping irregular movements (such as running and jumping, etc.). The volume of pets is usually small, so the radar signal reflected by them will be relatively weak, but their movement frequency characteristics have high irregularity. According to these rules, it can be judged whether there is a person in the target area.

[0032] The strength and time of arrival of the radar echo signal can reflect the position of the object, by which the distance and approximate position of the object are judged. If the signal strength is high and the time delay is stable, it indicates that the target is a human body, and further posture analysis is performed. The frequency change of the radar signal reflects the motion of the human body. According to the frequency shift amount (i.e., the size of the frequency offset), the speed and direction of the human body motion are inferred. When the human body performs different motions (such as walking, standing, sitting, squatting, etc.), the frequency change of the radar signal presents different patterns. For example, when the human body walks or stands, the frequency shift is relatively stable, the amplitude is small, and there is a certain periodicity; when the human body runs or moves quickly, the frequency shift has a larger amplitude and a shorter period, reflecting high-speed motion; when the human body sits or squats, the frequency change has a small amplitude, and the persistence and stability of the reflected signal are high. Through the frequency change of the radar echo signal, the motion state of the human body limbs is inferred. The human arms, legs, and other parts have different motion patterns during walking or motion, and the frequency change of the radar signal can reflect the dynamic characteristics of these different parts.

[0033] The fall monitoring slave and the respiratory and heart rate monitoring slave are respectively deployed in the center of the area (such as the bedroom, bathroom, etc.) that needs to be monitored. The radar module of the fall monitoring slave continuously transmits and receives radar signals, and the signals are transmitted to the host through the intranet communication module. In addition, in addition to the fall monitoring slave and the respiratory and heart rate monitoring slave, there is also a user presence monitoring slave, which focuses on monitoring whether there is a human body in the target area. The host and the slave are respectively set with a time period and a warning threshold. In the set time period, the user presence monitoring slave is used for monitoring, and if the presence of the user cannot be monitored in the preset time period, that is, no vital signs are detected in the time period when the target area should have a person (for example, an old person should get up and enter the monitoring area in a certain time period in the morning), it is judged as an abnormality, thereby triggering a warning. The presence monitoring slave transmits radar signals, and judges whether there is a person in the monitoring area through the reflected signals. The warning threshold is set to expect to monitor the signs or activities in the target area in a time period, that is, it is normal for a person to move in the target area in this time period. If this threshold is exceeded and no activity or sign is monitored, a warning is triggered. For example, if an old person is expected to enter the kitchen, bathroom, etc. after getting up in a certain time period (such as 5-7 o'clock), and the threshold set is "detecting at least 1 human activity or sign signal", if no sign is detected in this time period (i.e., the threshold is not met), it is considered that there is an abnormality, triggering a warning.

[0034] The computing module of the host determines the human posture according to the radar signals uploaded from the slave. The radar module of the respiratory and heart rate monitoring slave continuously transmits and receives radar signals, which are transmitted to the host through the intranet communication module. The computing module of the host determines the human respiratory and heartbeat state according to the radar signals uploaded from the slave. The host and the slave form a network through the intranet communication module. One host can connect multiple slaves, and one slave can only connect one host. Non-contact monitoring is achieved through radar and camera, improving user comfort. The computing module in the host has an AI data model built-in, which is used to process radar signals and video signals, realize human recognition, posture analysis, respiratory and heartbeat monitoring, etc. The main task of the computing module is to receive and process data uploaded from different sensors (such as radar, camera, etc.), perform signal interpretation and analysis, and be responsible for data processing, feature extraction, pattern recognition, and final health monitoring and alarm decision. The computing module usually has high-performance computing capability, especially when processing complex signals (such as radar and video signals), it often relies on high-speed computing, parallel processing and hardware acceleration. The AI data model is a model trained through machine learning, deep learning, etc., which can analyze, classify, predict or identify the input data, extract useful information from the input radar signals and video signals, and realize human recognition, posture analysis, respiratory and heartbeat monitoring, etc.

[0035] Collect a large amount of radar signal and video signal data, including normal behaviors (such as walking, standing, sitting) and abnormal behaviors (such as falling, etc.). During data collection, behavior samples need to be labeled, including behavior type, posture information, etc. According to the task requirements, select a convolutional neural network architecture. The collected labeled data is divided into training set and test set. The training set is used to train the model, and the test set is used to evaluate the performance of the model. Use the training set to train the model. Through the back propagation algorithm and gradient descent optimization algorithm, adjust the weights and biases of the model. After several training cycles, gradually learn the rules in the data. Evaluate the accuracy of the model through the test set, and adjust the structure or training parameters (such as learning rate, batch size, etc.) of the model as needed, use cross-validation technology to improve the generalization ability of the model. Set the convergence conditions of the model, such as the change of the validation loss is less than 0.01 for 5 consecutive rounds or the training set accuracy reaches 95%. When the model meets the convergence condition, stop training and get the AI data model.

[0036] The host and slave units begin working together to monitor the human body's status in real time. The radar module in the host emits a continuous frequency modulated (FM) radar signal to the target monitoring area, constantly changing the frequency of the emitted signal to achieve precise target detection. The position and velocity of the target object are calculated based on the frequency changes of the reflected signal. The radar signal travels through the target monitoring area and is reflected back by the human body or other objects. The received antenna receives the reflected radar echo signal to obtain the initial radar echo signal, which contains information such as the target object's position and velocity. Frequency shift filtering is performed on the initial radar echo signal. This filtering analyzes the frequency changes in the radar echo signal to identify frequency components related to the target object's motion. The purpose is to extract these motion-related frequency components from the echo signal. The radar feedback signal is generated by the radar module transmitting and receiving radar signals. The camera module records the target monitoring area in real time and receives image feedback signals containing information about the target object, such as its position, velocity, and direction of movement.

[0037] The camera module adjusts its acquisition angle according to a preset interval to ensure coverage of the entire monitoring area. The camera module acquires images from multiple perspectives, obtaining a series of image feedback signals. The preset interval is determined based on the target area and acquisition requirements, ensuring coverage of different parts of the monitoring area. In areas with frequent target activity, the preset interval is typically shorter, resulting in more frequent acquisition. Based on the adjusted acquisition angle, the camera module performs multi-view acquisition, capturing images from different times and angles. These images are then transmitted to the computing module in the host computer to form image feedback signals. The image feedback signals are real-time image data of the target monitoring area captured by the camera, providing visual information about the target object, such as its appearance, shape, color, and movement.

[0038] By combining radar and cameras, the limitations of a single sensor are overcome, ensuring coverage of the entire monitoring area and reducing the impact of environmental factors. Radar signals are highly robust in low-light and complex environments, while the visual information provided by cameras helps to accurately identify abnormal behaviors (such as falls, movement, etc.), providing more comprehensive image information and aiding in precise analysis of the target's status.

[0039] The monitoring and identification module 12 is used to traverse the target monitoring area to determine the target monitoring human data, and to monitor and identify the target monitoring human data based on the radar feedback signal and the image feedback signal respectively, and generate a first monitoring dataset and a second monitoring dataset.

[0040] Further details are attached. Figure 2 As shown, the monitoring and identification module 12 in the multifunctional health monitoring and early warning system based on radar and vision fusion is also used for:

[0041] The grid determination unit 21 is configured to divide the target monitoring area into N×M grid units, and determine a plurality of grid units, where N and M are positive integers greater than 1, and N and M can be equal. The scan result determination unit 22 is configured to control the radar module and the camera module to traverse the plurality of grid units according to a preset path for human body recognition scanning, and obtain target monitoring human body data, where the target monitoring human body data includes radar scan results and image scan results. The feature extraction unit 23 is configured to perform feature extraction on the radar scan results based on the radar feedback signal, determine human micro-motion feature data, perform change monitoring according to the human micro-motion feature data, and generate the first monitoring data set. The trajectory monitoring unit 24 is configured to perform feature extraction on the image scan results based on the image feedback signal, determine human contour feature data, perform trajectory monitoring according to the human contour feature data, and generate the second monitoring data set.

[0042] Further, the feature extraction unit 23 is further configured to: a multi-feature extraction subunit, configured to perform short-time Fourier transform on the radar scan results based on the radar feedback signal, extract human micro-motion feature data, and the human micro-motion feature data includes respiratory fundamental frequency features, heartbeat harmonic features, and body motion acceleration features; and a state monitoring subunit, configured to perform change state monitoring based on the respiratory fundamental frequency features, the heartbeat harmonic features, and the body motion acceleration features, obtain a plurality of body motion change events, and add the plurality of body motion change events to the first monitoring data set.

[0043] Further, the trajectory monitoring unit 24 is further configured to: a frame difference processing subunit, configured to perform frame difference processing on the image scan results based on the image feedback signal, and construct human bounding box parameters; an identifier allocation subunit, configured to allocate an identity identifier to the human bounding box parameters, and determine the human contour feature data; and a vector calculation subunit, configured to perform motion vector calculation based on the human contour feature data, obtain human motion trajectory data, and add the human motion trajectory data to the second monitoring data set.

[0044] Specifically, the target monitoring area is divided into a plurality of small grid units, and each grid unit represents an independent monitoring area. N and M are the number of rows and columns of the division, and N and M are greater than 1 and can be equal, which ensures the detailed division of the monitoring area. The size and number of each grid unit can be adjusted according to the scale of the actual monitoring area. For example, if the target monitoring area is a 10m×10m room, the area is divided into 5×5 grid units, and the size of each grid unit is 2m×2m, so that the monitoring area can be refined to ensure that the activities in each grid unit can be monitored by the radar and camera modules.

[0045] According to the preset path, the radar module and the camera module on the host control the scanning of these grid units in sequence, and obtain target monitoring human body data. The radar module obtains the dynamic information of the object by emitting signals and receiving reflected waves, and the camera records the shape and action of the target by shooting images. Each grid unit is scanned by the radar and the camera respectively to obtain human body identification data in the area. By traversing each grid unit, the radar and the camera can comprehensively monitor every corner in the target area, so as to obtain the target monitoring human body data.

[0046] According to the radar feedback signal, the radar scanning result in the target monitoring human body data is extracted, and data related to human body micro-motion (such as breathing, heartbeat, etc.) is obtained, that is, human body micro-motion feature data. By analyzing the changes of the human body micro-motion feature data, a first monitoring data set is generated. Specifically, the fall monitoring slave and the respiration and heart rate monitoring slave are devices specially used for monitoring the health status of the human body. The former is responsible for monitoring whether a fall occurs, and the latter is responsible for monitoring the breathing and heartbeat of the human body. The target area is scanned by the radar signal, and the radar data reflected back is collected. The computing module on the host processes and analyzes the received radar signal, extracts key information, and generates a monitoring data set.

[0047] The fall monitoring slave and the respiration and heart rate monitoring slave respectively obtain the radar signal in the target area and transmit it to the computing module on the host. The computing module is responsible for processing and analyzing the received radar signal. The computing module of the host performs short-time Fourier transform (STFT) on the received radar feedback signal, thereby extracting the frequency components of the signal. By analyzing the radar signal in the time and frequency domains through STFT, frequency information related to human body micro-motion is obtained. For example, assuming that the detected radar signal contains the micro-motion of the human body (such as breathing, heartbeat, etc.), the host obtains different frequency components through STFT analysis, and extracts feature data such as breathing fundamental frequency, heartbeat harmonic and body motion acceleration from them.

[0048] By analyzing the frequency components in the radar signal, the host extracts the fundamental frequency feature related to breathing, which reflects the normal breathing frequency of the human body. When the fundamental frequency is abnormal (such as too fast or too slow), it indicates that there may be an abnormal situation. The harmonic feature of the heartbeat is obtained by frequency analysis of the radar signal, which can provide the rhythm information of the heartbeat. When the heartbeat rhythm is abnormal, it is identified and marked as abnormal in time. By extracting the acceleration feature from the radar signal, the body motion of the human body is monitored. If abnormal body motion (such as rapid body motion or fall) is detected, corresponding marking is performed.

[0049] Based on the extracted respiratory fundamental frequency features, heartbeat harmonic features, and body motion acceleration features, change state monitoring is performed to obtain a plurality of body motion change events. Change state monitoring refers to dynamic monitoring based on the extracted micro-motion feature data (such as respiration, heartbeat, body motion, etc.) to identify whether an abnormal state has occurred in the human body. The plurality of body motion change events are obvious change behaviors of the human body occurring during the monitoring process, such as rapid breathing, arrhythmia, body motion abnormalities, etc. For example, the harmonic energy ratio of the respiratory fundamental frequency is calculated, and when the harmonic energy ratio is greater than 1.5, it is determined that the user is experiencing rapid breathing; the phase continuity of the heartbeat harmonic is analyzed, and if the phase jump exceeds 30 degrees and lasts for 3 seconds, it is marked as arrhythmia; in combination with the variance value of the acceleration signal, when the variance suddenly increases by more than 3 times the baseline, it is determined that the user is experiencing body motion abnormalities. All monitored body motion change events are added to the first monitoring data set as the basis for subsequent anomaly detection and early warning.

[0050] According to the image feedback signal, the image scanning results in the target monitoring human body data are subjected to feature extraction to obtain human body shape feature data, which usually involves information such as the contour and posture of the human body, and is mainly related to the subtle movements of the human body. By analyzing the trajectory of the human body contour feature data, the moving trajectory of the human body is tracked, the position change is analyzed, and whether there is an abnormal behavior is detected to generate a second monitoring data set. Specifically, based on the received image feedback signal, the image scanning results (i.e., consecutive video frames) are subjected to difference processing. Frame difference is to compare the pixel values between consecutive frames to find the changed part and then extract the region in the image where motion occurs. The region in the image that has changed usually corresponds to the movement of the target object (such as the human body).

[0051] After frame difference processing, the moving target region is identified, and a bounding box is constructed within the region. The bounding box is a rectangular box drawn around the target object (such as the human body) to represent the position and size of the target, and the position of the bounding box is usually determined according to the contour of the human body and the boundary of the changed region. For example, assuming that a user is detected walking in the monitoring area, a rectangular box is marked in the image to enclose the user's body contour, and the bounding box parameters such as position (x, y coordinates), size (width, height), etc. are calculated.

[0052] Whenever a new human target is detected, it will be assigned a unique identity identifier, which can be a number, letter or other unique marker, to distinguish different target objects. In this way, no matter how the human body moves in the monitoring area, each target can be accurately tracked and different targets can be distinguished. The identity identifier refers to a unique identifier assigned to each recognized target (such as a human body) to distinguish different target objects. The human body contour feature data includes the shape, size, shape, etc. of the human body, which helps to identify and track the human body. By analyzing the human body contour in the bounding box, it is determined whether the human body is standing upright, sitting or lying down, and the action state of the human body is further analyzed.

[0053] Based on the human body contour feature data, motion vector calculation is performed to obtain human body motion trajectory data. Motion vector can reflect the direction and speed of human body movement in the image. By calculating the displacement of the target in each frame, the moving trajectory of the target is inferred. The motion vector is a vector that describes the direction and speed of movement of a target object (such as a human body) in an image. The motion vector is obtained by calculating the change in pixel position between adjacent image frames. The human body motion trajectory data is obtained by tracking and calculating the motion vector of the human body, and is data that describes the motion path of the human body, which can provide the moving trajectory and direction of the target, helping to monitor and analyze human activity. For example, a human body walks from the left bedroom to the right living room, calculates the position change of the human body in each frame of image, and generates the corresponding motion vector, reflecting the walking direction and speed of the human body.

[0054] All extracted human body motion trajectory data and its corresponding human body contour feature data are associated and added to the second monitoring data set, which contains dynamic information of the target in the monitoring area, including the position change, motion trajectory and related visual feature data of the human body. By dividing the monitoring area into multiple grid units, each area is monitored in detail to reduce the possibility of false reporting, and the human body state is monitored in real time from two data sets to identify abnormal behaviors such as falling, rapid breathing, etc.

[0055] The abnormality determination module 13 is configured to fuse and analyze the first monitoring data set and the second monitoring data set to generate a health monitoring result for bidirectional abnormality determination, and generate a warning instruction containing warning information according to the abnormality determination result.

[0056] Further, the abnormality determination module 13 in the multifunctional health monitoring and warning system based on radar and vision fusion is further configured to:

[0057] The matrix construction unit is configured to perform wavelet denoising on the first monitoring data set, construct a radar feature matrix, perform key frame extraction on the second monitoring data set, and construct a visual feature matrix; the normalization fusion unit is configured to perform dynamic time normalization fusion on the radar feature matrix and the visual feature matrix to generate a joint feature vector; the feature division unit is configured to divide the joint feature vector into a physiological feature subset and a behavior feature subset; the correlation analysis unit is configured to perform time correlation analysis based on the physiological feature subset to obtain a physiological health monitoring result; the motion analysis unit is configured to perform skeletal point motion analysis based on the behavior feature subset to obtain a behavior health monitoring result; and the result generation unit is configured to associate and integrate the physiological health monitoring result and the behavior health monitoring result to generate the health monitoring result.

[0058] The physiological abnormality determination unit is configured to perform physiological abnormality determination based on the physiological health monitoring result to obtain a physiological abnormality determination result; the behavior abnormality determination unit is configured to perform behavior abnormality determination based on the behavior health monitoring result to obtain a behavior abnormality determination result; the physiological warning unit is configured to generate physiological warning information when only the physiological abnormality determination result exists, perform confidence analysis according to the physiological warning information, and trigger a first warning instruction according to a first confidence; the behavior warning unit is configured to generate behavior warning information when only the behavior abnormality determination result exists, perform confidence analysis according to the behavior warning information, and trigger a second warning instruction according to a second confidence; and the confidence analysis unit is configured to generate bidirectional warning information for confidence analysis when the physiological abnormality determination result and the behavior abnormality determination result coexist, and trigger a third warning instruction according to a third confidence.

[0059] Specifically, wavelet denoising is performed on the first monitoring data set, that is, wavelet denoising processing is performed on the radar feedback signal to remove noise and retain effective features in the signal. Wavelet denoising is a signal processing method for reducing noise in a signal and retaining useful features, based on the multi-scale characteristics of wavelet transform, noise is removed by analyzing the distribution of the signal in different frequency bands. Feature extraction is performed on the denoised radar feedback signal, and the radar feature matrix is a matrix constructed by the radar feedback signal after feature extraction and processing, which includes various human micro-motion features obtained from the radar feedback signal, such as breathing rate, heartbeat rhythm, and body acceleration, etc.

[0060] Key frame extraction is performed on the image sequence captured by the camera, and representative image frames are selected as key frames, which contain the main changes or actions in the target monitoring area, and a visual feature matrix is obtained. The visual feature matrix is a matrix constructed by the image feedback signal after key frame extraction and feature extraction, which contains visual feature data about the target object, such as human contour features, motion trajectories, etc.

[0061] Since radar data and vision data are often not synchronized in time and have different acquisition frequencies, directly comparing their feature matrices can lead to errors. Therefore, the radar feature matrix and the vision feature matrix are fused, and a dynamic time warping (DTW) algorithm is used to generate a joint feature vector representing the fusion result of radar and vision data by optimally aligning two different time series. DTW calculates the distance between the two time series to find the best matching path, thereby eliminating the misalignment problem on the time axis, allowing the two sets of data to be compared on the same time scale.

[0062] Specifically, the radar feature matrix and the vision feature matrix are input into the DTW algorithm as two time series. DTW measures the similarity of the two time series and outputs an optimal alignment path. The data in the radar feature matrix and the vision feature matrix are aligned according to the optimal alignment path, and a new joint feature vector is finally generated, containing fusion information from radar and vision, for subsequent health monitoring analysis.

[0063] The generated joint feature vector is divided into two subsets: a physiological feature subset and a behavior feature subset. The physiological feature subset includes user physiological data (such as respiratory rate, heartbeat rhythm, etc.), which is used to assess the individual's physiological health status, and the behavior feature subset includes data from user motion trajectory and behavior (such as posture change, gait, etc.), which is used to assess the individual's behavior health status.

[0064] The physiological feature subset is analyzed for temporal correlation. The physiological feature subset contains heartbeat, breathing, and body movement data from radar signals, which change over time, so temporal analysis is needed. For example, sudden changes in heart rate and breathing rate may indicate abnormal physiological conditions (such as heart problems, rapid breathing, etc.). During analysis, it can also be determined whether these changes are associated with certain behavior patterns (such as exercise, rest). Based on the physiological health monitoring results, it can be determined whether the individual has an abnormal physiological condition, thereby providing a health warning. For example, under normal conditions, a user's heart rate is 60-100 beats per minute, and it changes slowly over time. According to the monitoring results, it is found that the user's heart rate has suddenly changed from 72 beats per minute to 115 beats per minute in five seconds, indicating an abnormal heart rate increase and arrhythmia, which requires a warning. Under normal conditions, a user's gait is stable, and the body movement acceleration is 0.3 m / s². At a certain moment, the body movement acceleration suddenly jumps to 1.5 m / s², indicating an unusual sudden change, which may indicate a fall.

[0065] The skeletal point motion analysis is performed on the behavior feature subset. By analyzing the position changes of the skeletal points of the human body (such as the joints of the shoulders, elbows, knees, etc.) at different time points, the posture changes and motion trajectories of the human body are understood. For example, if certain skeletal points are detected to have a large displacement within a short time, and this displacement is inconsistent with the normal behavior pattern (such as standing, walking), it may indicate an abnormal event such as falling, imbalance, etc. Through the motion trajectory and speed of the skeletal points, specific behavior patterns such as fast walking, sitting, turning, etc. are identified. Abnormal patterns (such as rapid falling, sudden stopping, etc.) can be used as an indication of health risks. According to the behavior health monitoring result, it is identified whether there is an abnormal behavior such as sudden falling, unstable gait, etc., so as to perform health warning. For example, under normal circumstances, the user walks from the sofa to the kitchen, and the skeletal points of the shoulders, knees, etc. move along the normal trajectory (such as a step length of about 0.5 meters and a step speed of about 0.8 steps / second). When falling occurs, the skeletal points of the shoulders and knees rapidly displace, and a large displacement occurs, such as a displacement of the knee point of 3 meters / second, indicating that the user has a severe motion within a short time, which may have a serious health risk.

[0066] The physiological health monitoring result and the behavior health monitoring result are associated and integrated to generate a final health monitoring result. By combining the two, the health status of an individual is comprehensively evaluated, including the physiological state (such as heartbeat, breathing) and the behavior state (such as motion, posture). For example, it is detected that the user has a rapid heartbeat (physiological abnormality) and falls (behavioral abnormality), and through the association and integration, a comprehensive health monitoring result is generated, indicating that the user is in a dangerous state and may need emergency assistance. The health monitoring result is a comprehensive health assessment result generated by combining the physiological health monitoring result and the behavior health monitoring result, which is used to evaluate the overall health status of an individual. By separate analysis of physiological features and behavior features, the physiological health status and behavior health status of an individual are evaluated, which helps to detect potential health problems (such as heart disease, falling risk, etc.) earlier.

[0067] According to the physiological health monitoring result, physiological abnormality is determined, for example, the heart rate suddenly rises to more than 120 times / minute and is accompanied by rapid breathing, which may indicate that the user has an acute health problem such as heart disease or breathing difficulty. The physiological health monitoring result is compared with the preset normal value range to determine whether there is a physiological abnormality. According to the behavior health monitoring result, behavior abnormality is determined, for example, if the user is detected to have a sudden fall, or the motion trajectory is inconsistent with normal activity (such as rapid displacement, etc.), it is determined that there is a behavior abnormality such as falling or accidental injury.

[0068] If only the physiological abnormality determination result is abnormal, physiological warning information is generated. According to the physiological warning information, confidence analysis is performed to evaluate the reliability of the warning information. Confidence refers to the degree of certainty of the occurrence of an abnormal state. For example, if the heart rate is too fast (e.g., more than 100 times per minute) or too slow, the heart rate is abnormal; if the breathing rate is too fast or too slow, or the breathing is rapid, etc., the breathing is abnormal; if the body motion acceleration suddenly increases, it may indicate a sudden change in body state, such as a change in body position or a sudden illness, and the body motion is abnormal. When the physiological data is determined to be abnormal, the corresponding physiological warning information is generated, including the type of abnormality (e.g., heart rate too fast, rapid breathing, etc.), the degree of abnormality, the abnormal event, the duration, etc.

[0069] Confidence analysis is a key process for evaluating the reliability of warning information. By comparing the current physiological monitoring data with the user's historical health data, it can be determined whether the abnormality is consistent with the user's health condition. If the historical data indicates that the abnormality is relatively rare, the confidence may be high. At the same time, the state of the sensor is evaluated, and if the sensor is not abnormal and has high accuracy, a higher confidence is given. The severity of the abnormality affects the confidence, and a serious physiological abnormality (e.g., heart rate too fast, rapid breathing) results in a higher confidence. For example, if the heart rate acceleration occurs in a user with a history of heart disease and exceeds the preset threshold by 20%, the confidence may be 90%. According to the confidence analysis result, a warning instruction is triggered. If only the physiological abnormality occurs, the first warning instruction is triggered.

[0070] Similarly, if only the behavior abnormality determination result is abnormal, behavior warning information is generated. According to the behavior warning information, confidence analysis is performed to evaluate the reliability of the warning information. For example, when a fall event is detected, a warning may be issued for possible fall or injury risk. At the same time, confidence analysis is performed to analyze the likelihood of falling. Confidence analysis may combine images captured by the camera and motion data obtained by the radar, as well as historical data. If the confidence exceeds a certain threshold (e.g., 90%), a second warning instruction is generated to trigger an emergency response mechanism.

[0071] Similarly, if both physiological abnormalities and behavioral abnormalities are detected simultaneously (e.g., accelerated heartbeat and detected fall), a two-way warning information is generated, which needs to include both physiological abnormality information and behavioral abnormality information. The two-way warning information may include a more urgent warning that the two abnormalities may affect each other, such as heart problems causing a fall, which needs to be handled immediately. After the two-way warning information is generated, a confidence analysis is performed to evaluate the accuracy and reliability of the two-way warning. Considering the simultaneous occurrence of physiological abnormalities and behavioral abnormalities, the confidence needs to be determined by a confidence analysis of both physiological abnormalities and behavioral abnormalities to determine the reliability of the warning information. That is, based on the accuracy and abnormality degree of physiological data, the abnormal reliability of physiological health status is evaluated; at the same time, the possibility of behavioral abnormalities is evaluated through the analysis of behavioral data, such as the accuracy of the fall detection system. In addition, the data of radar, vision and other sensors are combined to improve the accuracy of the overall warning.

[0072] According to the confidence analysis result of the two-way warning information, a third confidence is generated, which is the final evaluation result after considering physiological and behavioral abnormalities. If the third confidence exceeds a preset threshold (e.g., 90%), a third warning instruction will be triggered, including an urgent prompt, an automatic response, a risk level evaluation, etc. Since the situation is relatively serious, an emergency call can be directly made to the hospital, and an alarm information can be sent to family members. Combined with the simultaneous occurrence of physiological and behavioral abnormalities, the health status of the user is comprehensively evaluated to provide more accurate warning information.

[0073] The risk analysis module 14 is configured to trigger a warning mechanism through the warning instruction, upload the abnormality determination result to a cloud platform for risk analysis, and send a warning information to a remote terminal according to the abnormality risk level.

[0074] Further, the risk analysis module 14 in the multifunctional health monitoring and warning system based on radar and vision fusion is further configured to:

[0075] The first risk analysis unit is configured to upload the physiological abnormality determination result to the cloud platform for risk analysis when the first early warning instruction triggers the early warning mechanism, and generate a first abnormality risk level. The first early warning unit is configured to send the physiological early warning information to the remote terminal according to the first abnormality risk level. The second risk analysis unit is configured to upload the behavior abnormality determination result to the cloud platform for risk analysis when the second early warning instruction triggers the early warning mechanism, and generate a second abnormality risk level. The second early warning unit is configured to send the behavior early warning information to the remote terminal according to the second abnormality risk level. The third risk analysis unit is configured to upload the physiological abnormality determination result and the behavior abnormality determination result to the cloud platform for risk analysis when the third early warning instruction triggers the early warning mechanism, and generate a third abnormality risk level. The third early warning unit is configured to send the bidirectional early warning information to the remote terminal according to the third abnormality risk level.

[0076] Specifically, when the first early warning instruction triggers the early warning mechanism, i.e., the user has a physiological abnormality, the physiological abnormality determination result is uploaded to the cloud platform for risk analysis to evaluate the risk level of the current health status. For example, if it is found that the heart rate of an elderly person suddenly increases to 180 bpm, which exceeds the normal value, it is identified as a heart abnormality, triggering the first early warning instruction. Once the first early warning instruction is triggered, the physiological abnormality determination result will be uploaded to the cloud platform for further analysis. The cloud platform will receive these data and run a gradient boosting decision tree to evaluate the risk in combination with the user's health history data (such as heart disease, respiratory problems, etc.), and generate a risk level for this abnormal event, i.e., generate a first abnormality risk level.

[0077] In the cloud platform, the risk analysis model performs risk assessment based on the uploaded physiological abnormality data and generates a first abnormality risk level for the event. If the risk of the event is high (such as a high likelihood of cardiac arrest), it will be marked as a high risk level; if the event is mild (such as a slightly accelerated heartbeat without other symptoms), it may be marked as a low risk. According to the generated first abnormality risk level, the cloud platform will send the physiological early warning information to the remote terminal through the network, and the device of the remote terminal will receive real-time early warning information.

[0078] For example, assume that the following data is obtained from a worn physiological monitoring device (e.g., a heart rate monitor): the user's heart rate has spiked to 180 bpm over the past 15 minutes and exceeds a set threshold (120 bpm); the user has a history of heart disease and has not experienced similar heart rate abnormalities recently. Based on these data, a first warning instruction is triggered, and the data is uploaded to the cloud platform. After receiving the data, the cloud platform processes it using a gradient boosting decision tree model and assesses the risk level of the event as high risk. The cloud platform then sends physiological warning information to a remote terminal, which displays that the user's heartbeat has accelerated and has exceeded a safe range, and recommends immediate action. Through the automated warning mechanism, physiological abnormalities are detected in a timely manner and warnings are issued, avoiding health risks due to delayed reactions.

[0079] Similarly, when the second warning instruction triggers the warning mechanism, it indicates that the user has a behavior abnormality, and the behavior abnormality determination result is uploaded to the cloud platform for risk analysis and evaluation, generating a second abnormality risk level. For example, if a person suddenly falls or moves uncoordinated during an activity, the second warning instruction is triggered immediately, indicating a behavior abnormality. The behavior abnormality data is uploaded to the cloud platform in real time, including the time of the event, the type of abnormality, the user's state information (e.g., vital signs data), and environmental factors (e.g., slippery ground). Based on the second abnormality risk level, the cloud platform sends behavior warning information to a remote terminal, including the type of behavior abnormality, the time, the risk level, and suggested handling measures.

[0080] When the third warning instruction triggers the warning mechanism, it indicates that the user has both physiological abnormalities and behavior abnormalities, and it is possible that the behavior abnormalities are caused by physiological abnormalities. For example, through heart rate monitoring, it is found that the user's heart rate has risen sharply (e.g., above 150 bpm), and behavior abnormalities (e.g., falling, losing balance) have been detected, indicating a serious health threat, and the third warning instruction is triggered. Once the third warning instruction is triggered, both the physiological abnormalities (e.g., rapid heart rate) and the behavior abnormalities (e.g., falling) are uploaded to the cloud platform. After receiving these data, the cloud platform will perform risk assessment based on a gradient boosting decision tree, generating a comprehensive abnormality risk level. Considering the dual threat to the user's health from the two abnormal events (physiological and behavior), a higher risk level (e.g., high risk) is generated. Based on the generated third abnormality risk level, the cloud platform sends bidirectional warning information to relevant remote terminals, including detailed abnormality descriptions, occurrence times, possible health risks, warning levels, and emergency measure suggestions.

[0081] The corresponding early warning mechanism is triggered by the early warning instruction, which is triggered based on preset conditions. Once the early warning instruction is triggered, the abnormality determination result (such as heart rate abnormality, fall, etc.) is uploaded to the cloud platform through the external network communication module, including the abnormality type, occurrence time, abnormality parameter (such as heart rate value), etc. The external network communication module supports 4G / 5G communication for data transmission with the cloud platform. After receiving the abnormality determination result, the cloud platform performs risk analysis through gradient boosting decision tree to determine the corresponding abnormality risk level, and sends early warning information to the remote terminal, including the detailed description of the abnormal event, the risk level, the emergency suggestion, etc., to notify the relevant personnel to take action in time to prevent the accident from worsening. The early warning methods include short message, telephone, APP push, etc., to ensure that the early warning information is conveyed in time.

[0082] Further, the risk analysis module 14 in the radar and vision fusion based multi-functional health monitoring and early warning system is also used for:

[0083] a level determination unit for retrieving a set of historical abnormal events in the target monitoring area, performing causal analysis based on the set of historical abnormal events, and identifying risk level tags for the set of historical abnormal events according to the analysis results to determine an abnormality risk level list; a weight optimization unit for performing weight optimization according to the abnormality risk level list to construct a gradient boosting decision tree; and a decision tree embedding unit for embedding the gradient boosting decision tree into a cloud platform for risk analysis.

[0084] Specifically, a set of historical abnormal events in the target monitoring area is obtained, i.e. all abnormal event records occurring during the past monitoring period stored by the cloud platform, including physiological abnormalities (such as irregular heartbeat, rapid breathing, etc.) and behavioral abnormalities (such as falling, losing consciousness, etc.). The cloud platform extracts all abnormal event data occurring in the target monitoring area within a certain period of time from the database, including the identification of physiological and behavioral abnormalities, occurrence time, severity assessment, and corresponding handling measures.

[0085] The cloud platform uses an embedded machine learning model to analyze historical abnormal events and discover potential causal relationships through analysis. Causal analysis can help determine whether the occurrence of certain abnormal events will lead to the occurrence of other events. Through causal analysis, it may be found that, for example, tachycardia may lead to a user falling, or there is a potential link between falling events and heart health. During the analysis process, each abnormal event is assigned a risk level tag indicating the potential risk of the event, such as low risk, medium risk, high risk, etc. The abnormality risk level list is a list of all historical abnormal events sorted according to the risk level tags of the events, which can identify which events pose a greater threat to user health and prioritize handling of these high-risk events.

[0086] The anomaly risk level list has been assigned a risk level for each anomaly. To improve the accuracy of risk prediction, gradient boosting decision trees adjust these levels through weight optimization. For example, if certain events (such as falls) are often associated with high risks (such as severe injuries, hospitalization, etc.) in past monitoring, the weights of these events will be increased.

[0087] During the training process, gradient boosting decision trees gradually build a series of decision trees. In the training of each tree, the model focuses on how to reduce the current prediction error (usually based on gradient descent of the loss function). Gradient boosting decision trees gradually enhance the prediction ability through ensemble learning, and each tree will correct the wrong prediction of the previous tree to ultimately obtain a powerful and accurate model. For example, if the model initially misjudges some low-risk events as high-risk, subsequent decision trees will correct this error by learning from the error to achieve the optimization goal.

[0088] Using the historical anomaly event set to train the gradient boosting decision tree model, during the training process, each round of model considers the residual error (i.e., prediction error) in the previous prediction result and optimizes the residual error to improve the prediction accuracy of the model. After training is completed, the performance of the model needs to be evaluated, and common evaluation indicators include accuracy, recall rate, F1-score, etc. By comparing the model's prediction results with the actual situation, it is ensured that the model can effectively identify different risk levels of abnormal events. For example, for fall events, the model should be able to identify high-risk events with a high recall rate (to capture as many fall events as possible), and for low-risk events (such as small body movements, repeated actions, etc.), false positives should be avoided.

[0089] Deploy the trained gradient boosting decision tree model to the cloud platform, and use the powerful computing power of cloud computing to perform real-time risk analysis on new data. Gradient boosting decision trees can receive real-time data from monitoring devices, combine historical data for real-time prediction, and generate abnormal risk levels. The cloud platform will issue warnings based on the analysis results to help relevant personnel take necessary intervention measures in a timely manner. Through causal analysis and weight optimization based on historical events, gradient boosting decision trees can more accurately identify and predict abnormal events, especially in high-risk events, and can issue early warnings.

[0090] Further, the risk analysis module 14 in the radar and vision fusion-based multifunctional health monitoring and early warning system is also used for:

[0091] The response recording unit is configured to record the response processing of the remote terminal to the early warning information, and generate a response record report; the processing efficiency determination unit is configured to analyze the processing result of the remote terminal by traversing the response record report, and generate terminal processing efficiency; the mapping relationship determination unit is configured to analyze the mapping relationship between the abnormal risk level and the early warning information, and determine a risk-early warning mapping relationship; and the early warning mode determination unit is configured to dynamically adjust the risk-early warning mapping relationship according to the terminal processing efficiency, generate a risk level score, and change the early warning mode for the remote terminal according to the risk level score.

[0092] Specifically, after receiving the early warning information, the remote terminal will perform response processing, automatically record the response time and response action (such as whether to contact rescue, whether to confirm user safety, etc.) of the terminal, and generate a response record report. The response record report records the response of the remote terminal to each early warning information, including the time, action content, and response personnel, and is used to evaluate the reaction efficiency and processing quality of the remote terminal.

[0093] All response records are traversed to calculate the response efficiency of each remote terminal, including response time (such as the time taken to process the early warning) and processing quality (such as whether the response is timely and effective). The terminal processing efficiency is a measure of the speed and quality of the remote terminal's response to early warning information, and is usually calculated based on response time, response frequency, and processing effectiveness. For example, if the response time of one terminal after receiving the early warning is 10 minutes, and the response time of another terminal is only 2 minutes, then the processing efficiency of the first terminal is lower.

[0094] According to different abnormal events and corresponding risk levels, a mapping relationship between risk and early warning information is established, different risk levels correspond to different early warning information content, triggering mode and processing priority. For example, high-risk events may trigger more urgent and frequent early warnings, while low-risk events only send mild early warnings. According to the response efficiency of the remote terminal, the risk-early warning mapping relationship is dynamically adjusted. If a terminal frequently fails to respond effectively, the early warning mode of the terminal is increased, such as increasing the frequency or intensity of early warning, to ensure that the terminal responds in time. At the same time, the risk level score is also adjusted according to the historical performance, for example, a certain event originally rated as medium risk may be upgraded due to the slow reaction of the remote terminal, and the future early warning mode is changed according to this adjustment.

[0095] When a terminal fails to respond effectively to the same type of warning for 3 times in succession, the warning mode is changed according to this situation, such as increasing the alarm frequency of the terminal, modifying the warning content (such as adding more emergency prompts), or converting the terminal to a higher level of emergency response mode. By dynamically adjusting the warning mode and risk level score, the response capability of the remote terminal is optimized in real time. The timeliness and processing efficiency of the warning are improved, and delayed or ineffective responses are avoided.

[0096] In summary, the multifunctional health monitoring and warning system based on radar and vision fusion provided in the present application has the following beneficial effects:

[0097] The feedback receiving module is configured to emit a radar signal through the radar module, receive a radar feedback signal, and record a target monitoring area in real time through the camera module to receive an image feedback signal. The monitoring and identifying module is configured to traverse the target monitoring area to determine target monitoring human body data, monitor and identify the target monitoring human body data based on the radar feedback signal and the image feedback signal, respectively, and generate a first monitoring data set and a second monitoring data set. The abnormality determining module is configured to fuse and analyze the first monitoring data set and the second monitoring data set to generate a health monitoring result for bidirectional abnormality determination, generate a warning instruction according to an abnormality determination result, and include warning information in the warning instruction. The risk analysis module is configured to upload the abnormality determination result to a cloud platform for risk analysis by triggering a warning mechanism through the warning instruction, and send warning information to a remote terminal according to an abnormal risk level. That is, the radar module and vision fusion are used for monitoring, bidirectional abnormality determination is performed on the monitoring result, abnormal risk is quickly identified and uploaded to the cloud platform for risk assessment, warning information is sent, false positives and false negatives are reduced, the accuracy and timeliness of abnormal situation identification are improved, and the real-time and accuracy of the warning are improved.

[0098] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0099] Obviously, various modifications and changes can be made to the present application by those skilled in the art without departing from the spirit and scope of the present application. Thus, if these modifications and changes fall within the scope of the present application and its equivalent technology, the present application is intended to include these modifications and changes.

Claims

1. A multi-functional health monitoring and early warning system based on radar and vision fusion, characterized in that, The method comprises the following steps: The feedback receiving module is used for transmitting radar signals through the radar module, receiving radar feedback signals, and recording the target monitoring area in real time through the camera module to receive image feedback signals; The monitoring and identifying module is used for determining target monitoring human body data by traversing the target monitoring area, monitoring and identifying the target monitoring human body data based on the radar feedback signals and the image feedback signals respectively, and generating a first monitoring data set and a second monitoring data set; The abnormality determining module is used for performing fusion analysis on the first monitoring data set and the second monitoring data set, generating a health monitoring result for bidirectional abnormality determination, generating a warning instruction according to the abnormality determination result, and the warning instruction containing warning information; The risk analysis module is used for triggering a warning mechanism through the warning instruction, uploading the abnormality determination result to a cloud platform for risk analysis, and sending warning information to a remote terminal according to an abnormal risk level; The abnormality determining module comprises: The matrix construction unit is used for wavelet denoising the first monitoring data set, constructing a radar feature matrix, extracting key frames from the second monitoring data set, and constructing a visual feature matrix; The normalization fusion unit is used for dynamically time warping and fusing the radar feature matrix and the visual feature matrix to generate a joint feature vector; The feature division unit is used for dividing the joint feature vector into a physiological feature subset and a behavior feature subset; The correlation analysis unit is used for performing time series correlation analysis based on the physiological feature subset to obtain a physiological health monitoring result; The motion analysis unit is used for performing skeletal point motion analysis based on the behavior feature subset to obtain a behavior health monitoring result; The result generation unit is used for associating and integrating the physiological health monitoring result and the behavior health monitoring result to generate the health monitoring result; The physiological abnormality determining unit is used for performing physiological abnormality determination based on the physiological health monitoring result to obtain a physiological abnormality determination result; The behavior abnormality determining unit is used for performing behavior abnormality determination based on the behavior health monitoring result to obtain a behavior abnormality determination result; The physiological warning unit is used for generating physiological warning information when only the physiological abnormality determination result exists, performing confidence analysis according to the physiological warning information, and triggering a first warning instruction according to a first confidence level; The behavior warning unit is used for generating behavior warning information when only the behavior abnormality determination result exists, performing confidence analysis according to the behavior warning information, and triggering a second warning instruction according to a second confidence level; The confidence analysis unit is used for generating bidirectional warning information for confidence analysis when the physiological abnormality determination result and the behavior abnormality determination result coexist, and triggering a third warning instruction according to a third confidence level.

2. The multi-functional health monitoring and warning system based on radar and vision fusion as claimed in claim 1, wherein, The feedback receiving module comprises: The radar signal receiving unit is used for transmitting continuous frequency modulation wave radar signals to the target monitoring area through the radar module in the host computer, receiving reflected signals in the target area through a receiving antenna, and obtaining initial radar echo signals; A frequency shift screening unit is configured to perform frequency shift screening based on the initial radar echo signal to obtain the radar feedback signal; A multi-view acquisition unit is configured to adjust a camera module in a host device at a preset interval period to determine a data acquisition angle, perform multi-view acquisition at the data acquisition angle through the camera module, and obtain the image feedback signal.

3. The radar and vision fusion based multi-functional health monitoring and warning system as claimed in claim 1, wherein, The monitoring and identifying module comprises: A grid determination unit is configured to divide the target monitoring area into N×M grid units and determine a plurality of grid units, where N and M are positive integers greater than 1 and N is equal to M; A scanning result determination unit is configured to control the radar module and the camera module to traverse the plurality of grid units to perform human body identification scanning according to a preset path, and obtain target monitoring human body data, which comprises radar scanning results and image scanning results; A feature extraction unit is configured to perform feature extraction on the radar scanning results based on the radar feedback signal to determine human body micro-motion feature data, perform change monitoring according to the human body micro-motion feature data, and generate the first monitoring data set; A trajectory monitoring unit is configured to perform feature extraction on the image scanning results based on the image feedback signal to determine human body contour feature data, perform trajectory monitoring according to the human body contour feature data, and generate the second monitoring data set.

4. The multi-functional health monitoring and warning system based on radar and vision fusion as claimed in claim 3, wherein, The feature extraction unit comprises: A multi-feature extraction subunit is configured to perform short-time Fourier transform on the radar scanning results based on the radar feedback signal to extract human body micro-motion feature data, which comprises respiratory fundamental frequency features, heartbeat harmonic features, and body motion acceleration features; A state monitoring subunit is configured to perform change state monitoring based on the respiratory fundamental frequency features, the heartbeat harmonic features, and the body motion acceleration features to obtain a plurality of body motion change events, and add the plurality of body motion change events to the first monitoring data set.

5. The radar and vision fusion based multi-functional health monitoring and warning system as claimed in claim 3, wherein, The trajectory monitoring unit comprises: A frame difference processing subunit is configured to perform frame difference processing on the image scanning results based on the image feedback signal to construct human body bounding box parameters; An identifier allocation subunit is configured to allocate an identity identifier to the human body bounding box parameters to determine the human body contour feature data; A vector calculation subunit is configured to perform motion vector calculation based on the human body contour feature data to obtain human body motion trajectory data, and add the human body motion trajectory data to the second monitoring data set.

6. The radar and vision fusion based multi-functional health monitoring and warning system as claimed in claim 1, wherein, The risk analysis module comprises: A first risk analysis unit is configured to upload the physiological abnormality determination result to a cloud platform for risk analysis when the first early warning instruction triggers the early warning mechanism to generate a first abnormality risk level; A first early warning unit is configured to send the physiological early warning information to a remote terminal according to the first abnormality risk level; A second risk analysis unit is configured to upload the behavior abnormality determination result to the cloud platform for risk analysis when the second early warning instruction triggers the early warning mechanism to generate a second abnormality risk level; A second early warning unit is configured to send the behavior early warning information to the remote terminal according to the second abnormality risk level. The third risk analysis unit is configured to upload the physiological abnormality determination result and the behavior abnormality determination result to a cloud platform for risk analysis when the third early warning instruction triggers an early warning mechanism, and generate a third abnormality risk level; The third early warning unit is configured to send the bidirectional early warning information to a remote terminal according to the third abnormality risk level.

7. The radar and vision fusion based multi-functional health monitoring and warning system as claimed in claim 1, wherein, The risk analysis module comprises: The level determination unit is configured to call a historical abnormality event set in a target monitoring area, perform causal analysis based on the historical abnormality event set, identify a risk level label for the historical abnormality event set according to an analysis result, and determine an abnormality risk level list; The weight optimization unit is configured to perform weight optimization according to the abnormality risk level list, and construct a gradient boosting decision tree; The decision tree embedding unit is configured to embed the gradient boosting decision tree into a cloud platform for risk analysis.

8. The radar and vision fusion based multi-functional health monitoring and warning system as claimed in claim 1, wherein, The risk analysis module further comprises: The response recording unit is configured to record response processing of the remote terminal receiving the early warning information, and generate a response record report; The processing efficiency determination unit is configured to analyze a processing result of the remote terminal by traversing the response record report, and generate a terminal processing efficiency; The mapping relationship determination unit is configured to perform mapping analysis on the abnormality risk level and the early warning information, and determine a risk-early warning mapping relationship; The early warning mode determination unit is configured to dynamically adjust the risk-early warning mapping relationship according to the terminal processing efficiency, generate a risk level score, and replace an early warning mode for the remote terminal according to the risk level score.

Images