Human storage detection system and method based on robot multi-sensor cooperation

Abstract

The invention relates to a human storage detection system and method based on robot multi-sensor cooperation. The human body detection system comprises a robot, a data acquisition module and a robot operating system, the robot operating system is arranged in the robot, the data acquisition module comprises a visual sensor, a laser radar sensor and a millimeter wave radar sensor, the visual sensor is arranged at the head of the robot, and the laser radar sensor is arranged at the tail of the robot. The millimeter wave radar sensor is arranged at the body part of the robot, the laser radar sensor is arranged at the bottom of the robot, and the visual sensor, the laser radar sensor and the millimeter wave radar sensor are all connected with the robot operating system. Through data fusion analysis of multiple sensors, the system can more accurately and comprehensively monitor and understand human body activities and physiological conditions thereof, and the monitoring accuracy and efficiency are improved.

Description

Technical Field

[0001] The present application relates to the technical field of human body monitoring, and in particular, to a system and method for detecting human presence based on robot multi-sensor collaboration. Background Art

[0002] In the current home health monitoring solutions, millimeter wave radar technology is used to detect the state of human presence. For example, the existing AI robot for health stewards uses millimeter radar waves to detect human presence, including recognition of standing, walking, sleeping, falling, heartbeat and breathing; and uses laser radar waves to detect obstacle avoidance, including recognition of pedestrians, furniture, obstacles, etc. However, the existing millimeter radar waves need a height difference to realize the human presence detection function. The detection method adopted is that the millimeter radar wave equipment is usually placed at a high place. After the user falls, the height difference is judged to determine whether the user has fallen. Its scope of application is relatively limited; and laser radar waves are mostly used for vehicle-mounted use and have not been widely promoted on robots.

[0003] However, the limitations of existing technologies lead to false positives and missed positives in the monitoring of human body conditions, and it is difficult to achieve accurate identification and all-round monitoring of each person. Summary of the invention

[0004] Based on the above technical problems, the present application aims to provide a human presence detection system and method based on robot multi-sensor collaboration to solve at least one of the above technical problems.

[0005] In a first aspect, the present application provides a human presence detection system based on robot multi-sensor collaboration, the human presence detection system comprising a robot, a data acquisition module and a robot operating system, the robot operating system being built in the robot, the data acquisition module comprising a visual sensor, a laser radar sensor and a millimeter wave radar sensor, the visual sensor being arranged on the head of the robot, the millimeter wave radar sensor being arranged on the body of the robot, the laser radar sensor being arranged on the bottom of the robot, the visual sensor, the laser radar sensor and the millimeter wave radar sensor being all connected to the robot operating system;

[0006] The visual sensor is used to collect image data of the target area in real time when the robot is walking and send the image data to the robot operating system;

[0007] The laser radar sensor is used to collect point cloud data of the target area in real time when the robot is walking and send the point cloud data to the robot operating system;

[0008] The millimeter wave radar sensor is used to collect human physiological data of a target area in real time when the robot is walking and send the human physiological data to the robot operating system;

[0009] The robot operating system is used to fuse and analyze the image data, the point cloud data and the human physiological data, and make decisions based on the analysis results to control the robot response.

[0010] In some embodiments of the present application, the fusion analysis of the image data, the point cloud data and the human physiological data includes:

[0011] Performing filtering processing on the image data, the point cloud data and the human physiological data respectively to obtain filtered image data, filtered point cloud data and filtered human physiological data;

[0012] Building a three-dimensional model based on the filtered point cloud data to obtain an environment model and a user model;

[0013] Based on the environment model and the user model, and the filtered image data, a fusion analysis is performed to obtain a first analysis result of the target user;

[0014] A fusion analysis is performed based on the primary analysis result and the filtered human physiological data to obtain a second analysis result of the target user.

[0015] In some embodiments of the present application, the step of constructing a three-dimensional model based on the filtered point cloud data to obtain an environment model and a user model includes:

[0016] Constructing a three-dimensional model of the filtered point cloud data by triangulation or voxelization method, determining a model representing the terrain of the target area, and using the model representing the terrain of the target area as an environment model;

[0017] A model representing the activity state of a human body is determined, and the model representing the activity state of a human body is used as a user model.

[0018] In some embodiments of the present application, the fusion analysis is performed based on the environment model and the user model, and the filtered image data to obtain a first analysis result of the target user, including:

[0019] determining potential problem users based on the environment model and the user model;

[0020] Extracting features from the filtered image data, and determining target users based on the extracted features and the potential problem users;

[0021] Analyze the human presence activity state of the target user to obtain a first analysis result of the target user, wherein the human presence activity state of the target user includes the location of the target user, the posture of the target user, and the interaction state of the target user with the environment.

[0022] In some embodiments of the present application, the fusion analysis is performed based on the environment model and the user model, and the filtered image data to obtain a first analysis result of the target user, including:

[0023] Performing pixel mapping on the environment model and the user model using the processed image data to identify the target user;

[0024] Analyze the human presence activity state of the target user to obtain a first analysis result of the target user, wherein the human presence activity state of the target user includes the location of the target user, the posture of the target user, and the interaction state of the target user with the environment.

[0025] In some embodiments of the present application, performing fusion analysis based on the primary analysis result and the filtered human physiological data to obtain a second analysis result of the target user includes:

[0026] Extracting features from the filtered human physiological data, and determining health information of the target user according to the extracted features;

[0027] A deep fusion analysis is performed on the first analysis result of the target user, the health information of the target user, and the historical data of the target user to obtain a second analysis result of the target user.

[0028] In some embodiments of the present application, the deep fusion analysis of the first analysis result of the target user, the health information of the target user, and the historical data of the target user to obtain the second analysis result of the target user includes:

[0029] Using the historical data of the target user to construct and train a long short-term memory network, wherein the historical data of the target user includes the historical behavior, activity records, historical medical data, and historical physiological data of the target user;

[0030] Predicting the health information of the target user through a trained long short-term memory network to obtain a prediction result;

[0031] A Framingham model is used to perform a multiple regression analysis on the prediction result and the first analysis result of the target user to obtain a second analysis result of the target user.

[0032] The second aspect of the present application provides a method for detecting human presence based on robot multi-sensor collaboration, the method comprising:

[0033] Collect image data, point cloud data and human physiological data of the target area in real time while the robot is walking;

[0034] The image data, the point cloud data and the human physiological data are fused and analyzed, and a decision is made according to the analysis result to control the robot response.

[0035] In a third aspect, the present application provides an electronic device, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the following steps:

[0036] Collect image data, point cloud data and human physiological data of the target area in real time while the robot is walking;

[0037] The image data, the point cloud data and the human physiological data are fused and analyzed, and a decision is made according to the analysis result to control the robot response.

[0038] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the following steps when executed by a processor: collecting image data, point cloud data, and human physiological data of a target area in real time when the robot is walking;

[0039] The image data, the point cloud data and the human physiological data are fused and analyzed, and a decision is made according to the analysis result to control the robot response.

[0040] The technical solution provided in the embodiments of the present application has at least the following technical effects or advantages:

[0041] The human presence detection system based on robot multi-sensor collaboration in each embodiment of the present application realizes the innovative integration of visual sensors, lidar sensors and millimeter-wave radar sensors in robots, realizes low-position human presence detection, facial recognition and in-depth monitoring of biometric characteristics (such as heart rate, blood pressure and sleep) from surface to point. This integrated application is the first of its kind in China and abroad, breaking through the limitations of a single sensor in human monitoring.

[0042] By using laser radar sensors and millimeter wave radar sensors, the system can work stably under different lighting and environmental conditions, enhancing its adaptability and stability in complex environments; through real-time data collection and analysis, the system can monitor the physiological and behavioral status of the target user in real time, which is suitable for rapid response to emergencies and long-term health management. Through multi-sensor data fusion analysis, the system can more accurately and comprehensively monitor and understand human activities and physiological conditions, improving the accuracy and efficiency of monitoring.

[0043] Moreover, in-depth analysis and health prediction based on the user's historical data can provide users with personalized health management and early warning services. In particular, when a user accidentally falls and loses consciousness, the system can serve as the first responder to complete medical alarm assistance, significantly improving the safety of home users and the timeliness of first aid, effectively saving lives.

[0044] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0046] Figure 1 It is a structural schematic diagram of a human presence detection system based on robot multi-sensor collaboration in an exemplary embodiment of the present application;

[0047] Figure 2 is a schematic structural diagram of another human presence detection system based on robot multi-sensor collaboration in an exemplary embodiment of the present application;

[0048] Figure 3 is a schematic diagram comparing properties of a camera and a laser radar wave sensor in an exemplary embodiment of the present application;

[0049] Figure 4 is a schematic diagram of transmitting linear frequency modulation pulses to a patient in an exemplary embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of the steps of a method for detecting human presence based on robot multi-sensor collaboration in an exemplary embodiment of the present application;

[0051] Figure 6 It is a structural schematic diagram of an electronic device provided by an exemplary embodiment of the present application.

[0052] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION

[0053] In the current home health monitoring program, the use of millimeter wave radar technology to detect human status (for example, to determine whether a fall has occurred) is widespread. One of the main applications of this technology is to place the detection equipment at a higher position, such as the wall of a room or next to a light source. This is done to identify different postures of the human body by measuring changes in body height. On the other hand, although lidar technology has been widely used in automotive systems, its application in the field of health monitoring robots is relatively limited. These limitations of current technology may lead to false alarms and missed alarms when monitoring the human body, and it is difficult to accurately and comprehensively monitor each individual.

[0054] Therefore, the present application provides a human presence detection system and method based on robot multi-sensor collaboration, which realizes all-round monitoring of the human presence status and health status of each target user in the target area by fusing and analyzing the image data, point cloud data, and physiological data collected by multiple sensors.

[0055] The present application is further described in detail below in conjunction with the accompanying drawings and several embodiments. It should be understood that the embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It should also be noted that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0056] Example 1

[0057] This embodiment provides a human presence detection system based on robot multi-sensor collaboration. Figure 1 It is a structural diagram of a human presence detection system based on robot multi-sensor collaboration, such as Figure 1 As shown, the human body detection system includes a robot, a data acquisition module and a robot operating system (ROS), the robot operating system is built into the robot, and the data acquisition module includes a visual sensor, a lidar sensor and a millimeter wave radar sensor, such as Figure 2 As shown, the visual sensor is configured on the head of the robot, the millimeter-wave radar sensor is configured on the body of the robot, and the lidar sensor is configured on the bottom of the robot. The visual sensor, the lidar sensor and the millimeter-wave radar sensor are all connected to the robot operating system.

[0058] Specifically, the visual sensor is a camera, and the camera is provided with a video sensor, and the video sensor is a human scene sensor. The millimeter wave radar sensor is a millimeter radar wave chip, and the millimeter radar wave chip is evenly distributed in the circumferential direction on the body part of the robot. The laser radar wave sensor is a laser radar wave chip, and the laser radar wave chip is used to collect point cloud data of a preset environment. Preferably, 1 camera, 3 millimeter radar wave chips and 1 laser radar wave chip are installed on the robot. The human presence detection system also includes an alarm module, and the alarm module is used to issue an alarm message when the human presence detection system detects abnormal data.

[0059] The camera can capture images and video data for identifying faces, postures and other visual features. The camera is set at the front end of the robot to collect photos in a preset environment (such as a family's living room) in real time and obtain image data in the detection environment; the camera is provided with a video sensor, which uses a human scene sensor to perform invisible and non-sensitive security detection by using FP2. When it is necessary to distinguish each person, the camera is called for face recognition. The human scene sensor uses FP2 for invisible and non-sensitive security detection. When it is necessary to distinguish each person, the camera is briefly called for face recognition; the system implements the standard GB 4943.1-2022, and the wireless connection uses Wifi IEEE 802.11 and Bluetooth4.2.

[0060] The millimeter radar wave chip is used to monitor human activities such as heartbeat and sleep in real time. The detection range of each millimeter radar wave chip is 120 degrees, the radial distance is about 8 meters, the data refresh frequency is 0.5 seconds, and 320 grid divisions are managed.

[0061] The laser radar chip is used to collect discrete point cloud data, accurately obtain three-dimensional structure data in the human presence detection environment, thereby realizing environmental modeling, and outputting the distance information of people in the human presence detection environment with high precision to identify their activity paths. The laser radar chip, combined with the data obtained by the camera, can realize the perception of complex and changing environments, enhance the intelligent level and data accuracy of the system's human presence detection, and make motion planning and decision-making safer and more reliable.

[0062] The collection frequency of the human presence detection system based on robot multi-sensor collaboration can be set according to user needs, and the default collection frequency is automatically collected once every hour. In addition, the system is also set with a detection parameter threshold. When the monitoring data reaches the parameter threshold, a new round of detection is automatically started; the parameter threshold includes: obtaining the volume of the behavior in the monitoring environment is higher than 50 decibels; obtaining the user's heartbeat information for 30 seconds; obtaining user behavior information, falling down for a long time, etc. After repeated detection, the data is abnormal, and the system automatically triggers an alarm and sends a help signal to the associated contact, community hospital or clinic hospital. It should be noted that the laser radar sensor is used for ranging and map construction. By emitting a laser beam and receiving the reflected beam to measure the distance, a high-precision 3D environment and object model can be generated, which is suitable for accurately monitoring objects and human postures in space. In this application, the laser radar sensor is used to collect point cloud data of the target area in real time when the robot is walking and send the point cloud data to the robot operating system.

[0063] Millimeter-wave radar sensors use radar waves in the millimeter-wave frequency band to measure distance, speed, and angle. Millimeter-wave radar is particularly suitable for monitoring human physiological signals (such as heartbeat, breathing) and motion status in harsh visual environments (such as smoke, night, or strong light environments) because it can penetrate some visual obstructions. In this application, at least three millimeter-wave radar sensors are used, and these sensors are arranged around the body of the robot to collect human physiological data of the target area in real time when the robot is walking and send the human physiological data to the robot operating system, so that more comprehensive and detailed human physiological data can be collected.

[0064] The target area can be different rooms in the home, such as the living room. When the robot is navigating indoors, the robot operating system ROS will decide to control the robot to plan the optimal path according to the environment, including going straight, bypassing static or dynamic obstacles, walking carefully in narrow passages, and performing turns when necessary, whether it is a sharp turn, an arc path or other complex walking patterns. In this way, the robot can recognize and adapt to various geometric shapes and obstacles in the environment to achieve efficient and safe navigation.

[0065] The visual sensor is configured on the head of the robot. For example, the camera is connected to the robot operating system through a USB interface or a similar high-speed data interface to achieve efficient image data transmission and processing. The lidar sensor is configured at the bottom of the robot. The sensor mainly uses a network interface and performs data communication based on the UDP protocol to cope with the large amount of data generated by the lidar. The network configuration of the lidar does not require the installation of a special driver, but the user needs to configure the network information according to the product manual to obtain data normally. The millimeter wave radar sensor is configured on the body of the robot. The connection with the robot operating system can also use a network interface or other interfaces suitable for high-speed data transmission to support real-time physiological signal and motion data collection.

[0066] The visual sensor, laser radar sensor and millimeter wave radar sensor in the detection system are connected to the robot operating system through corresponding interfaces. Figure 2 As shown in the figure, the entire detection system also works with the Zigbee gateway through the Wifi local area network to achieve wireless communication and data exchange. The robot operating system supports the implementation of standard GB4943.1-2022 and uses Wifi's IEEE 802.11 and Bluetooth 4.2 for wireless connection. For situations where invisible and non-perceptible security detection is required, the millimeter wave radar sensor uses FP2 technology. FP2 technology designs data encryption, non-perceptual data transmission or specific communication protocols to enhance security and privacy protection, especially when performing sensitive operations such as face recognition.

[0067] It can be understood that this embodiment can realize the planning and calling of cameras, lidar waves and millimeter radar waves, effectively integrating the functions of each part to realize overall decision-making planning; establishing a terrain surface model through lidar waves, and monitoring the activities of people in the environment in real time; cooperating with the camera to realize personnel identification, and at the same time dispatching millimeter radar waves to monitor the activities of people in the environment in an all-round manner, effectively enhancing the adaptability and stability of the health robot system in various complex environments, and reducing the possibility of data distortion and loss during the monitoring process; in addition, this system can more accurately and comprehensively realize the purpose of real-time monitoring of the physiological and behavioral status of the target user, and is suitable for rapid response to emergencies and long-term health management.

[0068] In short, the combination of these three sensors in health monitoring robots can greatly improve the accuracy and comprehensiveness of human monitoring. Visual sensors obtain detailed visual information, lidar sensors provide accurate spatial positioning and environmental modeling, and millimeter-wave radar sensors increase the monitoring capability in visually restricted environments, together realizing the ability to accurately identify individuals and conduct all-round health monitoring, effectively reducing false positives and missed positives. The detection system also realizes the innovative integration of visual sensors, lidar sensors, and millimeter-wave radar sensors in robots, realizing low-position human presence detection, facial recognition, and in-depth monitoring of biometrics (such as heart rate, blood pressure, and sleep) from surface to point. This integrated application is the first of its kind at home and abroad, breaking through the limitations of a single sensor in human monitoring.

[0069] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0070] Example 2

[0071] This embodiment provides a human presence detection system based on robot multi-sensor collaboration. Figure 1 and Figure 2 Based on the specific structure of the human presence detection system based on robot multi-sensor collaboration shown in Example 1, this example provides a working method of the detection system. The above-mentioned robot refers to a health robot.

[0072] It can be understood that the laser radar waves on the health robot sense and model the user's indoor environment. The laser radar sets the scanning frequency, laser beam angle and scanning speed according to the input monitoring environment information and monitoring mode. By frequently emitting laser beams and receiving reflected signals, it can obtain sufficiently accurate point cloud data in a short time. During the point cloud data collection process, it is necessary to ensure the stability of the position and angle of the health robot to avoid data collection errors or repeated scanning.

[0073] A continuous terrain surface model is obtained by processing the discrete point cloud data collected by the lidar sensor. First, the original data needs to be dedusted to eliminate some noise points that do not meet the requirements. Secondly, the point cloud data is Gaussian filtered and median filtered to eliminate irregular fluctuations and unevenness in the data. Then, data fitting and surface fitting are performed to obtain a smoother and more continuous terrain surface.

[0074] The terrain surface model constructed by the system depends on the division of the collected point cloud data, and can be set as a triangular mesh or a voxel mesh. Specifically, the point cloud data is divided into different areas through clustering methods, and each area is further processed and modeled. The triangular mesh method connects the point cloud data into a triangular mesh to form a continuous terrain surface model. The voxel mesh method divides the point cloud data into regular voxel units and interpolates each voxel to obtain a smoother and more continuous terrain surface model.

[0075] After completing terrain surface modeling, the modeling results need to be evaluated and checked for accuracy. The error between the modeling results and the actual data can be calculated by comparing them with the actual terrain data. At the same time, visual inspection can be performed to analyze the modeling results to determine the accuracy and reliability of the modeling results.

[0076] After confirming the terrain surface modeling, the next step is user modeling. User modeling is to use the visual judgment of the camera. Based on the first model (ground model) data, the first model data includes terrain information (Terrain) and user information (User), and the pre-trained model of the lidar wave is applied to the robot's vision to judge whether there is an arc ground error in the terrain model and user model modeled by the lidar wave, and which user is the face recognition positioning of each user; the second model (user model) data is obtained, and the second model data includes location information (Location) and user information (User).

[0077] Figure 3 This is a schematic diagram comparing the properties of camera and lidar wave sensors, where the brown line represents the camera and the red line represents the lidar wave. Figure 3 It can be seen that the camera can output high-density image information, but the distance estimation ability and robustness are poor; the lidar wave sensor can directly output high-precision distance information, and the three-dimensional structure data is clear, but the point cloud is sparse. It can also be seen from the radar map that the two sensors are highly complementary. By combining fine-grained camera images and remote detection capabilities, high-resolution depth maps and three-dimensional representations can be generated, which is much richer than any single sensor can provide. Compared with a single rich sensor reading, the reasonable fusion of vision-lidar data will greatly promote the perception of complex and changing environments, improve the intelligence level of perception systems, reduce uncertainty, and make motion planning and decision-making safer and more reliable.

[0078] The principle of the fusion of laser radar wave sensor and camera is as follows: first, the relative position relationship parameters of the laser radar and camera are determined by calibrating the laser radar and camera; then the three-dimensional space where the laser point cloud is located is mapped with the two-dimensional space of the camera image, so that the detected object in the image obtains its depth and position information in the three-dimensional space, as well as the three-dimensional space element composition information; then the ground part in the point cloud is extracted and separated from the laser point cloud; then, using the three-dimensional information of the detected target, the detected object in the laser point cloud is processed, the point cloud noise is filtered out, the point cloud matching relationship is further optimized, and the point cloud of the detected object is densified; then the point cloud parts of the detected object are separated one by one, and the three-dimensional bounding box where its point cloud is located is calculated; finally, the vertices of the three-dimensional bounding box of the detected object are projected into the camera image space to obtain its pseudo three-dimensional bounding box information.

[0079] The specific process of laser radar wave sensor and camera fusion processing includes: first, before using the laser radar wave sensor and camera, it is necessary to pre-calculate the relative position relationship between the laser radar wave sensor and the camera. The process of determining these position relationships is usually called the calibration process. After the calibration is completed, the data collected by the laser radar wave sensor and the camera are fused together. Secondly, data fusion is performed. The object of data processing here refers to the image and laser point cloud data collected by using the laser radar wave sensor and the camera. According to the calibrated parameters, the correspondence between the spatial points in the laser point cloud and the pixel points in the image is calculated, and the data collected by different sensors are fused together. The fusion processing can make full use of the data features collected by different sensors. For example, the color curvature information collected by the camera, the spatial coordinates and reflectivity information collected by the laser radar wave sensor, etc. After that, the preliminary fused data is clustered, because the preliminary fused data may contain some error information, which is usually caused by the matching error caused by the difference in the position of the sensor. Therefore, through the clustering processing based on feature information, some errors in the fusion processing can be corrected.

[0080] Next, data optimization is performed. When generating a three-dimensional space detection frame, the size and position of the bounding box are optimized based on object features, spatial position relationships between objects, environmental information and other data to obtain a more accurate three-dimensional space detection frame. When generating a three-dimensional space detection frame, the size and position of the bounding box can be optimized based on object features, spatial position relationships between objects, environmental information and other data to obtain a more accurate three-dimensional space detection frame. This application uses a lidar sensor to perceive the object being detected, which can obtain centimeter-level depth information and effectively reduce depth estimation errors. At the same time, a complete three-dimensional structure is established: Due to the difference in position between the lidar and the camera, and the laser point cloud has three-dimensional information, it is possible to construct the three-dimensional structural information of the object being detected.

[0081] Then combine the second model data output from the previous step with the millimeter radar wave on the robot to obtain the third model (i.e., the physiological model of the target user) data. The third model data includes monitoring information (Monitoring) and user information (User). This step calls the function of the millimeter radar wave, with indoor positioning at the cm level, MIMO virtual aperture super-resolution point cloud imaging, and intelligent recognition of the orientation and angle, distance and amplitude of the person. Accurately extract and intelligently identify micro-motion information such as breathing, heartbeat, sleep, and complex physiological characteristic signals. Then achieve accurate monitoring of human presence detection for each user in the room. Output the fourth model data, which includes user number information (UserNumber) and physiological status information (PhysioligicalStatus).

[0082] The advantage of the short wavelength of millimeter radar waves is high accuracy. Millimeter wave radars with a frequency of 60 or 77 GHz (corresponding to a wavelength in the 4 mm range) can detect movements as short as less than 1 mm. Figure 3 It is a schematic diagram of using millimeter radar waves to transmit linear frequency modulated pulses to the chest area of ​​a patient. Due to the movement of the chest, the reflected signal is phase modulated. The modulation covers all components of the motion, including the motion caused by heartbeat and breathing, such as Figure 4 As shown in the figure, the results of modulation and analysis of the reflected signal are that the respiration rate is 26 and the heart rate is 80. The millimeter radar wave sends multiple linear frequency modulation pulses according to the predetermined time interval. Each pulse performs a distance fast Fourier transform (FFT) and selects the distance file corresponding to the chest position of the person. Each linear frequency modulation pulse records the signal phase in the selected distance file. The phase change is calculated and the speed is obtained. The obtained speed still includes all motion components. The obtained speed is spectrally analyzed by performing Doppler FFT to resolve various component data.

[0083] Finally, a multiple regression analysis is performed on the physiological data of the target user based on the Framingham model. First, the Framingham model requires a variety of health indicators as input, which can be obtained from the fourth model data, such as the risk factors required by the Framingham model, such as the age, gender, blood pressure, cholesterol level, etc. of the target user. Then the regression equation of the Framingham model is applied to the data of the target user, that is, the health risk score of the target user is calculated using the Framingham model, which reflects the probability of the target user suffering from cardiovascular disease in the future. According to the output of the Framingham model, a risk threshold is set. When the risk score of the target user exceeds this threshold, the system will automatically issue a health warning, which can help the target user take preventive measures in time, such as changing lifestyle or seeking medical help. Optionally, the risk score and warning information of the Framingham model are integrated into the health report of the target user, and the system interacts with the user, such as sending a report to the target user's mobile phone. It can be understood that after modeling the health model for each individual user, not only the user's daily health data model is generated, but also the data is combined with other collected health data of the robot to complete a more comprehensive and extensive user-tailored health model. At the same time, through the Framingham model, the health trends of each user and the risk factors related to the occurrence of related diseases can be analyzed. When the danger threshold is reached, an early warning can be issued and a disease risk prediction model can be established in a forward-looking manner.

[0084] It can be seen that the human presence detection system based on robot multi-sensor collaboration realizes the innovative integration of visual sensors, lidar sensors and millimeter-wave radar sensors in robots, and realizes low-position human presence detection, facial recognition and in-depth monitoring of biometric characteristics (such as heartbeat, blood pressure and sleep) from surface to point. This integrated application is the first of its kind at home and abroad, breaking through the limitations of a single sensor in human monitoring.

[0085] In a specific implementation method, when fusing and analyzing image data, point cloud data and human physiological data, the image data, point cloud data and human physiological data are first filtered respectively to obtain filtered image data, filtered point cloud data and filtered human physiological data; secondly, a three-dimensional model is constructed based on the filtered point cloud data to obtain an environmental model and a user model; then, a fusion analysis is performed based on the environmental model and the user model, as well as the filtered image data, to obtain a first analysis result of the target user; finally, a fusion analysis is performed based on the primary analysis result and the filtered human physiological data to obtain a second analysis result of the target user.

[0086] Filtering is aimed at removing noise and outliers from data and improving data quality. Different filtering techniques can be used for point cloud data, image data, and human physiological data. For example, statistical analysis filtering (such as moving average or Gaussian filtering) is used for point cloud data, edge-preserving filtering algorithms (such as bilateral filtering or non-local mean filtering) are used for image data, and bandpass filters can be applied to human physiological data to remove non-target frequency components.

[0087] Specifically, when constructing a three-dimensional model based on the filtered point cloud data to obtain the environment model and the user model, the three-dimensional model is first constructed by triangulating the filtered point cloud data through a triangulation or voxelization method, and a model representing the terrain of the target area is determined, and the model representing the terrain of the target area is used as the environment model; then a model representing the existence and activity state of the human body is determined, and the model representing the existence and activity state of the human body is used as the user model.

[0088] In a possible implementation, a fusion analysis is performed based on the environment model and the user model, and the filtered image data to obtain a first analysis result of the target user, including the following steps.

[0089] S100: Determine potential problem users based on the environment model and the user model.

[0090] Specifically, the lidar sensor can efficiently collect accurate point cloud data by determining the scanning frequency, laser beam angle, and scanning speed. Subsequently, the robot operating system ROS removes and filters these point cloud data, and then constructs a continuous terrain surface model through triangulation or voxelization methods, including environmental models and user models according to the objects involved in the model. This process involves not only accurate perception of the environment, but also preliminary identification of potential problem users, such as identifying those users who may be on unstable ground or close to potential obstacles by analyzing the terrain model.

[0091] For example, the robot uses the lidar sensor at the bottom to collect point cloud data in the room, which includes the position and shape of the ground, furniture, walls, etc. ROS uses the point cloud data to build a three-dimensional model of the environment through triangulation or voxelization. This environmental model represents the static world around the robot, including the curved ground, the layout of furniture, and so on. ROS uses the environmental model to identify the exact position and shape of the curved ground, and then calculates an optimal path to control the robot to walk at an appropriate angle and speed to avoid slipping or deviating from the path.

[0092] For another example, if the robot encounters an obstacle during autonomous navigation (such as autonomous navigation once every hour), and the user model determines that the obstacle is a user, ROS controls the robot to adjust its path to bypass the user, and uses human physiological data for fusion analysis. For example, if a family member happens to fall, the family member can be identified as a potential problem user. If a dynamic obstacle such as a pet is encountered, ROS controls the robot to adjust its path in real time to ensure efficient navigation and avoid collisions.

[0093] S200: extracting features from the filtered image data, and determining target users based on the extracted features and potential problem users.

[0094] The potential problem users determined in step S100 may not be clearly identified. That is, ROS analyzes the point cloud data collected by the lidar sensor to determine the approximate geographical environment range (environmental model) and lock in users who may be in health danger (user model) at a macro level. For example, the area where a person fell needs to be further confirmed in combination with the filtered image data.

[0095] The purpose of step S200 is to identify and analyze image content, such as object detection, scene understanding, and face recognition. Optionally, a convolutional neural network (CNN) is used to extract image features and perform classification or regression analysis. By combining the potential problem users identified by the environment model and the user model, face recognition is performed to determine the target user, that is, which specific family member can be seen clearly.

[0096] S300, analyzing the human presence activity state of the target user to obtain a first analysis result of the target user, wherein the human presence activity state of the target user includes the location of the target user, the posture of the target user, and the interaction state between the target user and the environment.

[0097] In a specific implementation, assuming that ROS recognizes that a family member is close to a slip-prone area in the living room, such as a freshly cleaned and slippery floor, or someone has fallen to the ground, it will be based on a detailed analysis of the target user's human presence and activity status, including the target user's specific location, their body posture, and their interaction with the surrounding environment. For example, the first analysis result is: the target user is located on the floor near the center of the living room, specifically 2 meters away from the north wall and 3 meters away from the east wall; the target user's posture is lying flat on the ground, with the body slightly tilted to the left and the arms apart; the target user is close to the slippery floor area and there are no obstacles around. In this way, based on such analysis results, ROS can further process the human physiological data collected by the millimeter-wave radar sensor and decide whether to take action based on the fusion analysis results.

[0098] As a transformable implementation method, a fusion analysis is performed based on the environment model and the user model, and the filtered image data to obtain a first analysis result of the target user. The following steps can also be performed.

[0099] S101 : Utilize the processed image data to perform pixel mapping on the environment model and the user model to identify the target user.

[0100] The above pixel mapping actually establishes a pixel-level correspondence between the feature information (such as edges, shapes, etc.) of the environment model and the user model and the processed image data. The Robot Operating System, or ROS, maps the processed image data to the environment model, enhancing the realism of the model and making them closer to the appearance of the real world. Mapping the image data to the user model can help identify the human activity form in the user model, thereby accurately locating and identifying the target user. For example, by pixel mapping the user model, the robot can more accurately identify the characteristics of the target user, such as clothing color, possible posture, and relationship with the surrounding environment. This method greatly improves the accuracy of target recognition and the depth of environmental understanding.

[0101] S201. Analyze the human presence activity state of a target user to obtain a first analysis result of the target user, wherein the human presence activity state of the target user includes a location of the target user, a posture of the target user, and an interaction state between the target user and the environment.

[0102] If the robot encounters a user falling during autonomous navigation, the target user and his / her interaction with the surrounding environment can be determined through image data mapping, i.e., the environment model and user model with enhanced visual effects. For example, the first analysis result is: the target user is located on the floor near the center of the living room, 1 meter away from the north wall and 1 meter away from the east wall; the target user is lying flat on the ground, with the body slightly tilted to the right and the arms spread apart; the target user is close to the slippery floor area, and there is a standing clothes hanger next to him / her.

[0103] The above implementation method enhances the accuracy and realism of the environment and user model through pixel mapping, making the identification of the target user more accurate. At the same time, by analyzing the human body activity status of the target user, the robot can have a deeper understanding of the user's specific situation and needs, and can also provide more effective support for subsequent further fusion analysis.

[0104] In a specific implementation method, a fusion analysis is performed based on the primary analysis results and the filtered human physiological data to obtain a second analysis result of the target user, including: extracting features from the filtered human physiological data, and determining the health information of the target user based on the extracted features; performing a deep fusion analysis on the first analysis result of the target user, the health information of the target user, and the historical data of the target user to obtain a second analysis result of the target user.

[0105] Specifically, a deep fusion analysis is performed on the first analysis result of the target user, the health information of the target user, and the historical data of the target user to obtain a second analysis result of the target user, including:

[0106] A long short-term memory network is constructed and trained using the historical data of the target user, wherein the historical data of the target user includes the historical behavior, activity records, historical medical data and historical physiological data of the target user; the health information of the target user is predicted by the trained long short-term memory network to obtain a prediction result; and a multivariate regression analysis is performed on the prediction result and the first analysis result of the target user using the Framingham model to obtain a second analysis result of the target user.

[0107] During implementation, it is necessary to build and train a long short-term memory network (LSTM). First, collect extensive historical data of the target user, including but not limited to the user's historical behavior patterns, activity records, medical records, and physiological data (such as heart rate, blood pressure, etc.). These data provide a rich foundation for building a personalized health prediction model. Then use the collected historical data to build a long short-term memory network. LSTM is a special recurrent neural network, or RNN, that is particularly suitable for processing and predicting long-term dependency problems in time series data. By training LSTM, the model can learn to identify health trends and potential risks from the user's historical behavior and physiological data, and then use the trained LSTM network to predict the future health status of the current target user based on his or her health information. The prediction content includes the target user's possible health risks, the development trend of potential diseases, and the changing trend of the user's health status, etc.

[0108] Furthermore, the Framingham model is used to perform a multivariate regression analysis on the predicted results of the above-mentioned trained LSTM network and the first analysis results of the target user, and the second analysis results of the target user are obtained, which realizes the in-depth monitoring of the human body's existence status from the surface to the point. The Framingham model is a cardiovascular disease risk model, a tool used to assess the risk of cardiovascular disease in an individual in the future. The Framingham model takes into account multiple risk factors, based on historical medical data and historical physiological data, such as the target user's age, gender, blood pressure, cholesterol level, etc., and finally combines the user's first analysis results to perform a multivariate regression analysis.

[0109] Combining the future health information predicted by LSTM, the user's first analysis results, and the Framingham model score, a multivariate regression analysis is performed, which not only takes into account the user's physiological parameters and environmental interactions, but also combines the user's long-term health trends and risk factors to obtain a comprehensive, future-oriented health prediction and risk assessment. It can be seen that the second analysis result obtained by the fusion analysis provides the target user with a comprehensive health risk assessment, including possible health challenges, risk factors that need attention, and personalized health recommendations or preventive measures based on the prediction results. For example, if the prediction results show that the user has a higher risk of cardiovascular disease, the second analysis results may recommend changing lifestyle habits, undergoing regular medical examinations, or seeking medical advice and intervention when necessary.

[0110] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0111] Example 3

[0112] This embodiment provides a method for detecting human presence based on robot multi-sensor collaboration. Figure 5 As shown, the method includes the following steps.

[0113] S1. Collect image data, point cloud data and human physiological data of the target area in real time while the robot is walking.

[0114] Specifically, the camera installed on the head of the robot can collect real-time images of the target area (such as a room in the room, such as an activity hall on the basement floor of a villa), including environmental features, visual information of people present, etc. The image data provides intuitive visual information of the scene, such as human position, posture and environmental layout. The laser radar sensor installed at the bottom of the robot collects point cloud data by scanning to capture the three-dimensional structural information of the target area. The point cloud data strengthens the understanding of the three-dimensional shape and spatial position of the environment and objects (including the human body). The millimeter-wave radar sensor set around the body of the robot collects human physiological data, that is, real-time monitoring and recording of human physiological signals, such as heart rate, breathing rate, etc. These data provide important clues about the current health status of the human body and possible physiological stress responses.

[0115] S2. Fusion analysis of the image data, the point cloud data and the human physiological data, and making decisions based on the analysis results to control the robot response.

[0116] Specifically, all data are first processed for dedusting and filtering, and then the image data and point cloud data are fused and analyzed to build a detailed environment and human body model. Through this analysis, the position, posture, and interaction status of the target user with the environment can be identified, that is, the preliminary analysis results are obtained. Then, the human physiological data after dedusting and filtering are combined with the preliminary analysis results for further fusion analysis. This process involves the use of advanced algorithms, such as deep learning and pattern recognition technology, to understand and predict the health status and potential safety risks of the human body. The analysis process is similar to that of Example 2, and will not be repeated here.

[0117] Based on the results of the fusion analysis, the system will make decisions to guide the robot's response behavior. These decisions may include providing necessary support and assistance to the user, warning the user to avoid potential dangers, or adjusting the robot's action plan to better serve the user. For example, if the analysis results show that a user is at risk of falling, the robot may be instructed to provide an immediate sound warning, or move to the user's location to provide physical support.

[0118] The above-mentioned human presence detection method based on robot multi-sensor collaboration can provide users with personalized health management and early warning services by fusing and analyzing the image data, the point cloud data and the human physiological data. In particular, the system can serve as the first responder to complete medical alarm assistance when the user accidentally falls to the ground and loses consciousness, thereby significantly improving the safety of home users and the timeliness of first aid, and effectively saving lives.

[0119] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0120] Example 4

[0121] Please refer to the following Figure 6 , this embodiment provides a schematic diagram of an electronic device. Figure 6 As shown, the electronic device 2 includes: a processor 200, a memory 201, a bus 202 and a communication interface 203, and the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and when the processor 200 runs the computer program, it executes any one of the human presence detection methods based on robot multi-sensor collaboration in the embodiments of the present application.

[0122] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0123] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving the execution instructions. The human presence detection method based on robot multi-sensor collaboration disclosed in any implementation of the above-mentioned embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.

[0124] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the human presence detection method based on robot multi-sensor collaboration in combination with its hardware.

[0125] This embodiment also provides a computer-readable storage medium corresponding to the method for detecting human presence based on robot multi-sensor collaboration provided in the aforementioned embodiment, on which a computer program is stored, and when the computer program is executed by the processor, the method for detecting human presence based on robot multi-sensor collaboration provided in any of the aforementioned embodiments will be executed. In addition, examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which will not be described one by one here.

[0126] In addition, an embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements any one of the human presence detection methods based on robot multi-sensor collaboration in the aforementioned embodiments.

[0127] Those skilled in the art will appreciate that the various component embodiments of the present application may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art will appreciate that a microprocessor or digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components of the virtual machine creation device according to the embodiments of the present application.

[0128] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A human presence detection system based on robot multi-sensor collaboration, characterized in that: The human body detection system includes a robot, a data acquisition module and a robot operating system, wherein the robot operating system is built in the robot, the data acquisition module includes a visual sensor, a laser radar sensor and a millimeter wave radar sensor, the visual sensor is configured on the head of the robot, the millimeter wave radar sensor is configured on the body of the robot, the laser radar sensor is configured on the bottom of the robot, and the visual sensor, the laser radar sensor and the millimeter wave radar sensor are all connected to the robot operating system; The visual sensor is used to collect image data of the target area in real time when the robot is walking and send the image data to the robot operating system; The laser radar sensor is used to collect point cloud data of the target area in real time when the robot is walking and send the point cloud data to the robot operating system; The millimeter wave radar sensor is used to collect human physiological data of a target area in real time when the robot is walking and send the human physiological data to the robot operating system; The robot operating system is used to fuse and analyze the image data, the point cloud data and the human physiological data, and make decisions based on the analysis results to control the robot response.

2. The human presence detection system based on robot multi-sensor collaboration according to claim 1 is characterized in that: The fusion analysis of the image data, the point cloud data and the human physiological data includes: Performing filtering processing on the image data, the point cloud data and the human physiological data respectively to obtain filtered image data, filtered point cloud data and filtered human physiological data; Building a three-dimensional model based on the filtered point cloud data to obtain an environment model and a user model; Based on the environment model and the user model, and the filtered image data, a fusion analysis is performed to obtain a first analysis result of the target user; A fusion analysis is performed based on the primary analysis result and the filtered human physiological data to obtain a second analysis result of the target user.

3. The human presence detection system based on robot multi-sensor collaboration according to claim 2 is characterized in that: The step of constructing a three-dimensional model based on the filtered point cloud data to obtain an environment model and a user model includes: Constructing a three-dimensional model of the filtered point cloud data by triangulation or voxelization method, determining a model representing the terrain of the target area, and using the model representing the terrain of the target area as an environment model; A model representing the activity state of a human body is determined, and the model representing the activity state of a human body is used as a user model.

4. The human presence detection system based on robot multi-sensor collaboration according to claim 3 is characterized in that: The step of performing fusion analysis based on the environment model and the user model and the filtered image data to obtain a first analysis result of the target user includes: determining potential problem users based on the environment model and the user model; Extracting features from the filtered image data, and determining target users based on the extracted features and the potential problem users; Analyze the human presence activity state of the target user to obtain a first analysis result of the target user, wherein the human presence activity state of the target user includes the location of the target user, the posture of the target user, and the interaction state of the target user with the environment.

5. The human presence detection system based on robot multi-sensor collaboration according to claim 3 is characterized in that: The step of performing fusion analysis based on the environment model and the user model and the filtered image data to obtain a first analysis result of the target user includes: Performing pixel mapping on the environment model and the user model using the processed image data to identify the target user; Analyze the human presence activity state of the target user to obtain a first analysis result of the target user, wherein the human presence activity state of the target user includes the location of the target user, the posture of the target user, and the interaction state of the target user with the environment.

6. The human presence detection system based on robot multi-sensor collaboration according to claim 4 or 5, characterized in that: The step of performing fusion analysis based on the primary analysis result and the filtered human physiological data to obtain a second analysis result of the target user includes: Extracting features from the filtered human physiological data, and determining health information of the target user according to the extracted features; A deep fusion analysis is performed on the first analysis result of the target user, the health information of the target user, and the historical data of the target user to obtain a second analysis result of the target user.

7. The human presence detection system based on robot multi-sensor collaboration according to claim 6 is characterized in that: The deep fusion analysis of the first analysis result of the target user, the health information of the target user, and the historical data of the target user to obtain the second analysis result of the target user includes: Using the historical data of the target user to construct and train a long short-term memory network, wherein the historical data of the target user includes the historical behavior, activity records, historical medical data, and historical physiological data of the target user; Predicting the health information of the target user through a trained long short-term memory network to obtain a prediction result; A Framingham model is used to perform a multiple regression analysis on the prediction result and the first analysis result of the target user to obtain a second analysis result of the target user.

8. A method for detecting human presence based on robot multi-sensor collaboration, characterized in that: The method comprises: Collect image data, point cloud data and human physiological data of the target area in real time while the robot is walking; The image data, the point cloud data and the human physiological data are fused and analyzed, and a decision is made according to the analysis result to control the robot response.

9. An electronic device, comprising a memory and a processor, characterized in that: The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor executes the human presence detection method based on robot multi-sensor collaboration as described in claim 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for detecting human presence based on robot multi-sensor collaboration as claimed in claim 8 is implemented.

Images