A multi-sensor fusion four-dimensional integrated monitoring intelligent collaborative system and an implementation method thereof

Abstract

The application belongs to the technical field of intelligent robot health care, and provides a four-dimensional integrated monitoring intelligent collaborative system based on multi-sensor fusion and an implementation method thereof, wherein the system comprises a cloud platform, a data acquisition terminal, a simulation robot and a mobile bearing terminal; the cloud platform is configured with a general large model and a customized model, the general large model is used for data fusion, natural language understanding and general decision-making, and the customized model is used for health monitoring and human-computer interaction; through the cooperation of the general large model and the customized model, the application realizes full-process data transmission, instruction issuing, state feedback and multi-dimensional collaborative control, guarantees the operation synchronization of each component, effectively solves the problems of single function and lack of collaborative control existing in the prior art, and improves the intelligent level and response timeliness of health care services based on the simulation robot.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot health care technology. Specifically, it relates to a four-dimensional integrated monitoring intelligent collaborative system with multi-sensor fusion and its implementation method. Background Technology

[0002] With the increasing trend of population aging, the demand for health and wellness services is growing. Traditional health and wellness assistive devices have limited functions; for example, smart mattresses can only monitor physiological data, and monitoring cameras can only provide video footage. They operate independently and cannot form effective data linkages. Some health and wellness robots on the market typically only have simple voice interaction or remote monitoring functions. Their decision-making capabilities are limited, and they lack coordination with other smart terminals (such as wheelchairs and monitoring devices). They cannot adjust their service strategies and behaviors according to the user's real-time status and environmental changes, resulting in a poor user experience and untimely service response. Summary of the Invention

[0003] The purpose of this invention is to provide a four-dimensional integrated intelligent collaborative monitoring system based on multi-sensor fusion, so as to solve the technical problems existing in the prior art.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A multi-sensor fusion four-dimensional integrated intelligent collaborative monitoring system includes a cloud platform, a data acquisition terminal communicating with the cloud platform, a simulation robot, and a mobile carrier terminal. The cloud platform is configured with a general-purpose large model and a customized model. The general-purpose large model is used for data fusion, natural language understanding, and general decision-making, while the customized model is used for health monitoring and human-computer interaction. The data acquisition terminal is used to collect user characteristic data. The simulation robot is used to perform human-computer interaction with the user. The mobile carrier terminal is used to carry the simulation robot.

[0005] Preferably, the data acquisition terminal includes: Millimeter-wave radar sensors are used to collect human body position and movement trajectory data in real time. In further preferred solutions, different models / functions of sensors can be selected, which can also perform human presence detection, human position detection, human fall detection, voice intercom, human respiratory rate detection, human heart rate detection, and sleep quality detection, etc. The smart monitoring mattress is designed to conform to the human body and collect data on heart rate, respiratory rate, and sleep status. In further optimized solutions, different models / functions of smart monitoring mattresses can be selected, which can also monitor turning over and getting out of bed, provide abnormal alarms, and generate health reports. Brain-computer interfaces are used to collect and convert human brain signals. Infrared thermal imaging sensors are used to monitor human body surface temperature and ambient temperature. Inertial measurement unit (IMU) is used to collect data on human limb movements and postures. Sound sensors are used to capture unusual sounds and are applied to voiceprint recognition. A blood oxygen sensor is used to collect data on the oxygen saturation of the human body.

[0006] Preferably, the data acquisition terminal further includes an alarm module, which sends an alarm trigger signal to the cloud platform when any type of acquired data exceeds a preset threshold or meets alarm logic.

[0007] Preferably, the simulated robot includes: The voice interaction module is used for voice output during user interaction. The voiceprint recognition module is used to verify user identity through voiceprint recognition. The face recognition module is used for real-time face recognition. Object recognition module, used for real-time object recognition; The tracking module sends movement commands to the mobile carrier terminal to synchronously follow the identified face or object.

[0008] Preferably, the simulated robot further includes an expression driving module and an interactive action execution module, wherein the expression driving module is used to control the facial expression movements of the simulated robot, and the interactive action execution module is used to control the simulated robot to perform interactive limb movements.

[0009] Preferably, the mobile carrier terminal includes a smart wheelchair, a vehicle-mounted monitoring and recording module, an automatic navigation module, and a vehicle-mounted power supply module; the smart wheelchair is used to carry the simulated robot, the vehicle-mounted monitoring and recording module is used to collect video data of the surrounding environment in real time, and the vehicle-mounted power supply module is used to provide power to the smart wheelchair and the simulated robot.

[0010] To achieve the above objectives, the present invention also provides a method for implementing a four-dimensional integrated monitoring intelligent collaborative system based on multi-sensor fusion, comprising the following steps: Step S1: The cloud platform receives and aggregates various types of data uploaded by the data acquisition terminal, the simulation robot, and the mobile carrier terminal. The data includes health monitoring data, environmental data, interaction data, and equipment operation data. Step S2: The cloud platform cleans and merges the aggregated data to obtain the integrated feature data corresponding to the data acquisition terminal, the simulation robot, and the mobile carrier terminal. Step S3: Input the fused comprehensive feature data into the general large model to perform cross-component data correlation mining and identify potential anomalies; Step S4: Based on the recognition results, call the customized model to perform specific optimizations for health monitoring, human-computer interaction, and mobile control scenarios, and generate optimized operating parameters; Step S5: The optimized operating parameters are synchronously sent to the corresponding data acquisition terminals, simulation robots, and mobile carrier terminals to adjust their operating status and improve system performance.

[0011] Preferably, step S2 includes: Step S2.1: Preprocess the data collected from the millimeter-wave radar sensor, smart monitoring mattress, brain-computer interface, infrared thermal imaging sensor, inertial measurement unit, sound sensor and blood oxygen sensor to obtain preprocessed sensor data; Step S2.2: Extract physiological features, motion features, and environmental features from the sensor data, respectively; Step S2.3: Calculate the real-time confidence weights of the physiological features, motion features, and environmental features, and then perform weighted fusion of each feature to generate comprehensive feature data.

[0012] Preferably, the cross-component data association mining in step S3 includes: Step S3.1: Combine the integrated feature data from the data acquisition terminal, the simulation robot, and the mobile carrier terminal to construct a joint feature vector; Step S3.2: Input the joint feature vector into the general large model, and calculate the abnormal deviation between the current feature and the pre-trained normal mode baseline through the model; Step S3.3: When the abnormal deviation exceeds the preset threshold, locate and output the abnormal source.

[0013] Preferably, step S4 includes: Step S4.1: Based on the anomaly identification results, determine the health monitoring, human-computer interaction, or mobile control scenarios that need optimization; Step S4.2: Call the corresponding customized model respectively to generate optimized operating parameters for each scenario. The operating parameters include the sampling frequency and alarm threshold of the data acquisition terminal, the interaction content and action parameters of the simulation robot, and the moving speed and navigation strategy of the mobile carrier terminal.

[0014] Compared with the prior art, the present invention has the following beneficial effects: This invention utilizes a general large model on a cloud platform to deeply integrate and correlate heterogeneous data from data acquisition terminals, simulation robots, and mobile carrier terminals, thereby achieving a more comprehensive and accurate understanding of user status and environment.

[0015] This invention can generate optimization parameters by calling a customized model based on the anomalies and potential needs identified by the general large model, and then distribute them to each terminal to form a closed-loop control process of perception, analysis, decision-making and execution, thereby realizing dynamic synchronization and collaborative optimization of the operation of each component.

[0016] In this invention, the simulated robot incorporates a customized model, enabling more natural and emotional human-computer interaction. Furthermore, its behavior can be adjusted based on the fused data, making the service more personalized and precise.

[0017] The data acquisition terminal in this invention integrates multiple sensors and is equipped with an alarm module, enabling real-time monitoring of anomalies; the collaborative control of the cloud platform ensures timely information transmission and rapid command response, guaranteeing the reliability of health and wellness services. Attached Figure Description

[0018] Figure 1 This is a system block diagram of Embodiment 1 of the present invention. Detailed Implementation

[0019] To enable those skilled in the art to have a clearer understanding of the present invention, the present invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described below are merely for illustrative purposes and to facilitate understanding. The technical solutions provided by the present invention are not limited to those provided in the following embodiments, nor should they limit the scope of protection of the present invention. Example 1

[0020] like Figure 1 As shown, this embodiment provides a four-dimensional integrated monitoring intelligent collaborative system with multi-sensor fusion. The system includes: a cloud platform 100, a data acquisition terminal 200, a simulation robot 300, and a mobile carrier terminal 400 that communicate with the cloud platform 100 via a wireless network (such as 4G / 5G, Wi-Fi).

[0021] The cloud platform 100 is the data processing and decision-making center, deployed on a cloud server. The cloud platform 100 is configured with a general-purpose large model 101 and a customized model 102. The general-purpose large model 101 is a deep learning model (e.g., based on a Transformer architecture) pre-trained on large-scale, multi-source data. Its main functions are: 1) Data fusion: spatiotemporally aligning and fusing features from data of different terminals and types to construct a unified state representation; 2) Natural language understanding: understanding user voice commands transmitted from the simulation robot 300; 3) General decision-making: performing preliminary, cross-domain anomaly identification and intent judgment based on the fused global state. The customized model 102 consists of several miniaturized, high-efficiency models trained for specific tasks, including a health analysis model for processing health monitoring data and an interaction decision model for driving the simulation robot to perform emotional interactions. The customized model 102 uses the output of the general-purpose large model 101 as part of its input for further specialized analysis.

[0022] To make the technical solution of this invention clearer and more complete, specific implementation examples of the general large model and customized model configured in the cloud platform are provided below: (1) For general large models, this example is based on the Transformer architecture and obtains general understanding ability through pre-training with massive multi-source data. The pre-training data includes: medical and health literature, daily dialogue data, sensor time series data, image and video data, etc. The model input layer is designed as a multi-source data adapter, which can simultaneously receive three types of input: text input: user speech-to-text results, system logs; time series input: millimeter-wave radar trajectory data, IMU attitude sequence, heart rate and respiratory waveforms; image input: infrared thermal image, key frames of monitoring video. The model adopts a cross-modal attention mechanism to map data from different modalities to a unified semantic space, realizing the deep fusion of heterogeneous data. For example, when the millimeter-wave radar detects a fall event, the model can automatically associate the heart rate changes at the same time, whether the sound sensor has captured the cry for help, and the robot's current position, comprehensively judge the severity of the event and generate a response strategy.

[0023] (2) For the customized model, the customized model 102 is a model that is lightweight and professionally adjusted for a specific scenario based on the general large model. The following implementation methods can be adopted: (21) This example is for the health monitoring scenario (i.e., the health monitoring customized model). It adopts a lightweight small language model architecture, extracts professional knowledge for physiological signal analysis from the general large model, controls the number of model parameters to less than 100 million, and can run with low latency at the edge. Its main functions include: abnormal warning: real-time analysis of physiological indicators such as heart rate, respiration, and blood oxygen, and issuing a warning when an abnormality exceeding the personalized threshold is detected; trend prediction: predicting the user's health trend based on historical data, such as changes in sleep quality and blood pressure fluctuation patterns; slow Disease management: For chronic diseases such as hypertension and diabetes, personalized monitoring plans and intervention suggestions are provided; (22) This example is for human-computer interaction (i.e., human-computer interaction customized model), which adopts a basic model combined with memory enhancement architecture. The model contains four core modules: User profile module: continuously collects and updates the user's basic information (age, gender, health status), preferences (favorite name, topic), and interaction history; Emotional computing module: infers the user's emotional state in real time through voiceprint recognition, facial expression analysis, and language content understanding; Interaction strategy module: decides what kind of interaction content and tone (such as comfort, encouragement, reminder) to use based on the user profile and current emotion; Action generation module: transforms the interaction strategy into specific actions. The system outputs voice, facial expression-driven commands, and body movement commands. After each interaction with the user, the system will structure and store the core data of this interaction (such as changes in user emotions, response content, and feedback effects) to form a user memory archive that can be continuously iterated. In subsequent interactions, the model first reads the user's long-term profile and historical memory, and then generates a response in combination with the current dialogue context to achieve true personalized continuation. For example, if the model remembers that the user mentioned knee pain yesterday, it will automatically ask about the knee condition in today's interaction. This cross-cycle memory ability significantly improves the realism of the interaction. (23) This example is for mobile control scenarios (i.e., mobile customized models) and adopts a method that combines deep reinforcement learning and path planning. It includes: an environmental perception module that integrates data from LiDAR, visual sensors, and millimeter-wave radar to build an environmental map in real time and identify dynamic obstacles; a path planning module that generates a globally optimal path based on the RRT algorithm / obstacle avoidance algorithm; and a motion control module that uses a deep reinforcement learning algorithm, trained in a simulation environment and then transferred to the physical robot to achieve smooth and safe following motion. The model receives high-level instructions from a general large model (such as following the user, going to the bedroom, and returning to the charging station), and combines real-time environmental perception data to output low-level speed and steering control instructions. During training, tens of thousands of hours of data on elderly care scenarios are introduced to enable the model to learn safe navigation strategies in narrow spaces and low-light conditions at night.

[0024] In addition, the cloud platform has a built-in high-capacity data storage module for persistent storage of the operation logs and parameter configuration data of general large models and customized vertical small models, as well as the collected data and operation status data uploaded by various components, and supports encrypted data backup and on-demand retrieval.

[0025] The data acquisition terminal 200 is deployed in the user's living space or worn on the user's body to collect multi-dimensional feature data of the user. In this embodiment, the data acquisition terminal 200 includes multiple sensors: a millimeter-wave radar sensor 201: installed on an indoor wall or ceiling, which transmits and receives millimeter-wave signals to collect the human body's position coordinates, movement trajectory, and posture (such as standing, walking, falling) in real time, unaffected by light and privacy; a smart monitoring mattress 202: laid on the bed, with built-in piezoelectric film sensors or fiber optic sensing units, conforming to the human body to collect data such as heart rate, respiratory rate, body movement, and deep / light sleep state; and a brain-computer interface 203: in the form of a non-invasive head-mounted device, it collects the user's electroencephalogram (EEG) signals and processes them through a built-in signal processing unit. The system amplifies, filters, and performs analog-to-digital conversion, outputting digital signals that can be used to characterize attention, relaxation, or specific intentions; Infrared thermal imaging sensor 204: Non-contactly monitors the distribution of human body surface temperature and ambient temperature, which can be used to make a preliminary judgment of fever or environmental discomfort; Inertial measurement unit 205: Usually integrated into wearable bracelets or pendants, containing a three-axis accelerometer, gyroscope, and magnetometer, used to collect posture data of human limb movements, such as arm swing amplitude and walking gait; Sound sensor 206: Also known as a microphone array, placed in key indoor locations to capture abnormal sounds in the environment, such as crying, calls for help, or objects falling; Blood oxygen sensor 207: Integrated into smart bracelets or finger clip devices, used to collect human blood oxygen saturation data.

[0026] In addition, the data acquisition terminal 200 also includes an alarm module 208, which is connected to each sensor and has built-in preset thresholds (such as heart rate below 40 beats / min or above 120 beats / min) and alarm logic (such as millimeter-wave radar detecting a fall, sound sensor not detecting activity). When any acquired data triggers an alarm condition, the alarm module 208 immediately sends an alarm trigger signal to the cloud platform 100. This signal includes the alarm type and preliminary data. Based on the received signal, the cloud platform automatically triggers the emergency contact process and sends alarm information to preset emergency contacts via SMS, voice call, or APP push, simultaneously pushing the specific data that triggered the alarm, the corresponding sensor type, location information, and risk level, so as to quickly understand the cause of the warning.

[0027] It should be noted that the above sensors / modules can all use existing mature technologies or products, so they will not be elaborated on here.

[0028] The humanoid robot 300 is a physical robot with a humanoid appearance and interactive capabilities. Its hardware can adopt existing mature humanoid robot structures. In this embodiment, it also includes the following functional modules: a voice interaction module 301, containing a speaker and a speech synthesizer, used to convert text replies sent from the cloud platform into voice output for dialogue with the user; a voiceprint recognition module 302, receiving user voice, extracting voiceprint features, and comparing them with a pre-stored user voiceprint model to quickly verify the user's identity; a face recognition module 303, capturing face images through a high-definition camera built into the head, performing real-time face detection and recognition for identity verification and emotional state (such as facial expressions) analysis; and an object recognition module 304, also using a camera and combined with object detection algorithms, to identify objects in the environment in real time. The system uses objects such as furniture, medicine, and water cups to provide contextual information for interaction; the tracking module 305 receives target information (such as the user's face) from the cloud platform or its own face recognition module, generates movement commands (such as direction and speed), and sends them to the mobile carrier terminal 400 to control the movement of the mobile carrier terminal 400, thereby achieving synchronous following of the face or specific object; the expression driving module 306 receives expression commands from the cloud platform and drives the silicone skin to make corresponding facial expressions, such as smiling, surprise, and concern, by controlling multiple micro motors (the robot's facial muscle simulation structure) on the robot's face; the interactive action execution module 307 receives action commands from the cloud platform and controls the joint motors of the robot's arms, head, and torso to make interactive physical actions such as greeting, handing over items, and hugging.

[0029] In this embodiment, the mobile carrier terminal 400 is specifically implemented as an intelligent wheelchair, which includes: an intelligent wheelchair 401 (vehicle body): having a power drive system and a rechargeable battery, used to carry the simulation robot 300 for movement; an on-board monitoring and recording module 402: a camera installed on the vehicle body, used to collect video data of the surrounding environment in real time, to assist navigation and security monitoring; an automatic navigation module 403: integrating a GPS / BeiDou dual positioning module, including LiDAR and SLAM algorithm, capable of building an environmental map and realizing autonomous path planning and obstacle avoidance navigation; and an on-board power supply module 404: including a large-capacity battery (such as a rechargeable lithium battery pack) and a power management system, with power monitoring, overcharge protection, and undervoltage warning functions, not only providing power for the intelligent wheelchair's own movement, but also providing continuous power to the simulation robot 300 carried on it through wired or wireless means, ensuring its long-term operation.

[0030] In this embodiment, the workflow and data flow of each part of the system are as follows: various sensors of the data acquisition terminal 200 collect data in real time and upload it to the cloud platform 100 through the alarm module 208 or directly; the simulated robot 300 uploads the collected voice and image data during the interaction process; the mobile carrier terminal 400 uploads its own power, location, and monitoring video data. The general large model 101 of the cloud platform 100 performs fusion analysis on all the data. Once a potential problem is identified (such as the user's low mood or abnormal activity level) or an alarm signal is received, the corresponding customized model 102 is invoked to generate specific response strategies. For example, comforting words with caring expressions are generated (and sent to the simulated robot 300). At the same time, the following strategy of the mobile carrier terminal 400 is adjusted to make it closer to the user. These instructions are sent out synchronously to achieve coordinated response of each terminal in time and space. Example 2

[0031] This embodiment provides a method for implementing a smart collaborative system for elderly care based on simulated robots. This method can be applied to the system described in Embodiment 1, and includes the following steps: Step S1: Data Summarization The cloud platform 100 receives and aggregates various data uploaded in real time from the data acquisition terminal 200 (various sensor data, alarm signals), the simulation robot 300 (interaction logs, audio and video streams), and the mobile carrier terminal 400 (location, power consumption, monitoring video) through message queues or API interfaces. The cloud platform adds a timestamp and device ID tag to each piece of data for easy subsequent processing.

[0032] Step S2: Data Cleaning and Fusion The cloud platform 100 processes the aggregated multi-source heterogeneous data. This process specifically includes: Step S2.1: Preprocessing, including filtering and point cloud clustering of millimeter-wave radar data; removing baseline drift from smart monitoring mattress data; removing artifacts from brain-computer interface data; calculating pose of IMU data; encoding and extracting keyframes from audio and video data, etc. Step S2.2: Feature extraction. High-level semantic features are extracted from the preprocessed data: Physiological features: heart rate variability, apnea index, mean blood oxygen saturation, body temperature change rate, and EEG rhythm (α, β, θ waves) are extracted from mattress, blood oxygen, brain-computer interface, and thermal imaging data; Motion features: step frequency, stride length, postural stability, fall impact force, and joint range of motion are extracted from millimeter-wave radar and IMU data; Environmental features: ambient temperature, humidity (indirectly), light intensity, and obstacle distribution are extracted from infrared thermal imaging and vehicle monitoring video. Step S2.3: Weighted fusion. Based on the real-time confidence of each sensor (e.g., millimeter-wave radar has a high confidence when the light is strong, and smart mattresses have a high confidence at night), calculate dynamic weights and perform weighted fusion on the above physiological, motion, and environmental feature vectors to generate a comprehensive feature vector representing the user's current state.

[0033] Step S3: Cross-component data association mining The comprehensive feature data generated in step S2, corresponding to the data acquisition terminal, the simulation robot, and the mobile carrier terminal respectively, are subjected to correlation analysis. This step includes: Step S3.1: Construct a joint feature vector by concatenating the comprehensive feature data of the three into a joint feature vector. This vector contains the global state information of the entire system, such as [user heart rate, body temperature, posture, robot interaction frequency, robot current position, wheelchair battery level, etc.]. Step S3.2: Calculate the anomaly deviation degree. Input the joint feature vector into the general large model 101. The general large model 101 calculates the difference between the current system state and the pre-trained baseline state representing the "normal health care mode" through its self-attention mechanism and forward propagation network, and outputs anomaly deviation degree scores in one or more dimensions. The baseline state is learned from the daily data of a large number of healthy users.

[0034] Step S3.3: Locate the source of the anomaly. When the deviation of an anomaly in a certain dimension exceeds the preset threshold, the general large model 101 locates and outputs the main data source that caused the anomaly through its interpretability module (such as attention weight distribution), i.e., the source of the anomaly. For example, "the heart rate variability is too high and the user is still, which is judged as a physiological abnormality" or "the robot is separated from the target for a long time and the wheelchair navigation path is blocked, which is judged as a mobile coordination abnormality".

[0035] Step S4: Specific Optimization and Parameter Generation Based on the anomaly identification results of step S3, the system calls the customized model 102 for specific optimization. This step includes: Step S4.1: Determine the optimization scenario. Based on the located anomaly source, determine the scenario that needs intervention. For example, if the anomaly source is a physiological indicator, then determine the optimization scenario for health monitoring; if the anomaly source is poor interaction, then optimize the human-computer interaction scenario; if the anomaly source is obstructed movement, then optimize the movement control scenario.

[0036] Step S4.2: Call the custom model to generate parameters, and call the corresponding custom models respectively: (1) Health monitoring custom model: input abnormal information and user historical data, output optimized parameters, such as increasing the sampling frequency of heart rate monitoring from 1Hz to 10Hz, and tightening the alarm threshold from 40~120 times / min to 45~110 times / min; (2) Human-computer interaction custom model: input user emotional state (which can be obtained by analysis of the general large model) and context, output optimized interaction content and action parameters, such as generating a caring greeting text, and issuing caring facial expression and shoulder patting action instructions; (3) Motion control custom model: input obstacle information and target location, output optimized movement speed and navigation strategy, such as planning a new path and reducing the maximum movement speed from 1m / s to 0.5m / s to ensure safety.

[0037] Step S5: Parameter Synchronization and Status Adjustment The cloud platform 100 packages the optimized operating parameters generated in step S4 and synchronously sends them to the corresponding data acquisition terminal 200, simulation robot 300, and mobile carrier terminal 400 via the command channel. Upon receiving the commands, each terminal adjusts its operating status in real time; for example, the data acquisition terminal adjusts its sampling rate, the simulation robot changes its interaction content and actions, and the mobile carrier terminal adjusts its movement strategy. The adjusted new data is then collected and uploaded to the cloud, initiating a new round of collaborative optimization, thereby achieving continuous improvement in system performance.

[0038] To more clearly illustrate the collaborative control process of this invention, let's take a specific scenario as an example: Scenario: User Zhang is home alone and resting in his bedroom at night.

[0039] Initial state: The simulation robot 300 is parked on the intelligent wheelchair 400, waiting in the living room, and the intelligent monitoring mattress 202 and millimeter-wave radar 201 are working normally.

[0040] Incident occurred at 2 a.m. Zhang got up to use the toilet. The smart monitoring mattress 202 detected the signal of Zhang getting out of bed, and the millimeter-wave radar 201 captured Zhang's movement trajectory as he got up and moved towards the toilet. At the same time, the infrared thermal imaging sensor 204 showed that the room temperature was slightly low, at 18°C.

[0041] Data Upload and Fusion: All data is uploaded to the cloud platform 100. The general model 101 integrates information such as getting out of bed, slow movement, and low room temperature. No emergency anomalies such as falls were found, but it was determined that the user got up at night and the environment was relatively cold.

[0042] Collaborative decision-making and instruction issuance: For mobile carrier terminals: General large model 101 calls the mobile control customized model, and the decision result is to enable the companion mode. The model generates the instruction: the intelligent wheelchair 400 carries the simulation robot 300 and automatically navigates to the bathroom door at a low speed (such as 0.3m / s) to wait, turns on the night light, and turns on the vehicle monitoring and recording module 402.

[0043] For the simulation robot: The human-computer interaction customized model is invoked, and the decision result is to provide lighting and a caring greeting. The model generates instructions: The simulation robot 300 can control the IoT lights in the home through instructions, and when Mr. Zhang returns to the bedroom, it greets him through the voice interaction module 301 (set voice): Hello, master, it's cool at night, be careful not to catch a cold.

[0044] For the data acquisition terminal: call the customized health monitoring model, the decision result is to improve the monitoring sensitivity, the model generation instruction: improve the sensitivity of the fall detection algorithm of millimeter-wave radar, and pay attention to the heart rate data after the user returns to the bed.

[0045] The above description represents the preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements without departing from the design principles and technical solutions of the present invention, and these improvements should also be considered within the scope of protection of the present invention.

Claims

1. A four-dimensional integrated intelligent collaborative monitoring system with multi-sensor fusion, characterized in that, The system includes a cloud platform (100), a data acquisition terminal (200) communicating with the cloud platform (100), a simulation robot (300), and a mobile carrier terminal (400); wherein the cloud platform (100) is configured with a general large model (101) and a customized model (102), the general large model (101) is used for data fusion, natural language understanding and general decision making, and the customized model (102) is used for health monitoring and human-computer interaction; the data acquisition terminal (200) is used to collect user feature data; the simulation robot (300) is used to perform human-computer interaction with the user; and the mobile carrier terminal (400) is used to carry the simulation robot (300).

2. The four-dimensional integrated monitoring intelligent collaborative system based on multi-sensor fusion according to claim 1, characterized in that: The data acquisition terminal (200) includes: Millimeter-wave radar sensor (201) is used to collect human body position and motion trajectory data in real time; The intelligent monitoring mattress (202) is designed to conform to the human body and collect data on heart rate, respiratory rate and sleep status. Brain-computer interface (203) is used to collect human brain electrical signals and complete signal conversion; Infrared thermal imaging sensor (204) is used to monitor human body surface temperature and ambient temperature; An inertial measurement unit (205) is used to collect human limb movement and posture data; A sound sensor (206) is used to detect unusual sounds; A blood oxygen sensor (207) is used to collect human blood oxygen saturation data.

3. The four-dimensional integrated monitoring intelligent collaborative system based on multi-sensor fusion according to claim 2, characterized in that: The data acquisition terminal (200) also includes an alarm module (208), which is used to send an alarm trigger signal to the cloud platform (100) when any type of acquired data exceeds a preset threshold or meets the alarm logic.

4. The four-dimensional integrated monitoring intelligent collaborative system based on multi-sensor fusion according to claim 3, characterized in that: The simulated robot (300) includes: The voice interaction module (301) is used for voice output during user interaction. The voiceprint recognition module (302) is used to complete user authentication through voiceprint recognition; The face recognition module (303) is used for real-time face recognition; The object recognition module (304) is used for real-time object recognition; The tracking module (305) sends a movement command to the mobile carrier terminal (400) to synchronously follow the identified face or object.

5. The four-dimensional integrated monitoring intelligent collaborative system based on multi-sensor fusion according to claim 4, characterized in that: The simulated robot (300) further includes an expression driving module (306) and an interactive action execution module (307), wherein the expression driving module (306) is used to control the facial expression movements of the simulated robot (300), and the interactive action execution module (307) is used to control the simulated robot (300) to perform interactive limb movements.

6. The four-dimensional integrated monitoring intelligent collaborative system based on multi-sensor fusion according to claim 5, characterized in that: The mobile carrier terminal (400) includes an intelligent wheelchair (401), an in-vehicle monitoring and recording module (402), an automatic navigation module (403), and an in-vehicle power supply module (404); the intelligent wheelchair (401) is used to carry the simulation robot (300), the in-vehicle monitoring and recording module (402) is used to collect video data of the surrounding environment in real time, and the in-vehicle power supply module (404) is used to provide power to the intelligent wheelchair (401) and the simulation robot (300).

7. The method for implementing the multi-sensor fusion four-dimensional integrated monitoring intelligent collaborative system as described in any one of claims 1 to 6, characterized in that: Includes the following steps: Step S1: The cloud platform receives and aggregates various types of data uploaded by the data acquisition terminal, the simulation robot, and the mobile carrier terminal. The data includes health monitoring data, environmental data, interaction data, and equipment operation data. Step S2: The cloud platform cleans and merges the aggregated data to obtain the integrated feature data corresponding to the data acquisition terminal, the simulation robot, and the mobile carrier terminal. Step S3: Input the fused comprehensive feature data into the general large model to perform cross-component data correlation mining and identify potential anomalies; Step S4: Based on the recognition results, call the customized model to perform specific optimizations for health monitoring, human-computer interaction, and mobile control scenarios, and generate optimized operating parameters; Step S5: The optimized operating parameters are synchronously sent to the corresponding data acquisition terminals, simulation robots, and mobile carrier terminals to adjust their operating status and improve system performance.

8. The method for implementing the multi-sensor fusion four-dimensional integrated monitoring intelligent collaborative system according to claim 7, characterized in that: Step S2 includes: Step S2.1: Preprocess the data collected from the millimeter-wave radar sensor, smart monitoring mattress, brain-computer interface, infrared thermal imaging sensor, inertial measurement unit, sound sensor and blood oxygen sensor to obtain preprocessed sensor data; Step S2.2: Extract physiological features, motion features, and environmental features from the sensor data, respectively; Step S2.3: Calculate the real-time confidence weights of the physiological features, motion features, and environmental features, and then perform weighted fusion of each feature to generate comprehensive feature data.

9. The method for implementing the multi-sensor fusion four-dimensional integrated monitoring intelligent collaborative system according to claim 8, characterized in that: The cross-component data association mining in step S3 includes: Step S3.1: Combine the integrated feature data from the data acquisition terminal, the simulation robot, and the mobile carrier terminal to construct a joint feature vector; Step S3.2: Input the joint feature vector into the general large model, and calculate the abnormal deviation between the current feature and the pre-trained normal mode baseline through the model; Step S3.3: When the abnormal deviation exceeds the preset threshold, locate and output the abnormal source.

10. The method for implementing the multi-sensor fusion four-dimensional integrated monitoring intelligent collaborative system according to claim 9, characterized in that: Step S4 includes: Step S4.1: Based on the anomaly identification results, determine the health monitoring, human-computer interaction, or mobile control scenarios that need optimization; Step S4.2: Call the corresponding customized model respectively to generate optimized operating parameters for each scenario. The operating parameters include the sampling frequency and alarm threshold of the data acquisition terminal, the interaction content and action parameters of the simulation robot, and the moving speed and navigation strategy of the mobile carrier terminal.

Images