Old-age care robot capable of sending emergency help and shooting

Abstract

The invention relates to an old-age care robot capable of sending out emergency help and shooting, belongs to the technical field of robots, and aims to solve the problems that the perception ability of an existing old-age care robot is reduced and a human-computer interaction response mechanism is single when light is poor or shielding exists. According to the robot, environment and behavior data are collected through a multi-mode perception fusion unit, behavior recognition and intention inference are conducted through an intention understanding and decision center, and self-adaptive response is triggered based on a multi-level response strategy and comprises voice inquiry, video recording or emergency help seeking. According to the method, vision, millimeter wave radar and acoustic perception are fused, knowledge graphs and graph neural network reasoning are combined, the accuracy of abnormal behavior recognition and the ability to analyze fuzzy intentions are improved, and meanwhile flexible navigation and quick response are achieved through an omni-directional moving chassis.

Description

Technical Field

[0001] This invention belongs to the field of robotics technology, specifically relating to an elderly care robot equipped with a camera that can issue emergency calls for help. Background Technology

[0002] With the accelerating aging of the global population and the increasing strain on elderly care resources, smart elderly care has become an important area for the deep integration of robotics technology and health monitoring systems. This field is dedicated to improving the safety and quality of care for the elderly through intelligent means, and elderly care robots with emergency assistance functions are one of the key devices for achieving these goals.

[0003] Among them, camera-equipped elderly care robots capable of issuing emergency assistance are a specific technological branch of the smart elderly care field. Their core objective is to automatically identify and promptly alert elderly people living alone to abnormal conditions by integrating visual perception and communication modules. Such systems typically trigger pre-set assistance procedures based on environmental perception data, aiming to provide safety guarantees for the elderly in unattended scenarios.

[0004] Current technologies generally employ a single type of sensor as the primary means of environmental perception, such as relying solely on visual cameras or LiDAR for behavior monitoring. Such solutions face significant limitations in real-world home environments: when indoor lighting conditions are poor or obstructions such as furniture and walls are present, the system's perception capabilities drastically decrease, leading to a significant drop in the accuracy of identifying key abnormal behaviors such as "elderly falls," resulting in a persistently high false alarm rate. Furthermore, most existing systems lack a deep understanding of the ambiguous behavioral intentions of the elderly, making their human-computer interaction stiff and unnatural, and their response mechanisms relatively simple and rigid, making it difficult to achieve timely detection and reliable assistance in real-world, complex home scenarios. Therefore, how to achieve high-precision, low-false-alarm abnormal behavior recognition in changing real-world environments and establish a natural and efficient emergency response mechanism has become a pressing technical challenge in this field. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the technical defects of existing elderly care robots, such as the reduced perception ability when the light is poor or there are obstructions, resulting in low accuracy of abnormal behavior recognition and rigid human-computer interaction response mechanism, and to provide an elderly care robot with a camera that can issue emergency help.

[0006] The technical solution of this invention is: an elderly care robot capable of issuing emergency calls and equipped with a camera. Its system architecture consists of a multimodal perception fusion unit, an intent understanding and decision-making center, an adaptive response execution unit, and a robot mobile platform. The multimodal perception fusion unit integrates a visual perception module, a millimeter-wave radar perception module, and an acoustic perception module to collect environmental and behavioral data of the elderly. The visual perception module uses a combination of a color camera with wide dynamic range imaging capabilities and a low-light enhanced infrared thermal imager. The color camera captures high-resolution color images under normal lighting conditions, while the low-light enhanced infrared thermal imager generates thermal images by detecting human body thermal radiation under insufficient lighting conditions. The millimeter-wave radar perception module uses a frequency-modulated continuous wave radar system, operating in the 60 GHz to 64 GHz frequency band. By emitting millimeter-wave signals and receiving echo signals reflected from the human body, it analyzes the target's distance information, radial velocity information, and micro-motion characteristic information. The acoustic perception module consists of a microphone array composed of two or more microphones, used to collect environmental sound signals and perform sound source localization and voice enhancement based on beamforming technology. The multimodal perception fusion unit is further configured with a data fusion processor, which runs a feature-level fusion algorithm based on a deep neural network. The algorithm first synchronizes and aligns the image features from the visual perception module, the point cloud and micro-Doppler features from the millimeter-wave radar perception module, and the acoustic event features from the acoustic perception module. Then, these heterogeneous features are input into a multi-branch feature extraction network for deep feature extraction. Finally, a unified multimodal behavior representation vector is output through a feature fusion layer weighted by an attention mechanism.

[0007] The intent understanding and decision-making center is connected to the multimodal perception fusion unit to receive multimodal behavior representation vectors and perform behavior recognition and intent inference. The intent understanding and decision-making center includes an abnormal behavior recognition engine and a fuzzy intent parsing module. The abnormal behavior recognition engine has a built-in behavior classifier based on a hybrid model of temporal convolutional networks and long short-term memory networks. This classifier takes the time series of multimodal behavior representation vectors as input, captures local temporal patterns through the temporal convolutional network, and models long-term temporal dependencies through the long short-term memory network. The final output is a probability distribution corresponding to a preset abnormal behavior category, which includes at least falling behavior, prolonged stillness, and violent struggling behavior. The fuzzy intent parsing module is connected to the abnormal behavior recognition engine and is used to perform deep analysis of the elderly person's potential request for help when the abnormal behavior recognition engine does not trigger a high-level alarm. The fuzzy intent parsing module runs an intent inference process based on multimodal contextual reasoning. This process first constructs a knowledge graph that includes the elderly’s daily behavior patterns, current environmental context, and historical interaction records. Then, it combines real-time acquired multimodal perception data and performs inference calculations through graph neural networks to generate a probabilistic assessment of the elderly’s possible current intents. The types of intents assessed include explicit verbal requests for help, nonverbal cues for help, and unconscious warning signs of dangerous behavior.

[0008] The adaptive response execution unit connects to the intent understanding and decision-making center, and executes corresponding response strategies based on the center's output. The adaptive response execution unit includes a multi-level response strategy manager and a human-machine collaborative interaction interface. The multi-level response strategy manager predefines three levels of response strategies, from low to high. Level 1 response strategy targets low-probability potential risks assessed by the fuzzy intent parsing module; its action is to control the robot's mobile platform to approach the elderly person and initiate a gentle voice inquiry through the human-machine collaborative interaction interface. Level 2 response strategy targets medium-confidence abnormal behavior identified by the abnormal behavior recognition engine or medium-probability risks assessed by the fuzzy intent parsing module; its action is to automatically activate the robot's camera module to record video while initiating a voice inquiry for confirmation, and generate a pending alarm message to be sent. Level 3 response strategy targets high-confidence severe abnormal behavior identified by the abnormal behavior recognition engine; its action is to immediately trigger an emergency assistance process. The emergency assistance process includes automatically dialing a pre-set emergency contact number via a 4G or 5G communication module integrated into the robot. Simultaneously, it packages and sends recorded high-definition video footage, robot location information, and abnormal behavior analysis reports to a cloud-based monitoring platform and the emergency contact's terminal. The human-machine collaborative interaction interface integrates a natural language processing module and an emotion computing module. The natural language processing module understands the elderly person's voice input and generates context-appropriate natural language responses. The emotion computing module analyzes the elderly person's tone, speech rate, and facial expressions to infer their emotional state and adjusts the robot's tone and wording accordingly.

[0009] The robot mobile platform uses an omnidirectional chassis, and its drive system consists of three or more Mecanum wheels or omnidirectional wheels, enabling translational and rotational movements in any direction within a plane. This ensures the robot can flexibly avoid obstacles and reach its target location in the complex home environment. The robot mobile platform also integrates a simultaneous localization and mapping (SLAM) module. This module utilizes the fusion of LiDAR and visual odometry technology to build and update a map of the home environment in real time, providing support for the robot's navigation and path planning.

[0010] Furthermore, the feature-level fusion algorithm based on deep neural networks running in the data fusion processor has the following specific structure for its multi-branch feature extraction network: for the visual branch, a deep residual network is used to extract the spatial features of the image; for the millimeter-wave radar branch, a one-dimensional convolutional network is used to process the radar signal spectrogram to extract motion features; for the acoustic branch, a convolutional recurrent neural network is used to process the acoustic spectrogram to extract acoustic event features; the attention-weighted feature fusion layer calculates the importance weight of each feature channel and concatenates the weighted features to form the final multimodal behavior representation vector.

[0011] Furthermore, the behavior classifier in the abnormal behavior recognition engine employs a hard sample mining strategy during its training process. This strategy dynamically identifies samples that are misclassified by the model or have low classification confidence during model training and assigns higher loss weights to these samples in subsequent training iterations to improve the model's classification accuracy for difficult-to-identify abnormal behaviors.

[0012] Furthermore, the knowledge graph constructed in the fuzzy intent parsing module includes node types such as elderly entity, furniture object entity, daily activity event entity, and time segment entity. Edge relationships represent the association strength and co-occurrence frequency between these entities. The reasoning process of the graph neural network updates the node representation through a message passing mechanism and finally converges on the node representing the elderly's current intent for classification.

[0013] Furthermore, the policy switching logic of the multi-level response policy manager is based on a dual-threshold comparison mechanism. When the highest category probability output by the abnormal behavior recognition engine exceeds a preset high threshold, the third-level response policy is directly triggered. When the highest category probability is between a preset low threshold and a preset high threshold, the second-level response policy is triggered. When the abnormal behavior recognition engine does not exceed the low threshold but the intent probability evaluated by the fuzzy intent parsing module exceeds its own set intent threshold, the first-level response policy is triggered.

[0014] Furthermore, the natural language processing module of the human-machine collaborative interaction interface integrates a domain adaptation function. This function continuously learns the personalized vocabulary and expression habits used by the elderly in daily interactions and dynamically optimizes their language model, making the robot's voice interaction more in line with the specific language style of the elderly.

[0015] Furthermore, the synchronous positioning and mapping module of the robot mobile platform distinguishes between dynamic and static obstacles when constructing the environmental map, and only incorporates the information of static obstacles into the long-term map to ensure the stability of the map and the reliability of navigation.

[0016] Compared with the prior art, the beneficial effects of the present invention are:

[0017] (1) This invention constructs a robust environmental perception capability by deeply fusing visual, millimeter-wave radar, and acoustic multimodal perception information and employing a feature fusion algorithm based on deep neural networks and attention mechanisms. The visual module provides rich texture and color information, the millimeter-wave radar module is unaffected by lighting and has a certain ability to penetrate obstructions, and the acoustic module provides additional auditory event cues. The three complement each other and effectively overcome the perception limitations of a single sensor in poor lighting or obstructed scenarios, significantly improving the accuracy and reliability of abnormal behavior recognition and fundamentally reducing the false alarm rate of the system.

[0018] (2) This invention introduces an intent understanding and decision-making center, particularly a fuzzy intent parsing module. By constructing a knowledge graph and using graph neural networks for contextual reasoning, it achieves a deep understanding and probabilistic assessment of the elderly's unclear and ambiguous requests for help. This enables the system not only to respond to obvious emergencies but also to proactively detect and intervene in potential, yet not fully manifested, risks, moving the safety safeguards forward and greatly enhancing its adaptability to complex and subtle home scenarios.

[0019] (3) This invention designs an adaptive response execution unit with a multi-level response strategy. Its response mechanism is not static but rather refined and gradient-based based on risk level and intent probability. From low-level proactive inquiry and confirmation to mid-level preparatory alarms and recording, and then to high-level immediate emergency assistance, this hierarchical response strategy avoids overreaction to low-risk situations while ensuring rapid and decisive handling in truly critical situations. Combined with a human-computer interaction interface possessing affective computing and natural language processing capabilities, the entire response process becomes more natural and humanized, improving the elderly's acceptance and user experience.

[0020] (4) The robot mobile platform of the present invention is based on an omnidirectional mobile chassis and advanced synchronous positioning and mapping technology, which gives the robot a high degree of mobility and autonomous navigation capability in complex home environments, ensuring that it can reach the elderly in time to interact or observe, and providing a solid physical foundation for comprehensive active monitoring and immediate response.

[0021] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, embodiments of the present invention are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the overall technical solution architecture of the elderly care robot with camera that can issue emergency calls for help, as proposed in this invention.

[0024] Figure 2 This is a schematic diagram of the core principle framework of the multimodal perception fusion unit based on deep neural networks and attention mechanisms in this invention;

[0025] Figure 3 This is a logical flowchart of the intent understanding and decision-making center in this invention;

[0026] Figure 4 This is a schematic diagram of the multi-level response strategy management logic of the adaptive response execution unit in this invention;

[0027] Figure 5 This is a schematic diagram of the multi-level data flow of the robot mobile platform interacting with the environment in this invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0029] Example 1

[0030] Please refer to the attached document. Figure 1 This embodiment details the technical implementation of an elderly care robot capable of issuing emergency assistance and equipped with a camera. The system architecture consists of four core components: a multimodal perception fusion unit, an intent understanding and decision-making center, an adaptive response execution unit, and a robot mobile platform. These units exchange data and transmit control commands via a high-speed internal bus and a dedicated communication protocol, ensuring the system's real-time performance and reliability.

[0031] The multimodal perception fusion unit, serving as the system's perception front-end, is responsible for comprehensively collecting environmental and behavioral data from the elderly. This unit integrates a visual perception module, a millimeter-wave radar perception module, and an acoustic perception module. The visual perception module employs a combined deployment scheme of a color camera with wide dynamic range imaging capabilities and a low-light enhanced infrared thermal imager. The color camera is mounted on a rotatable gimbal on the robot's head, with an image sensor resolution of 1920 x 1080 pixels, a frame rate of 30 frames per second, and supports automatic white balance and exposure compensation, used to capture high-resolution color images under normal lighting conditions. The low-light enhanced infrared thermal imager is coaxially mounted with the color camera, possessing a thermal sensitivity better than 0.05 degrees Celsius and operating in the 8-14 micrometer wavelength range. It is used to generate a 320 x 240 pixel resolution thermal image by detecting human thermal radiation under insufficient lighting conditions (below 10 lux). Both image data are transmitted to the data fusion processor via a gigabit Ethernet interface.

[0032] The millimeter-wave radar sensing module employs a frequency-modulated continuous wave radar system. Its core radio frequency chip operates in the 60 GHz to 64 GHz frequency band, with a transmit power of 10 dBmW. The radar antenna array is positioned at the front of the robot's torso. It transmits linear frequency-modulated continuous wave signals and receives echo signals reflected from the human body. Through processing via a mixer, intermediate frequency amplifier, and analog-to-digital converter, the system extracts the target's distance, radial velocity, and micro-motion characteristics. The range resolution reaches 5 cm, and the maximum detection range is 15 meters. Micro-motion characteristics are obtained by extracting the signal phase change rate, enabling the detection of subtle physiological activities such as breathing and hand tremors.

[0033] The acoustic perception module consists of a uniform linear microphone array of four digital microphones with an array aperture of 20 cm. The microphone sampling rate is 16 kHz, and the quantization bit depth is 16 bits. This module uses a beamforming algorithm based on a generalized sidelobe canceller structure for sound source localization and speech enhancement, with a localization accuracy within 3 degrees. After pre-emphasis and frame-windowing processing, the ambient sound signal undergoes a Fast Fourier Transform to obtain a spectrogram, which is then used to extract acoustic event features through an acoustic event detection algorithm.

[0034] Please refer to the attached document. Figure 2 The multimodal perception fusion unit is further configured with a data fusion processor, which adopts a heterogeneous computing architecture, including a multi-core central processing unit and a graphics processing unit. The data fusion processor runs a feature-level fusion algorithm based on a deep neural network. This algorithm first performs timestamp synchronization and spatial coordinate alignment processing on image features from the visual perception module, point cloud and micro-Doppler features from the millimeter-wave radar perception module, and acoustic event features from the acoustic perception module. The time synchronization accuracy is required to be within 10 milliseconds, and spatial alignment is achieved through a calibrated coordinate system transformation matrix.

[0035] The specific structure of the multi-branch feature extraction network in the fusion algorithm is as follows: For the vision branch, a 50-layer deep residual network is used to extract spatial features of the image. The input image is uniformly scaled to 224 x 224 pixels, and the output is a 1024-dimensional feature vector. For the millimeter-wave radar branch, a 5-layer one-dimensional convolutional network is used to process the radar signal spectrum to extract motion features. The convolutional kernel sizes are 7, 5, 3, 3, and 3, respectively, and the output is a 512-dimensional feature vector. For the acoustic branch, a convolutional recurrent neural network is used to process the acoustic spectrogram to extract acoustic event features. The convolutional layers use 3 x 3 convolutional kernels, and the recurrent layers use 64-unit gated recurrent units, outputting a 256-dimensional feature vector.

[0036] The attention-weighted feature fusion layer calculates the importance weight of each feature channel, and the specific implementation formula is as follows:

[0037]

[0038] Where a i f represents the attention weight of the i-th feature. i f represents the eigenvector of the i-th mode; j w represents the eigenvector of the j-th mode; i w is the weight vector associated with the i-th feature. j w is the weight vector associated with the j-th feature. i and w j b is a learnable parameter; i b is the bias term associated with the i-th feature. j This is the bias term associated with the j-th feature. The weighted features are concatenated to form the final 2048-dimensional multimodal behavior representation vector. This vector is updated every 100 milliseconds and transmitted to the intent understanding and decision-making center via a shared memory area.

[0039] The intent understanding and decision-making hub is connected to the multimodal perception fusion unit and is implemented using a dual-core security processor to ensure the real-time performance and reliability of the decision-making process. This hub comprises two core components: an abnormal behavior recognition engine and a fuzzy intent parsing module.

[0040] Please refer to the attached document. Figure 3 The abnormal behavior recognition engine incorporates a behavior classifier based on a hybrid model of temporal convolutional networks and long short-term memory networks. The temporal convolutional network consists of three convolutional layers with kernel widths of 5, 3, and 3, and filter counts of 64, 128, and 256, employing causal convolution to ensure temporal characteristics. The long short-term memory network consists of two layers, each with 128 hidden units. The classifier takes the time series of multimodal behavior representation vectors as input, with a sequence length of 50, corresponding to a 5-second time window. It captures local temporal patterns through the temporal convolutional network, models long-term temporal dependencies through the long short-term memory network, and finally outputs a probability distribution corresponding to a preset abnormal behavior category through a fully connected layer and a softmax activation function. The preset abnormal behavior categories include three types: falling behavior, prolonged stillness, and violent struggling behavior, with a classification confidence threshold set at 0.85.

[0041] The training process of the abnormal behavior recognition engine employs a hard sample mining strategy. This strategy dynamically identifies samples that are misclassified by the model or have a classification confidence level below 0.6 during model training. In subsequent training iterations, these samples are given higher loss weights through a focus loss function, with the focus factor set to 2, to improve the model's classification accuracy for difficult-to-identify abnormal behaviors. Training data augmentation techniques include time warping, sensor noise injection, and modality random dropout.

[0042] The fuzzy intent parsing module connects to the abnormal behavior recognition engine, employing an architecture where a graph database and inference engine work collaboratively. This module runs an intent inference process based on multimodal contextual reasoning. First, a knowledge graph is constructed, containing the elderly person's daily behavior patterns, the current environmental context, and historical interaction records. The knowledge graph's node types include four categories: elderly person entities, furniture object entities, daily activity event entities, and time segment entities. Edge relationships represent the association strength and co-occurrence frequency between these entities; association strength is calculated by statistically analyzing co-occurrence counts and normalized using the Jaccard similarity coefficient.

[0043] The reasoning process of the graph neural network updates node representations through a message-passing mechanism. Specifically, it employs a graph attention network architecture with 8 attention heads per layer and a hidden layer dimension of 256. Node feature initialization includes spatial location encoding, temporal encoding, and semantic encoding. The reasoning process uses three layers of graph convolutional operations, ultimately converging on the node representing the elderly person's current intention for classification, generating a probabilistic assessment of the elderly person's possible current intention. The assessed intention types include three categories: explicit verbal requests for help, nonverbal cues for help, and unconscious warning signs of dangerous behavior. The assessment period is 2 seconds.

[0044] The adaptive response execution unit is connected to the intent understanding and decision-making center and adopts a dual-redundancy architecture of programmable logic controller and embedded system. This unit includes a multi-level response strategy manager and a human-machine collaborative interaction interface.

[0045] Please refer to the attached document. Figure 4 The multi-level response strategy manager predefines three levels of response strategies, from low to high. Level 1 response strategy targets low-probability potential risks (less than 0.3) assessed by the fuzzy intent parsing module. Its action involves controlling the robot's mobile platform to approach the elderly person at a speed of 0.3 meters per second, maintaining a safe distance of 1.5 meters, and initiating a gentle voice inquiry through the human-machine collaborative interaction interface. The voice content templates include five preset options such as "Do you need help?" and "How are you feeling?".

[0046] The Level 2 response strategy targets anomalous behaviors identified by the abnormal behavior recognition engine with a confidence level between 0.6 and 0.85, or moderate-probability risks assessed by the fuzzy intent parsing module with a probability between 0.3 and 0.6. Its action involves, simultaneously initiating a voice inquiry for confirmation, automatically activating the robot's camera module to record 1080p resolution video at a bitrate of 4 megabits per second, and generating a pending alarm message ready for transmission. The alarm message includes 15 fields such as timestamp, location coordinates, and behavior type.

[0047] The Level 3 response strategy targets highly suspicious behaviors (with a confidence level exceeding 0.85) identified by the abnormal behavior recognition engine. The immediate action triggered is an emergency assistance process. This process includes automatically dialing three pre-set emergency contact numbers via a 4G or 5G communication module integrated into the robot, employing a loop redial mechanism until the connection is established. Simultaneously, a 10-second high-definition video clip, robot location information, and an abnormal behavior analysis report are encrypted using an Advanced Encryption Standard (AES) algorithm and packaged before being sent to the cloud monitoring platform and the emergency contact's terminal. Transmission Control Protocol (TCP) is used to ensure data integrity.

[0048] The multi-level response strategy manager's strategy switching logic is based on a dual-threshold comparison mechanism. When the highest category probability output by the abnormal behavior recognition engine exceeds the high threshold of 0.85, the Level 3 response strategy is triggered directly. When the highest category probability is between the low and high thresholds (between 0.6 and 0.85), the Level 2 response strategy is triggered. When the abnormal behavior recognition engine does not exceed the low threshold of 0.6, but the intent probability evaluated by the fuzzy intent parsing module exceeds the intent threshold of 0.3, the Level 1 response strategy is triggered. All threshold parameters can be configured remotely via a network interface.

[0049] The human-computer collaborative interaction interface integrates a natural language processing (NLP) module and an emotion computing module. The NLP module uses a bidirectional encoder to represent a pre-trained model architecture from the transformer, with a Chinese vocabulary containing 22,000 words. This module is used to understand the elderly person's voice input and generate context-appropriate natural language responses, with a response latency controlled within 500 milliseconds. The NLP module also integrates a domain adaptation function. This function continuously learns the personalized vocabulary and expression habits used by the elderly person in daily interactions, dynamically optimizing its language model using an incremental learning algorithm with a learning rate of 0.001, making the robot's voice interaction more closely match the specific language style of the elderly person.

[0050] The emotion computing module infers the emotional state of an elderly person by analyzing their speech tone, speech rate, and facial expressions. Speech feature extraction includes 12 acoustic parameters such as fundamental frequency contour, energy distribution, and speech rate variation. Facial expression analysis uses a convolutional neural network to extract action unit intensity, with an input image size of 48 x 48 pixels. Emotional states are categorized into three types: calm, anxious, and distressed, with activation occurring when the confidence level exceeds 0.7. The emotion computing module adjusts the robot's tone and wording in response based on the emotional state. Speech synthesis parameters include speech rate adjusted between 120 and 180 words per minute, and fundamental frequency varying between 80 and 150 Hz.

[0051] The robot's mobile platform utilizes an omnidirectional chassis, with its drive system consisting of four Mecanum wheels, each driven by an independent 200-watt brushless DC motor and equipped with a 1024-line photoelectric encoder for position feedback. The platform enables translational and rotational motion in any direction within a plane, achieving a maximum linear velocity of 1 meter per second and a rotational speed of 60 degrees per second. Motion control employs a proportional-integral-derivative (PID) controller, achieving a position control accuracy of 2 millimeters.

[0052] Please refer to the attached document. Figure 5 The robot's mobile platform integrates a simultaneous localization and mapping (SLAM) module, which utilizes a fusion technology of LiDAR and visual odometry. The LiDAR employs a 905nm wavelength, a 10Hz scanning frequency, an angular resolution of 0.5 degrees, and a maximum range of 30 meters. The visual odometry uses a feature point method to extract ORB feature points, processing 1000 feature points per frame. A home environment map is built and updated in real time, with a map resolution of 5 cm and an update frequency of 2Hz.

[0053] The simultaneous localization and mapping (SMR) module distinguishes between dynamic and static obstacles when constructing the environmental map. Dynamic obstacle detection is achieved through point cloud difference analysis across three consecutive frames; objects moving at a speed exceeding 0.1 meters per second are classified as dynamic obstacles. Only static obstacle information is incorporated into the long-term map; dynamic obstacle information is stored in a temporary cache with a validity period of 10 seconds. Map management employs a multi-level data structure, including an occupancy grid map, a feature point map, and a semantic map. Figure 3 This system has multiple layers to ensure map stability and navigation reliability.

[0054] The navigation system employs a dynamic window method for local path planning, taking into account robot dynamics constraints. Global path planning uses the A* algorithm, with Euclidean distance as the heuristic function. The obstacle avoidance system uses an artificial potential field method, with the repulsion coefficient dynamically adjusted based on obstacle type: 0.5 for people and 1.0 for furniture. The safety monitoring area is set as a circular zone with an 8-meter radius centered on the robot; navigation priority is automatically adjusted when an elderly person enters this area.

[0055] The power management system uses a 100 Ah lithium-ion battery pack, supporting 8 hours of continuous operation. The charging system employs both wireless and wired charging modes, with wireless charging power at 30 watts and wired charging power at 65 watts. The battery management system monitors battery voltage, current, and temperature parameters in real time, enabling intelligent charging and discharging control and thermal management.

[0056] All sensor data is acquired via a data acquisition card, with sampling rates configured according to sensor type: 30 Hz for visual data, 50 Hz for radar data, and 16 kHz for acoustic data. Data storage employs a circular buffer mechanism; emergency event data is permanently stored, while routine data is retained for 7 days. System logs record all operational events and abnormal states; log files are rotated daily and retained for 30 days.

[0057] The communication system adopts a multi-mode network architecture, supporting wireless LAN, 4G, and 5G connectivity. The network handover algorithm automatically selects the optimal connection based on signal strength and network latency, with a handover delay of less than 1 second. Data transmission employs a dual-protocol stack of User Datagram Protocol (UDP) and Transmission Control Protocol (TCP). Control commands use UDP to ensure real-time performance, while media data transmission uses TCP to ensure reliability.

[0058] The system startup process includes a self-test procedure that checks the status of all sensors, actuators, and communication modules. There are 35 self-test items; failure of any one will trigger a safe mode. The system software adopts a microservice architecture, with each functional module deployed independently and communicating via message queues. A watchdog timer monitors the status of each process and automatically restarts the service in case of an anomaly. The system supports remote upgrades, using differential upgrade technology to reduce bandwidth consumption, and the upgrade package uses digital signatures to verify its integrity.

[0059] Example 2

[0060] This embodiment provides an alternative implementation of an elderly care robot equipped with a camera that can issue emergency assistance, focusing on the optimized design in terms of multimodal perception fusion and decision-making mechanisms.

[0061] In the multimodal perception fusion unit, the data fusion processor adopts an architecture that combines edge computing and cloud computing. A lightweight feature extraction network is deployed on a local edge device, comprising an 8-layer convolutional neural network for vision, a 3-layer one-dimensional convolutional network for radar, and a 2-layer convolutional recurrent neural network for acoustics. These networks output a 512-dimensional compressed feature vector, which is transmitted to the cloud via a secure tunnel to complete deep feature fusion. The cloud fusion server uses a 128-layer deep residual network for visual feature refinement, a 64-layer temporal convolutional network for radar signal analysis, and a 32-layer transformer network for acoustic event understanding.

[0062] The attention-weighted feature fusion layer is implemented in the cloud, employing a multi-head self-attention mechanism with 16 heads and a hidden layer dimension of 1024. The feature fusion formula is extended to a multi-head version:

[0063] MultiHead(Q,K,V)=Concat(head1,...,head h W O

[0064] Each attention head is calculated as follows:

[0065]

[0066] Where Q, K, and V represent the Query, Key, and Value matrices, respectively; head1,...,head h W represents the outputs of the first to the hth attention heads respectively; O Represents the output projection matrix; These represent the projection matrices for the query, key, and value specific to the i-th head. This design allows the system to simultaneously monitor multiple dependencies between features from different modalities, improving the fusion effect. The fused feature vector has a dimension of 4096 and is updated every 200 milliseconds.

[0067] The abnormal behavior recognition engine employs an ensemble learning architecture, comprising three base classifiers: a temporal convolutional network classifier, a long short-term memory network classifier, and a transformer classifier. Each classifier is trained independently, and the prediction results are combined during the inference phase through a weighted voting mechanism. The weights are dynamically adjusted based on the classifier's performance on the validation set, with higher-accurate classifiers assigned higher weights. This ensemble learning framework significantly improves the model's robustness, especially in scenarios with varying data distributions.

[0068] The knowledge graph construction of the fuzzy intent parsing module introduces a real-time update mechanism. By continuously monitoring changes in elderly people's behavioral patterns, the edge weights in the knowledge graph are dynamically adjusted. An exponentially weighted moving average algorithm is used to update the association strength, with a decay factor set to 0.95. A new abnormal event detection function has been added; when behavior that significantly deviates from historical patterns is detected, a temporary node is automatically created in the knowledge graph and marked as an observation state.

[0069] The multi-level response strategy manager introduces a predictive response mechanism. A time-series prediction model trained on historical data can predict the behavioral trends of elderly individuals within the next 5 minutes. When the prediction indicates a high probability of risk, the system proactively initiates preventative measures, such as increasing monitoring frequency and preloading emergency resources. The prediction model employs the Prophet algorithm framework, considering multiple factors including seasonality, trends, and holiday effects.

[0070] The human-computer collaborative interaction interface has been upgraded to a multimodal interaction system. In addition to voice interaction, gesture recognition and eye tracking functions have been added. Gesture recognition uses a depth camera to collect hand skeleton information and supports real-time recognition of 10 basic gestures. Eye tracking is achieved through a near-infrared camera with a sampling rate of 120 Hz, capable of detecting gaze points and blink frequency. After multimodal information fusion, the system's accuracy in understanding the elderly person's intentions has improved by approximately 25 percentage points.

[0071] The navigation system of the robotic mobile platform incorporates semantic navigation capabilities. The map generated by the simultaneous localization and mapping (SMR) module contains rich semantic information, such as room type, furniture function, and activity area. Path planning considers not only geometric constraints but also semantic context, such as reducing movement speed in the bedroom area and increasing safe distance in the kitchen area. The semantic map is automatically labeled using a deep learning model, employing a semantic segmentation network pre-trained on an indoor scene dataset.

[0072] The power management system employs an intelligent scheduling algorithm. Based on the elderly person's activity patterns and system task priorities, it dynamically adjusts the power consumption of each module. During low-activity periods at night, it automatically enters energy-saving mode, shutting down unnecessary sensors and reducing processing frequency. When an emergency is detected, it immediately switches to high-performance mode, ensuring all resources are prioritized for emergency response.

[0073] The communication system incorporates multipath transmission technology. Important data is transmitted simultaneously via both Wi-Fi and mobile networks, employing an error recovery mechanism that combines forward error correction and automatic retransmission requests. The transmission protocol is optimized to a custom protocol based on QUIC, reducing connection establishment latency and improving data transmission efficiency. Network bandwidth is adaptively adjusted, dynamically adjusting video bitrate and image quality according to available bandwidth.

[0074] The system's reliability design incorporates multiple backup mechanisms. Key sensors are redundantly configured, automatically switching to backup sensors in case of primary sensor failure. The decision-making algorithm integrates multiple model versions, periodically running and comparing them in shadow mode to ensure smooth algorithm updates. All configuration parameters are saved in multiple historical versions, supporting rapid rollback. The system automatically performs daily integrity checks, using the SHA-256 algorithm to verify system file integrity.

[0075] The maintenance system is designed with a self-diagnostic and self-healing architecture. By monitoring various system metrics, including processor load, memory usage, disk space, and network latency, it automatically identifies potential problems. When an abnormal pattern is detected, corresponding repair processes are triggered, such as cleaning temporary files, restarting abnormal services, and optimizing database indexes. The system's health status is reported to a cloud monitoring center in real time, supporting remote diagnosis and intervention.

[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A camera-equipped elderly care robot capable of issuing emergency calls for help, characterized in that: include: A multimodal perception fusion unit is used to collect environmental and behavioral data of the elderly. The multimodal perception fusion unit includes a visual perception module, a millimeter-wave radar perception module, an acoustic perception module, and a data fusion processor. The visual perception module uses a combination of a color camera with wide dynamic range imaging capability and a low-light enhanced infrared thermal imager. The millimeter-wave radar perception module uses a frequency-modulated continuous wave radar system. The data fusion processor runs a feature-level fusion algorithm based on a deep neural network. The intent understanding and decision-making center is connected to the multimodal perception fusion unit and is used to receive multimodal behavior representation vectors and perform behavior recognition and intent inference. The intent understanding and decision-making center includes an abnormal behavior recognition engine and a fuzzy intent parsing module. The abnormal behavior recognition engine has a built-in behavior classifier based on a hybrid model of temporal convolutional network and long short-term memory network. The fuzzy intent parsing module runs an intent inference process based on multimodal contextual reasoning; An adaptive response execution unit, connected to the intent understanding and decision-making center, is used to execute corresponding response strategies based on the output of the intent understanding and decision-making center. The adaptive response execution unit includes a multi-level response strategy manager and a human-computer collaborative interaction interface. The multi-level response strategy manager predefines multiple levels of response strategies. The human-computer collaborative interaction interface integrates a natural language processing module and an emotion computing module. The robot mobile platform adopts an omnidirectional mobile chassis, and its drive system consists of three or more Mecanum wheels or omnidirectional wheels. The robot mobile platform integrates a synchronous positioning and map building module.

2. The elderly care robot with camera capable of issuing emergency calls as described in claim 1, characterized in that, The feature-level fusion algorithm based on deep neural networks running in the data fusion processor performs the following processing: synchronizing and aligning image features from the visual perception module, point cloud and micro-Doppler features from the millimeter-wave radar perception module, and acoustic event features from the acoustic perception module; inputting heterogeneous features into a multi-branch feature extraction network for deep feature extraction; and outputting a unified multimodal behavior representation vector through a feature fusion layer weighted by an attention mechanism.

3. The elderly care robot with camera capable of issuing emergency calls according to claim 2, characterized in that, The specific structural configuration of the multi-branch feature extraction network is as follows: for the vision branch, a deep residual network is used to extract the spatial features of the image; for the millimeter-wave radar branch, a one-dimensional convolutional network is used to process the radar signal spectrum to extract motion features; for the acoustic branch, a convolutional recurrent neural network is used to process the acoustic spectrum to extract acoustic event features. The attention-weighted feature fusion layer calculates the importance weight of each feature channel and concatenates the weighted features.

4. The elderly care robot with camera capable of issuing emergency calls as described in claim 1, characterized in that, The behavior classifier in the abnormal behavior recognition engine takes the time series of multimodal behavior representation vectors as input, captures local time patterns through a temporal convolutional network, and models long-term time dependencies through a long short-term memory network. Finally, it outputs a probability distribution corresponding to a preset abnormal behavior category, which includes at least falling behavior, prolonged stillness behavior, and violent struggling behavior.

5. The elderly care robot with camera capable of issuing emergency calls as described in claim 4, characterized in that, The training process of the behavior classifier in the abnormal behavior recognition engine adopts a hard sample mining strategy. This strategy dynamically identifies samples that are misclassified by the model or have low classification confidence during the model training process, and gives these samples higher loss weights in subsequent training iterations.

6. The elderly care robot with camera capable of issuing emergency calls according to claim 1, characterized in that, The intent inference process based on multimodal contextual reasoning, which runs in the fuzzy intent parsing module, performs the following steps: constructing a knowledge graph containing the elderly person's daily behavior patterns, current environmental context, and historical interaction records; combining real-time acquired multimodal perception data, performing inference calculations through a graph neural network; generating a probabilistic assessment of the elderly person's current possible intent, the types of intent being assessed including explicit verbal requests for help, nonverbal cues for help, and unconscious warning signs of dangerous behavior.

7. The elderly care robot with camera capable of issuing emergency calls according to claim 6, characterized in that, The knowledge graph includes elderly entities, furniture objects, daily activity events, and time segments. Edge relationships represent the strength of association and co-occurrence frequency between these entities. The reasoning process of the graph neural network updates the node representation through a message passing mechanism and finally converges on the node representing the elderly person's current intention for classification.

8. The elderly care robot with camera capable of issuing emergency calls according to claim 1, characterized in that, The multi-level response strategy manager predefines three levels of response strategies from low to high; the strategy switching logic of the multi-level response strategy manager is based on a dual threshold comparison mechanism; when the highest category probability output by the abnormal behavior recognition engine exceeds the preset high threshold, the third-level response strategy is directly triggered. When the probability of the highest category is between the preset low threshold and the high threshold, the second-level response strategy is triggered; when the abnormal behavior recognition engine does not exceed the low threshold but the intent probability evaluated by the fuzzy intent parsing module exceeds its own set intent threshold, the first-level response strategy is triggered.

9. The elderly care robot with camera capable of issuing emergency calls according to claim 1, characterized in that, The natural language processing module of the human-computer collaborative interaction interface integrates domain adaptation functionality. This functionality continuously learns the personalized vocabulary and expression habits used by the elderly in daily interactions and dynamically optimizes their language model. The emotion computing module analyzes the characteristics of the elderly's tone of voice, speech rate, and facial expressions to infer their emotional state and adjusts the tone and wording of the robot's response accordingly.

10. A camera-equipped elderly care robot capable of issuing emergency assistance as described in claim 1, characterized in that, The synchronous localization and mapping module of the robot mobile platform uses a method of fusion of lidar and visual odometry to build and update the home environment map in real time. When building the environment map, dynamic obstacles and static obstacles are distinguished, and only the information of static obstacles is incorporated into the long-term map.

Images