An autonomous patrol robot system based on multi-sensor fusion and intelligent anomaly recognition

Abstract

The application provides an autonomous patrol robot system based on multi-sensor fusion and intelligent anomaly recognition. By integrating multi-dimensional perception data such as vision, sound, and environment, it breaks through the limitations of traditional single sensors and significantly improves the accuracy and scene adaptability of anomaly recognition. For example, in personnel behavior recognition, the thermal imaging camera of the visual perception unit can capture human temperature characteristics at night or in smoky environments. Combined with the microphone array positioning of the sound perception unit, the gas sensor of the environmental monitoring unit detects the concentration of flammable gas. After data fusion, the AI anomaly recognition and behavior analysis module can more accurately determine whether there is an emergency situation of fire accompanied by trapped personnel, avoiding misjudgment caused by environmental interference of single visual or sound sensors.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an autonomous patrol robot system based on multi-sensor fusion and intelligent anomaly recognition. Background Technology

[0002] Currently, traditional autonomous patrol robots mostly rely on single sensors for environmental perception, making them susceptible to factors such as lighting, occlusion, and complex acoustic environments. This results in low accuracy in anomaly detection and high false alarm rates. For example, robots relying solely on visual sensors struggle to clearly capture details of human behavior at night or in bright light; single sound sensors are easily affected by background noise, failing to accurately identify critical abnormal sounds such as calls for help or unusual noises. Furthermore, most systems lack deep fusion mechanisms for multimodal data, processing data from each sensor independently, making it difficult to form complementary verification and resulting in insufficient robustness in complex scenarios. In addition, existing robot anomaly detection models are often designed for specific scenarios, exhibiting weak generalization capabilities, and their remote handling functions are limited, supporting only video feedback and lacking real-time voice interaction and remote intervention, leading to delayed responses to abnormal events. Summary of the Invention

[0003] The purpose of this application is to at least address one of the aforementioned technical deficiencies, particularly the low accuracy of anomaly identification, high false alarm and missed alarm rates, weak generalization ability, and delayed response to abnormal events in existing autonomous patrol robots.

[0004] This application provides an autonomous patrol robot system based on multi-sensor fusion and intelligent anomaly recognition, the system comprising:

[0005] The autonomous navigation and path planning module is used to realize environmental mapping, localization, path planning and dynamic obstacle avoidance based on multi-sensor data;

[0006] The multi-sensor fusion sensing module includes a visual sensing unit, a sound sensing unit, and an environmental monitoring unit, which is used to collect and fuse visual, sound, and environmental data;

[0007] The AI ​​anomaly recognition and behavior analysis module is used to intelligently analyze the fused multimodal data to identify personnel behavior, abnormal objects, and fire incidents.

[0008] The two-way voice intercom and remote handling module supports real-time voice communication and remote announcements between the monitoring center and the robot.

[0009] The automatic charging and energy management module is used for robot power monitoring, automatic navigation charging, and multi-robot collaborative charging scheduling.

[0010] The cloud-based management and data analysis module is used for robot status monitoring, task scheduling, abnormal event recording, and data analysis.

[0011] Optionally, the autonomous navigation and path planning module includes:

[0012] The mobile chassis features four-wheel differential drive and is equipped with lidar, depth camera, ultrasonic sensor, IMU, and wheeled odometer.

[0013] SLAM mapping units are used to construct 2D raster maps and 3D semantic maps, and to annotate key points;

[0014] The positioning unit uses the AMCL algorithm combined with Kalman filtering for multi-sensor fusion positioning.

[0015] The path planning unit uses the A* algorithm for global path planning and combines it with the DWA algorithm to achieve dynamic obstacle avoidance.

[0016] Optionally, the visual perception unit includes:

[0017] High-definition camera, infrared night vision camera and thermal imaging camera, supporting day and night mode switching;

[0018] Edge AI chip for real-time video analysis with latency of less than 100ms;

[0019] It supports personnel detection, behavior recognition, and visual recognition of flames and smoke.

[0020] Optionally, the sound sensing unit includes:

[0021] Microphone arrays are used to pick up ambient sounds and locate sound sources;

[0022] The audio processing unit supports echo cancellation, noise reduction, and beamforming.

[0023] An abnormal sound recognition model, based on a 1D-CNN+LSTM model, identifies cries for help, fighting, destruction, explosions, and abnormal machine noises.

[0024] Optionally, the environmental monitoring unit includes:

[0025] Gas sensors are used to detect smoke, combustible gases, carbon monoxide, and toxic gases.

[0026] Temperature and humidity sensors are used to monitor ambient temperature and humidity.

[0027] The monitoring strategy unit triggers multi-level alarms based on preset thresholds and pushes abnormal information.

[0028] Optionally, the AI ​​anomaly recognition and behavior analysis module includes:

[0029] The personnel behavior recognition unit, based on YOLOv8 and DeepSORT, realizes target detection and tracking, and identifies behaviors such as intrusion, loitering, fighting, falling and climbing.

[0030] The item anomaly detection unit is used to identify abandoned items, moved items, and damaged facilities;

[0031] The fire detection unit combines visual flame and smoke recognition with sensor data to achieve multi-source verification.

[0032] Optionally, the two-way voice intercom and remote handling module includes:

[0033] Real-time voice transmission unit supports two-way communication between the monitoring center and the robot;

[0034] Preset voice warning unit, supports one-click playback of preset warning content;

[0035] The emergency call button, when triggered, automatically connects to the monitoring center and pushes on-site video and location information.

[0036] Optionally, the automatic charging and energy management module includes:

[0037] The charging station supports magnetic contact docking and fast / slow charging modes.

[0038] The robot-side charging management unit is equipped with an infrared alignment sensor and a charging management chip.

[0039] The intelligent charging strategy unit automatically schedules charging tasks based on the battery status.

[0040] The multi-robot collaborative scheduling unit supports task allocation and charging queuing for multiple robots.

[0041] Optionally, the cloud management and data analysis module includes:

[0042] The real-time monitoring platform supports electronic map display, video wall display, alarm list and remote control;

[0043] The mobile app supports notifications for abnormal events, remote monitoring, voice intercom, and playback of past events.

[0044] The data analysis unit is used for statistical analysis of patrol data, distribution of abnormal events, false alarm rate analysis, and trend prediction.

[0045] Optionally, the data analysis unit further includes:

[0046] The patrol heatmap generation function is used to visually display the distribution of patrol frequency;

[0047] The report generation function supports exporting daily, weekly, and monthly reports.

[0048] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0049] This application provides an autonomous patrol robot system based on multi-sensor fusion and intelligent anomaly recognition. By integrating multi-dimensional perception data such as vision, sound, and environment, it overcomes the limitations of traditional single sensors, significantly improving the accuracy and scene adaptability of anomaly recognition. For example, in terms of human behavior recognition, the thermal imaging camera of the visual perception unit can capture human body temperature characteristics at night or in dense smoke environments. Combined with the microphone array of the sound perception unit to locate abnormal sound sources, and the gas sensor of the environmental monitoring unit to detect the concentration of combustible gases, the AI ​​anomaly recognition and behavior analysis module can more accurately determine whether there is an emergency situation of fire accompanied by trapped personnel, avoiding misjudgments caused by environmental interference from a single visual or sound sensor. At the same time, the autonomous navigation and path planning module of this application adopts multi-sensor fusion positioning. Even in complex scenarios where the lidar is blocked, stable positioning can still be achieved through the complementarity of IMU and wheeled odometer, ensuring the continuity of the patrol path. In addition, the cloud management and data analysis module of this application can perform statistical analysis and trend prediction of abnormal events. For example, if the patrol heat map reveals that items are frequently left behind in a certain area, the system can automatically adjust the patrol frequency of that area to achieve proactive security. The two-way voice intercom and remote response module solves the problem of slow response in traditional systems. The monitoring center can use the robot to disperse suspicious persons in real time or guide on-site personnel in emergency handling, significantly improving the efficiency of handling abnormal events. Compared with existing technologies, this system forms a complete closed loop in multimodal data fusion, intelligent analysis, and remote interaction, providing a more reliable and efficient autonomous patrol solution for various security scenarios. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 An architecture diagram of an autonomous patrol robot system based on multi-sensor fusion and intelligent anomaly recognition is provided for an embodiment of this application;

[0052] Figure 2 This is a schematic diagram of the structure of the AI ​​anomaly recognition and behavior analysis module provided in an embodiment of this application;

[0053] Figure 3This is a schematic diagram of the structure of the two-way voice intercom and remote processing module provided in an embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the automatic charging and energy management module provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] In one embodiment, such as Figure 1 As shown, Figure 1 This application provides an architecture diagram of an autonomous patrol robot system based on multi-sensor fusion and intelligent anomaly recognition. The system may include:

[0057] The autonomous navigation and path planning module is used to achieve environmental mapping, localization, path planning, and dynamic obstacle avoidance based on multi-sensor data.

[0058] The multi-sensor fusion sensing module includes a visual sensing unit, a sound sensing unit, and an environmental monitoring unit, which is used to collect and fuse visual, sound, and environmental data.

[0059] The AI ​​anomaly recognition and behavior analysis module is used to intelligently analyze the fused multimodal data to identify abnormal personnel behavior, objects, and fire incidents.

[0060] The two-way voice intercom and remote handling module supports real-time voice communication and remote announcements between the monitoring center and the robot.

[0061] The automatic charging and energy management module is used for robot power monitoring, automatic navigation charging, and multi-robot collaborative charging scheduling.

[0062] The cloud-based management and data analysis module is used for robot status monitoring, task scheduling, abnormal event recording, and data analysis.

[0063] In this embodiment, the autonomous patrol robot system may include an autonomous navigation and path planning module, a multi-sensor fusion perception module, an AI anomaly recognition and behavior analysis module, a two-way voice intercom and remote handling module, an automatic charging and energy management module, and a cloud management and data analysis module.

[0064] The autonomous navigation and path planning module can perform environmental mapping, localization, path planning, and dynamic obstacle avoidance based on multi-sensor data. For example, the autonomous patrol robot of this application can conduct security patrols in places such as prisons, factories, parks, and warehouses. During this process, the autonomous navigation and path planning module can collect environmental data through sensors such as LiDAR and depth cameras pre-installed on the robot, constructing a 2D grid map and a 3D semantic map covering the patrol area, accurately marking key points such as entrances, equipment rooms, and fire exits. When the robot performs patrol tasks, it can combine the 2D grid map and the 3D semantic map for accurate localization, and can generate a globally optimal path from the current location to the target patrol point through relevant algorithms. At the same time, it can adjust its movement trajectory in real time through relevant algorithms to avoid dynamic obstacles such as pedestrians and stacked objects, ensuring the smoothness and efficiency of the patrol process.

[0065] The multi-sensor fusion sensing module of this application works collaboratively through three types of units: vision, sound, and environment. The vision sensing unit is equipped with a high-definition camera, an infrared night vision camera, and a thermal imaging camera. It can switch between corresponding modes in bright daylight, dark night, or dense smoke environments, and complete personnel detection, behavior recognition, and visual recognition of flames and smoke. The sound sensing unit can pick up ambient sounds and locate the sound source. After the audio processing unit eliminates echoes and reduces noise, it can identify abnormal sounds such as calls for help, fighting, and facility damage. The gas sensor of the environmental monitoring unit can detect smoke, combustible gases, etc., and the temperature and humidity sensor can collect environmental data in real time. The monitoring strategy unit can trigger a three-level alarm based on preset thresholds. When the temperature exceeds the normal range but does not reach the danger value, a yellow warning is pushed. When a combustible gas leak is detected, an orange warning is pushed. At the same time, the vision and sound units are triggered for secondary verification. If a fire is confirmed, a red warning is pushed and the on-site data is synchronized to the monitoring center.

[0066] Furthermore, the AI ​​anomaly recognition and behavior analysis module of this application can perform in-depth fusion analysis on the above-mentioned multimodal data. This analysis process includes, but is not limited to, three core links: personnel behavior recognition, object anomaly recognition, and fire recognition. In the personnel behavior recognition unit, the system can perform target detection on real-time video frames collected by the visual perception unit and continuously track relevant personnel. When it detects personnel entering restricted areas (such as key points marked "equipment rooms"), lingering in sensitive areas for more than 15 minutes, or engaging in physical conflict or falling, it immediately marks them as abnormal events. The object anomaly recognition unit can identify unreported abandoned packages, moved fire-fighting equipment, or damaged access control equipment by comparing visual data at different time points based on the layout of fixed facilities and objects pre-recorded in the 3D semantic map. The fire recognition unit can build a multi-source verification mechanism. When the visual perception unit detects flame or smoke features, it simultaneously retrieves smoke concentration and temperature data from the environmental monitoring unit and abnormal sounds (such as explosions and alarms) from the sound perception unit. If all three types of data trigger preset thresholds, it is determined to be a real fire event, avoiding visual false alarms caused by strong light reflection, steam, and other factors.

[0067] The two-way voice intercom and remote handling module of this application supports real-time voice communication and remote announcements between the monitoring center and the robot. Its core consists of a real-time voice transmission unit, a preset voice warning unit, and an emergency call button. The real-time voice transmission unit can use low-latency coding technology to ensure that the voice latency between the monitoring center and the robot is controlled within a preset duration, achieving "zero-lag" two-way dialogue. The preset voice warning unit can be equipped with more than 10 scenario-based voice packages such as "Please leave the restricted area immediately" and "Fire exit must not be blocked." The monitoring center can trigger playback with one click through the cloud management platform to provide initial warnings without the need for manual real-time announcements. The emergency call button is integrated in a conspicuous position on the robot body. When on-site personnel discover an anomaly, pressing the button will automatically connect to the monitoring center and simultaneously push the robot's current location coordinates, real-time video footage, and surrounding environmental data, making it easy for monitoring personnel to quickly grasp the situation on-site and formulate a handling plan.

[0068] The automatic charging and energy management module ensures the robot's 24 / 7 uninterrupted patrol. Specifically, the robot can be equipped with a power monitoring chip to collect real-time data on battery voltage, current, and remaining power. When the power level falls below a preset threshold, a charging request is automatically triggered. Upon receiving the request, the cloud management and data analysis module plans the optimal charging path for the robot based on the current patrol progress and the availability of charging stations. The robot can identify the magnetic contacts of the charging station using an infrared alignment sensor, achieving centimeter-level precise docking. For multi-robot collaborative scenarios, this application can employ a priority queue algorithm to prioritize charging robots with lower power levels or those undertaking important patrol tasks, avoiding charging station resource conflicts. Furthermore, this unit can also develop personalized charging strategies for each robot based on historical charging data and battery health status. For example, for robots that have been in use for more than one year, the fast charging time can be appropriately shortened to reduce the risk of battery degradation.

[0069] The cloud management and data analysis module, serving as the system's "central brain," integrates four major functions: real-time monitoring, task scheduling, abnormal event recording, and data analysis. The real-time monitoring platform uses an electronic map to intuitively display the location, battery level, patrol status, and current alarm information of all robots, supporting simultaneous display of real-time footage from multiple robots on a video wall. The task scheduling unit can create task types such as scheduled patrols and fixed-point patrols according to user needs. For example, setting up fixed-point patrols around the park perimeter from 0:00 to 6:00 daily allows the system to automatically assign tasks to idle robots and track task progress in real time. The abnormal event recording unit archives the entire process of each abnormal event, including trigger time, location, type, handling process, and result, generating event data packages containing video clips, audio files, and environmental data, supporting quick retrieval by time, region, and event type. The data analysis unit, based on big data mining technology, can statistically analyze the distribution patterns of abnormal events weekly. For example, it was found that abnormal events involving the movement of goods in the warehouse area accounted for 35% on Monday mornings. Further analysis of video data confirmed that the issue stemmed from untimely reporting of material transfers during employee shift changes. The system then immediately pushed optimization suggestions to the administrator, reminding them to improve the material transfer reporting process.

[0070] In summary, the autonomous patrol robot system of this application breaks through the limitations of single-modality perception by multi-sensor fusion perception, achieves complementary verification of multi-source data by using AI anomaly recognition, shortens response time by relying on two-way voice and remote handling modules, ensures continuous operation by combining automatic charging and energy management, and finally forms a complete closed loop of "perception-analysis-handling-optimization" through cloud management and data analysis, providing a more intelligent and efficient security solution for diverse scenarios such as prisons, factories, parks, and warehouses.

[0071] In one embodiment, the autonomous navigation and path planning module may include:

[0072] The mobile chassis features four-wheel differential drive and is equipped with lidar, depth camera, ultrasonic sensors, IMU, and wheeled odometer.

[0073] SLAM mapping units are used to construct 2D raster maps and 3D semantic maps, and to label key points.

[0074] The positioning unit uses the AMCL algorithm combined with Kalman filtering for multi-sensor fusion positioning.

[0075] The path planning unit uses the A* algorithm for global path planning and combines it with the DWA algorithm to achieve dynamic obstacle avoidance.

[0076] In this embodiment, the robot's mobile chassis features a four-wheel differential drive structure that enables in-situ turning and lateral translation, with a maximum speed of 1.2 m / s, a climbing angle of ≤15°, a battery life of 12 hours (48V / 100Ah lithium battery), IP65 protection, dimensions of 600×500×1200 mm, and a weight of ≤80 kg, adapting to the turning requirements of narrow passages. Furthermore, the robot of this application is equipped with a lidar, depth camera, ultrasonic sensor, IMU, and wheeled odometer.

[0077] The lidar can perform 360° scanning (range 0.1-30m, accuracy ±2cm) and acquire environmental point cloud data within a preset distance range at a preset scanning frequency. The depth camera can simultaneously collect RGB-D information, and the ultrasonic sensor can supplement the detection of nearby obstacles at a distance of 0.1-3m. All three, along with the nine-axis attitude data of the IMU and the pulse counting data of the wheel odometer, are transmitted to the positioning unit in real time.

[0078] When the robot first patrols, the SLAM mapping unit of this application can automatically mark different types of key points such as "high voltage power distribution room (no entry)", "fire hydrant (equipment point)" and "north gate of the park (entrance and exit)" through point cloud matching of LiDAR and semantic segmentation of depth camera. The constructed 3D semantic map can distinguish between dynamic and static targets such as "fixed wall", "moving shelf" and "pedestrian".

[0079] The positioning unit of this application can use the AMCL algorithm to initialize the particle filter distribution, and combine the absolute positioning of the lidar with the relative positioning data of the IMU and wheel odometer by Kalman filtering. Even in the corner of the warehouse where the lidar is blocked by the container, the positioning error can still be controlled within ±5cm.

[0080] The path planning unit of this application can use the A* algorithm to plan and calculate the optimal path for patrol routes and locations set by the administrator. It supports multiple preset routes (day shift / night shift / key routes) and supports timed automatic or manual triggering. Furthermore, the path planning unit of this application can also be combined with the DWA algorithm to achieve dynamic obstacle avoidance: this application can detect obstacles in real time through LiDAR and depth camera, and the DWA algorithm can avoid obstacles in real time (response time <0.1 seconds). When a moving obstacle is detected, it automatically decelerates / stops / goes around it, with a minimum passage width of 700mm. When a person is detected, it automatically decelerates to 0.3m / s and stops at a distance of <0.5m. In addition, this application can be equipped with 4 drop sensors on the bottom of the robot to prevent falls, and the surrounding anti-collision rubber strips and pressure sensors can make the robot automatically retreat after a collision.

[0081] In one specific implementation, when the robot enters the warehouse to perform nighttime fixed-point patrol tasks, the SLAM mapping unit has pre-built a 3D semantic map that marks information such as "Shelf A (fixed facility)," "Fire exit (must-pass path)," and "Electrical distribution box (key location)." The mobile chassis travels smoothly in a four-wheel differential drive mode, while the LiDAR acquires point cloud data in real time within a range of 0.1-30m at a scanning frequency of 10Hz. The depth camera simultaneously collects RGB-D information, and the ultrasonic sensor performs close-range supplementary detection of cardboard boxes at a distance of 0.2m between shelves. At this time, the localization unit initializes a distribution of 1000 particle filters using the AMCL algorithm, fusing the absolute positioning data from the LiDAR, the nine-axis attitude data from the IMU, and the pulse count data from the wheeled odometer. Even in the southwest corner of the warehouse where the LiDAR signal is blocked by a 2.5m high container, the localization error can still be controlled within ±4cm. The path planning unit, based on the preset "night shift key patrol route," uses the A* algorithm to plan the globally optimal path from the warehouse entrance, through shelf A, the electrical distribution box, to the fire escape, with a total length of 120m and an estimated time of 180 seconds. When the robot reaches the narrow passage (750mm wide) between shelves B and C, the LiDAR detects a 1.2m high moving cardboard box (dynamic obstacle) suddenly appearing in the center of the passage. The DWA algorithm recalculates the trajectory within 0.08 seconds, controlling the robot to decelerate at 0.5m / s and detour around the box, maintaining a safe distance of 0.3m. Simultaneously, the IMU monitors the robot's posture in real time, ensuring that the tilt angle does not exceed 5° during the turn. When the robot approaches the electrical distribution box (a key location), the depth camera detects that a person has lingered in front of the box for more than 10 seconds. The mobile chassis automatically decelerates to 0.3m / s and comes to a complete stop when it is 0.4m away from the person, triggering the visual perception unit to reconfirm the person's behavior. After the personnel left, the path planning unit replanned the local path and continued to complete the fixed-point inspection task of the power distribution box. In addition, the four drop sensors on the bottom of the robot monitored changes in ground height in real time. When passing a 0.1m high step at the warehouse entrance, the sensors detected the height difference and triggered the chassis lifting mechanism (achieved by adjusting the four-wheel suspension system) to ensure that the robot passed smoothly. When the anti-collision rubber strips around the body accidentally collided with the corner of the shelf, the pressure sensor immediately fed back a signal, and the robot automatically retreated 0.5m and adjusted its travel direction to avoid secondary collisions.

[0082] In one embodiment, the visual sensing unit may include:

[0083] It features a high-definition camera, an infrared night vision camera, and a thermal imaging camera, and supports switching between daytime and nighttime modes.

[0084] Edge AI chip for real-time video analytics with latency of less than 100ms.

[0085] It supports personnel detection, behavior recognition, and visual recognition of flames and smoke.

[0086] In this embodiment, the high-definition camera can use an 8-megapixel CMOS sensor, paired with a 120° wide-angle lens and autofocus, to clearly capture facial features of people and details of equipment within a 15m range. In bright daylight, dynamic exposure compensation technology suppresses overexposure, ensuring the integrity of texture information within the monitored area. The infrared night vision camera has a built-in 940nm red-exposure-free infrared fill light with a fill distance of up to 20m. In low-light conditions, it switches to black-and-white night vision mode, with noise controlled to within 0.02%, and can identify the basic movement trajectory of people at 5m. The thermal imaging camera can use an infrared detector with a resolution of 384×288 and a thermal sensitivity of ≤50mK. It can identify heat sources hidden behind smoke by temperature distribution differences. For example, in a dense smoke environment, even if the high-definition camera cannot image due to insufficient visibility, the thermal imaging camera can still accurately locate human heat sources or abnormal heat points of equipment within a 6m range. The edge AI chip can be an embedded neural network processor with a computing power of 8 TOPS, with a built-in lightweight YOLOv8 object detection model and behavior recognition model. The processing latency for real-time video frames is controlled within 85ms: the personnel detection module can track more than 10 dynamic targets at the same time with an accuracy of 98.5%; the behavior recognition module extracts key points of the human skeleton (such as the head, torso, and limb joints) and analyzes the action sequence of 15 consecutive video frames, with an accuracy of 96% in identifying abnormal behaviors such as "wandering", "falling", and "climbing"; the flame and smoke visual recognition module, through training a CNN model containing 100,000+ labeled samples (covering scenes with different lighting, smoke concentration, and flame shapes), can detect flame or smoke areas accounting for ≥0.5% of the screen within 0.1 seconds, with a false alarm rate of less than 0.3%.

[0087] In a specific application scenario, when the robot is performing patrol tasks in the clean area of ​​a pharmaceutical factory, the visual perception unit automatically switches modes according to the ambient light: During the day, from 8:00 to 18:00, a high-definition camera is activated to capture the proper wearing of protective clothing by personnel at the entrance of the clean area. The target detection model identifies violations such as "not wearing a mask" and "protective clothing zipper not zipped up," and immediately marks and synchronizes the identification to the monitoring center; at night, from 18:00 to 8:00 the next day, it switches to an infrared night vision camera to monitor whether there are unauthorized personnel entering the clean area. When the infrared camera captures a moving target, the edge AI chip completes personnel detection and tracking within 90ms, and at the same time triggers the thermal imaging camera for auxiliary verification. If the thermal imaging shows that the target is a human heat source (temperature 36-37℃), an abnormality warning is pushed to the monitoring center. If a slight smoke is generated in a cleanroom equipment room due to a circuit malfunction, the high-definition camera will first detect the pale white smoke feature in the image. This will then trigger the thermal imaging camera to detect the equipment temperature. When the thermal imaging shows that the surface temperature of the equipment reaches 180℃ (exceeding the preset threshold of 120℃), the visual perception unit will synchronize the smoke and high-temperature data to the AI ​​anomaly recognition module, providing basic data support for subsequent multi-source verification. Furthermore, the edge AI chip also supports local storage of key video clips (1080P resolution, 25fps) from the past three days. When the network is interrupted, the video data is automatically cached to the local SD card and synchronized to the cloud after the network is restored, ensuring no data loss.

[0088] In one embodiment, the sound sensing unit may include:

[0089] Microphone arrays are used to pick up ambient sounds and locate sound sources.

[0090] The audio processing unit supports echo cancellation, noise reduction, and beamforming.

[0091] An abnormal sound recognition model, based on a 1D-CNN+LSTM model, identifies cries for help, fighting, destruction, explosions, and abnormal machine noises.

[0092] In this embodiment, the microphone array can adopt an 8-microphone ring layout to achieve 360° omnidirectional sound pickup, with a pickup range covering 0.5-20m. The sound source is located using the TDOA (Time Difference of Arrival) algorithm, with a positioning accuracy of ±10°. The audio processing unit has a built-in adaptive echo cancellation module, which can eliminate the robot's own motion noise (such as motor running noise and fan noise) and environmental background noise (such as air conditioning fan noise and low-frequency noise from equipment operation), with a noise reduction of up to 35dB. At the same time, beamforming technology is used to focus on the target sound source, improving the signal-to-noise ratio by 15dB and ensuring clear acquisition of weak sounds at long distances. The abnormal sound recognition model is built on a hybrid architecture of 1D-CNN and LSTM: the 1D-CNN layer is responsible for extracting the temporal features (such as energy and zero-crossing rate) and frequency features (such as Mel-frequency cepstral coefficients, MFCC) of the sound signal, while the LSTM layer captures the temporal dependencies of the sound signal (such as the continuous syllables of a cry for help and the periodicity of abnormal machine noises). The model training samples cover more than 50,000 labeled audio segments, including cries for help (such as "Help!" or "Is anyone there?"), fighting sounds (such as physical collisions and shouts), destructive sounds (such as glass breaking and metal knocking), explosion sounds (such as the shock wave sound of a small explosion), and abnormal machine noises (such as the "buzzing" sound of bearing wear and the "humming" sound of a motor with a missing phase). The model has an accuracy of 95% in recognizing abnormal sounds, a response time of ≤0.5 seconds, and a false alarm rate of less than 0.8%.

[0093] In a specific application scenario, when the robot is performing nighttime patrols in a prison area, the sound perception unit is in a continuous monitoring state. The microphone array picks up the ambient sound of the prison corridor in real time, and the audio processing unit first eliminates the low-frequency noise (50-100Hz) of the corridor ventilation system and the operating noise of the robot's own motors (150-200Hz), and then focuses the sound source at the end of the corridor through beamforming. When the sound of breaking glass (destructive sound) is heard from a cell in the prison area, the microphone array locates the sound source 12 meters directly in front of the robot (cell number 302) within 0.3 seconds. The audio processing unit transmits the noise-reduced breaking sound signal to the abnormal sound recognition model. The model extracts the high-frequency impact features (2000-5000Hz) of breaking glass through a 1D-CNN layer, and the LSTM layer identifies the transient nature and energy peak characteristics of the sound. Within 0.4 seconds, the abnormal type of "breaking glass" is determined, and the visual perception unit is simultaneously triggered to turn towards the sound source. The high-definition camera (switching to infrared mode at night) immediately points at the window area of ​​cell 302, capturing the image of the broken window glass, achieving multi-source verification of sound and vision. At the same time, the sound perception unit packages the audio clip of breaking glass (5 seconds in length) and the location data and pushes it to the cloud management module, triggering the abnormal event recording and alarm process.

[0094] In addition, when the robot patrols the factory workshop, the sound perception unit detects a periodic "buzzing" sound (abnormal machine noise) emanating from a certain motor. The microphone array locates the source of the sound as the injection molding machine (number M-08) on the east side of the workshop. The audio processing unit can extract the frequency characteristics of the abnormal noise (800-1000Hz, periodic interval of 0.2 seconds). The abnormal sound recognition model, combined with historical machine noise data, determines the fault type as "bearing wear" and pushes the fault warning to the workshop administrator. The administrator can retrieve the real-time video footage and sound clips of the injection molding machine through the cloud platform, arrange equipment maintenance in advance, and prevent the fault from escalating.

[0095] In one embodiment, the environmental monitoring unit may include:

[0096] Gas sensors are used to detect smoke, combustible gases, carbon monoxide, and toxic gases.

[0097] Temperature and humidity sensors are used to monitor ambient temperature and humidity.

[0098] The monitoring strategy unit triggers multi-level alarms based on preset thresholds and pushes abnormal information.

[0099] In this embodiment, the gas sensor adopts a modular design, integrating a smoke sensor (ionization + photoelectric dual detection), a combustible gas sensor (catalytic combustion type, detection range 0-100% LEL), a carbon monoxide sensor (electrochemical, detection range 0-500 ppm), and a toxic gas sensor (such as hydrogen sulfide electrochemical sensor, detection range 0-100 ppm). Each sensor has a response time ≤3 seconds and an accuracy better than ±5%FS. The temperature and humidity sensor uses a digital integrated chip, with a temperature detection range of -40℃ to 85℃ (accuracy ±0.3℃) and a humidity detection range of 0-100%RH (accuracy ±2%RH). The sampling frequency can reach 1Hz, and it supports temperature compensation to eliminate the influence of temperature drift in humidity measurement. The monitoring strategy unit incorporates multi-level threshold logic: for example, smoke sensors have preset "early warning threshold (smoke concentration ≥ 0.1 mg / m³)" and "alarm threshold (smoke concentration ≥ 0.3 mg / m³)"; combustible gases have preset "low alarm threshold (10% LEL)" and "high alarm threshold (25% LEL)"; temperature and humidity are set with specific thresholds for different scenarios, such as preset temperature of 18-26℃ and humidity of 45-65%RH for clean areas in pharmaceutical factories, and preset temperature of 0-30℃ and humidity of 30-70%RH for warehouses. The monitoring strategy unit can automatically load corresponding threshold templates according to different scenarios. When a parameter exceeds the early warning threshold, it triggers a local audio-visual prompt on the robot (flashing yellow warning light + low-volume buzzer) and records the data; when it exceeds the alarm threshold, it immediately triggers a solid red warning light + high-decibel buzzer (80dB), and simultaneously synchronizes abnormal data (including sensor type, detection value, timestamp, and robot position) to the monitoring center, supporting linkage verification with vision and sound units.

[0100] In a specific application scenario, when the robot is performing a patrol mission in the storage tank area of ​​a chemical industrial park, the environmental monitoring unit collects data in real time: the combustible gas sensor in the gas sensor detects that the methane concentration in the B3 area of ​​the storage tank area reaches 12% LEL (exceeding the low reporting threshold of 10% LEL), with a response time of 2.5 seconds, which immediately triggers the first-level warning of the monitoring strategy unit. The robot's local yellow warning light flashes, and at the same time, the methane concentration data and location coordinates (storage tank area B3, latitude 31°24′12″N, longitude 120°58′36″E) are pushed to the monitoring center; as the robot continues to approach the B3 storage tank, the methane concentration rises to 28% LEL (exceeding the high reporting threshold of 25% LEL), and the monitoring strategy unit immediately triggers the second-level alarm. The red warning light stays on and emits an 85dB buzzer. At the same time, the optical sensing unit is triggered to turn to the B3 storage tank, the infrared camera captures the faint leakage traces at the storage tank valve, and the sound sensing unit focuses on the "hissing" airflow sound at the leakage point. After the data of the three are integrated, the leakage event is confirmed, and the monitoring center receives multi-source abnormality reports simultaneously and initiates the emergency response process. If smoke is generated in a corner of the tank area due to spontaneous combustion of accumulated materials, the ionization detection module of the smoke sensor will first detect smoke particles (concentration 0.2mg / m³), the photoelectric module will assist in the verification, the monitoring strategy unit will trigger a smoke warning, and at the same time the environmental monitoring unit will link with the thermal imaging camera to confirm the location and temperature of the heat source (the spontaneous combustion temperature reaches 320℃), thus achieving dual verification by gas and vision.

[0101] In addition, when the robot patrols the archives, the temperature and humidity sensors detect that the temperature in the storage room rises to 28°C (exceeding the preset threshold of 25°C) and the humidity rises to 65%RH (exceeding the preset threshold of 60%RH). The monitoring strategy unit triggers an abnormal temperature and humidity warning and pushes the information to the archives administrator. The administrator can remotely adjust the storage room's air conditioning system through the cloud platform. If a carbon monoxide leak occurs in a certain area of ​​the storage room due to aging circuits, the carbon monoxide sensor detects a concentration of 30ppm (exceeding the warning threshold of 25ppm), immediately triggering an alarm and linking with the visual perception unit to focus on the circuit equipment near the leak point to help locate the source of the fault.

[0102] In one embodiment, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the structure of the AI ​​anomaly recognition and behavior analysis module provided in an embodiment of this application; the AI ​​anomaly recognition and behavior analysis module may include:

[0103] The personnel behavior recognition unit, based on YOLOv8 and DeepSORT, realizes target detection and tracking, and identifies behaviors such as intrusion, loitering, fighting, falling and climbing.

[0104] The item anomaly identification unit is used to identify abandoned items, moved items, and damaged facilities.

[0105] The fire detection unit combines visual flame and smoke recognition with sensor data to achieve multi-source verification.

[0106] In this embodiment, the personnel behavior recognition unit can adopt a three-level architecture of "detection-tracking-analysis": The front end uses the YOLOv8 model to perform target detection on the video frames collected by the visual perception unit, and outputs the bounding box and confidence score of the personnel (≥0.8 is considered a valid target). Then, the DeepSORT algorithm is used to achieve cross-frame tracking based on the appearance features of the target (such as clothing color and body shape) and motion trajectory (such as speed and direction). The tracking ID is maintained for ≥30 seconds. Even if the target is briefly occluded (≤2 seconds), it can be re-matched to ensure the continuity of the target. The back-end behavior analysis subunit extracts the target motion features (such as displacement and angular velocity) and skeletal key point features (the coordinate sequence of 17 joints is extracted through MediaPipe) of 20 consecutive frames, constructs a behavior feature vector, and inputs it into a pre-trained behavior classification model (based on the Transformer architecture, the training samples contain 200,000+ labeled behavior sequences). Finally, the behavior type and confidence score are output.

[0107] Among them, "intrusion" behavior recognition can be achieved through preset electronic fence areas (such as factory restricted areas or prison boundaries). When the bounding box of the tracked target overlaps with the electronic fence area by ≥30% for a duration of ≥5 seconds, it is judged as an intrusion, with an accuracy rate of 97%. "Lingering" behavior can be recognized by analyzing the target's dwell time (≥120 seconds) and movement trajectory complexity (trajectory overlap rate ≥60%) within a 10m×10m area, with an accuracy rate of 95.5%. "Fighting" behavior is identified by detecting the overlap of skeletal key points (≥40%), limb movement amplitude (such as arm swing angle ≥120°), and relative speed of two or more targets in consecutive frames. The accuracy rate for identifying "falling" behavior is 96.2% by monitoring changes in the target's vertical height (descent ≥50%) and the angles of key skeletal points (e.g., hip angle ≤90°, knee angle ≤60°). The response time is ≤1.2 seconds, and the accuracy rate is 97.8%. The accuracy rate for identifying "climbing" behavior is 95.8% by analyzing the distance between the target and vertical objects (e.g., walls, railings) (≤0.5m) and limb movements (e.g., hand key points are higher than head).

[0108] The object anomaly recognition unit incorporates two modules: object detection and background modeling. The object detection module, based on the Faster R-CNN model, is pre-trained with annotations on common objects in the scene (such as office desks and chairs, equipment cabinets, and fire extinguishers), achieving a 98% accuracy rate in recognizing common objects. The background modeling module uses a Gaussian Mixture Model (GMM) to construct the scene background in real time, updating the background template every 30 minutes. When a new target not in the common object database (such as a backpack or cardboard box) is detected and remains in the scene for ≥60 seconds, it is identified as an "abandoned object." When the positional offset of a common object is ≥0.5m (such as a moved fire hydrant or rotated surveillance camera), it is identified as an "object moved." When the appearance integrity index (calculated based on an edge detection algorithm, with a normal value ≥0.9) is ≤0.6 (such as broken glass or damaged door locks), it is identified as "facility damaged." This unit achieves a 94% accuracy rate in recognizing abandoned objects, a 95% accuracy rate in recognizing moved objects, and a 96% accuracy rate in recognizing facility damaged objects, with a false alarm rate of less than 0.5%.

[0109] The fire detection unit employs a "multi-source data fusion verification" mechanism: First, it receives flame and smoke recognition results (including area location, area percentage, and confidence level) from the visual perception unit and gas sensor data (smoke concentration and combustible gas concentration) from the environmental monitoring unit. Then, a weighted fusion algorithm is used to calculate the overall confidence level—the visual recognition confidence level has a weight of 0.6, and the sensor data confidence level has a weight of 0.4 (if the smoke concentration is ≥0.3mg / m³ or the combustible gas concentration is ≥25%LEL, the sensor confidence level is set to 1; otherwise, it is linearly mapped according to the concentration value). When the overall confidence level is ≥0.85, it is determined to be a fire event; if the overall confidence level is between 0.7 and 0.85, a secondary verification process is triggered: the robot is controlled to move within 5m of the suspected fire area, and visual and sensor data are collected again to recalculate the overall confidence level; if the overall confidence level is still <0.8, it is determined to be a false alarm. This unit achieves a fire detection accuracy of 98.5%, a false alarm rate of less than 0.2%, and a response time of ≤1.5 seconds.

[0110] In a specific application scenario, when the robot is patrolling the underground parking lot of a large shopping mall, the AI ​​anomaly recognition and behavior analysis module simultaneously processes multi-source data: the high-definition camera of the visual perception unit captures a person lingering at the fire exit in Zone B of the parking lot for more than 150 seconds. The personnel behavior recognition unit tracks the person's movement trajectory using DeepSORT (trajectory overlap rate reaches 65%), and combines it with skeletal key point analysis (small limb movements, no obvious intention to move), determining it as "loitering" behavior, immediately marking it and pushing it to the mall's monitoring center; at the same time, the object anomaly recognition unit discovers a new black backpack next to a pillar in Zone B (staying for ≥70 seconds) through background modeling. If the item is not in the regular inventory, it is identified as an "abandoned item." This triggers the visual perception unit to focus on the backpack area, magnifying details (such as the logo and shape on the backpack surface) through a high-definition camera, and simultaneously pushing the data to the monitoring center. If the smoke sensor of the environmental monitoring unit detects a concentration of 0.4 mg / m³, and the flame recognition module of the visual perception unit detects a flame area accounting for 1.2% in the corner of area B, the fire recognition unit calculates the comprehensive confidence level (visual confidence level 0.9 + sensor confidence level 1) × 0.5 = 0.95, determines it as a fire event, immediately triggers an alarm, and links the robot to move towards the fire area. At the same time, it pushes multi-source data to the fire control center and initiates the emergency evacuation process.

[0111] In addition, when the robot is patrolling the school campus, the human behavior recognition unit detects a student climbing the school wall. YOLOv8 detects that the student's bounding box overlaps with the wall area by 40%. DeepSORT tracks continuous frames of the student's climbing action (skeletal key points show that the hands are higher than the head and the body is perpendicular to the wall), which is determined to be "climbing" behavior. It immediately triggers an audible and visual alarm and notifies the campus security personnel. The object anomaly recognition unit detects that the emergency light at the entrance of the teaching building has been moved (the position offset is 0.8m), which is determined to be "object movement". The information is pushed to the logistics department to assist in timely repositioning.

[0112] In one embodiment, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of the two-way voice intercom and remote handling module provided in the embodiments of this application; the two-way voice intercom and remote handling module may include:

[0113] The real-time voice transmission unit supports two-way communication between the monitoring center and the robot.

[0114] Preset voice warning unit, supports one-click playback of preset warning content.

[0115] The emergency call button, when triggered, automatically connects to the monitoring center and pushes on-site video and location information.

[0116] In this embodiment, the real-time voice transmission unit can adopt a 4G / 5G dual-mode communication module, supporting high-definition voice encoding (sampling rate 16kHz, encoding format OPUS), with voice latency controlled within 0.2 seconds. Even in weak signal environments (such as underground parking garages and enclosed workshops), it can ensure smooth voice communication through adaptive bit rate adjustment. The module has built-in echo cancellation and noise suppression algorithms, which can filter background noise (such as machine noise and vehicle horn sounds), improving the signal-to-noise ratio to over 45dB, ensuring clear communication between the two parties. The preset voice warning unit can have multiple scene-based voice templates built-in, covering content such as "Do not approach the restricted area," "Please leave immediately," and "No smoking here." Administrators can customize and add or modify voice files (MP3 / WAV format, duration ≤10 seconds) through the cloud platform. When the robot detects abnormal behavior (such as intrusion or loitering), the administrator can trigger the corresponding warning voice with one click in the monitoring center. The playback volume can be adjusted in multiple levels (such as 60dB / 70dB / 80dB) to adapt to the sound environment requirements of different scenarios. The emergency call button adopts a physical press-type design (IP65 protection level) and is installed in a conspicuous position on the top of the robot. When on-site personnel discover an abnormality, pressing the button will immediately activate the "emergency mode" of the robot: automatically connect to the priority communication channel of the monitoring center, and simultaneously push real-time video footage (1080P resolution, 25fps frame rate), current GPS positioning (accuracy ≤1 meter) and environmental monitoring data (such as gas concentration, temperature and humidity) to the monitoring center. At the same time, it will trigger the local red warning light to stay on and a high-decibel buzzer (85dB) to continuously remind people in the vicinity to pay attention.

[0117] In a specific application scenario, when a robot is patrolling a prison area, the human behavior recognition unit detects an inmate attempting to approach the prison area fence (intrude into the electronic fence area). The monitoring center administrator speaks directly to the inmate via the real-time voice transmission unit: "Please immediately return to the designated area, otherwise coercive measures will be taken." At the same time, the preset voice warning unit is triggered to play a repeated warning: "The prison area fence is a restricted area; approaching is prohibited." If the inmate still does not stop, the administrator can control the robot to move in that direction (with the speed adjusted to 0.5 m / s) via the remote handling module, maintaining continuous voice warnings and video monitoring. If a patrolling prison guard discovers an abnormal sound in a certain area, they can press the robot's emergency call button. The monitoring center immediately receives real-time video and location information for that area. The guard reports to the monitoring center via voice intercom: "There is an abnormal sound in the warehouse on the northwest side of the prison area; it is suspected that some items have collapsed." The monitoring center then dispatches other robots to provide support.

[0118] Furthermore, when the robot patrols the hospital inpatient department, if the environmental monitoring unit detects a slight increase in smoke concentration in a ward corridor (0.2 mg / m³, below the alarm threshold), the monitoring center administrator communicates with the medical staff in that ward via the real-time voice transmission unit: "Please confirm whether there is smoking in the ward; the smoke concentration in the corridor is abnormal." If the medical staff reports that "disinfectant spray is being used," the administrator can adjust the monitoring threshold for that area through the remote handling module to avoid false alarms. When the robot patrols the bank lobby, if the item anomaly recognition unit detects a suitcase left next to an ATM, the monitoring center administrator reminds the lobby security guard via the real-time voice transmission unit: "There is an item left on the north side of the ATM; please go and check it immediately." At the same time, a close-up image and location coordinates of the suitcase are pushed to the monitoring center. After arriving at the scene, the security guard can use the robot's voice intercom function to report to the monitoring center: "The item is a document bag left behind by a customer; it has been properly kept."

[0119] In one embodiment, such as Figure 4 As shown, Figure 4 This is a schematic diagram of the structure of the automatic charging and energy management module provided in the embodiments of this application; the automatic charging and energy management module may include:

[0120] The charging station supports magnetic contact docking and fast / slow charging modes.

[0121] The robot's charging management unit is equipped with an infrared alignment sensor and a charging management chip.

[0122] The intelligent charging strategy unit automatically schedules charging tasks based on the battery status.

[0123] The multi-robot collaborative scheduling unit supports task allocation and charging queuing for multiple robots.

[0124] In this embodiment, the charging pile can adopt an integrated waterproof and dustproof design (protection level IP67), which is suitable for various indoor and outdoor installation scenarios. Its magnetic charging contacts can be gold-plated, with a contact resistance of ≤5mΩ, and support more than 100,000 insertion and removal cycles. In fast charging mode, the output power can reach 60W, which can charge the robot battery from 20% to 80% in 30 minutes. In slow charging mode, the output power is 15W, which is suitable for trickle charging during low-load periods at night and extends the battery cycle life (≥2000 cycles). The robot's charging management unit can integrate multiple infrared alignment sensors (horizontal angle ±30°, vertical angle ±15°). When the robot's battery level is lower than a preset threshold, such as 20%, the intelligent charging strategy unit can trigger a "low battery return" command. The robot plans the optimal return path based on a global map built using SLAM (avoiding obstacles and congested areas). When it approaches the preset range of the charging station, the infrared alignment sensors can capture the infrared beacon of the charging station and adjust the robot's heading angle and position deviation through a PID algorithm (adjustment accuracy ≤2cm) to achieve automatic docking of the magnetic contacts, with a docking success rate of 99.8%. The charging management chip can monitor the battery's voltage (accuracy ±0.01V), current (accuracy ±0.1A), and temperature (accuracy ±0.5℃) in real time. When the battery temperature exceeds 45℃, it automatically switches to slow charging mode. When the temperature exceeds 50℃, charging is paused and the cooling fan is started (speed 3000rpm). Charging resumes when the temperature drops below 40℃ to avoid battery overheating and damage.

[0125] The intelligent charging strategy unit is based on a two-dimensional scheduling approach: "power prediction - task priority". First, a power consumption prediction model is trained using historical patrol data (input parameters include patrol area, obstacle density, and number of activated sensors; output is the power consumption rate per unit time). When a robot performs a high-priority task (such as fire alarm linkage), even if the battery level is below 20%, it will prioritize completing the task before returning to charge. If the battery is about to run out during task execution (estimated remaining power ≤ 5%), "emergency nearby charging" is triggered, automatically searching for the nearest backup charging station (distance ≤ 50 meters). The multi-robot collaborative scheduling unit manages charging resources based on a distributed queue algorithm: When multiple robots request charging simultaneously, the unit sorts them according to their remaining power (lower power, higher priority) and task urgency (e.g., robots handling abnormal events have higher priority than idle robots), generating a charging queue and pushing it to each robot. Robots waiting in the queue automatically remain in the "waiting area" (a pre-set radius of 2 meters) near the charging station. After the previous robot finishes charging (power ≥ 95%), it automatically enters the charging position, avoiding charging resource conflicts. The queue scheduling response time is ≤ 0.5 seconds.

[0126] In a specific application scenario, an industrial park deployed 5 autonomous patrol robots and 3 charging piles. When a carbon monoxide leak occurred in area A of the park (the environmental monitoring unit detected a concentration of 35 ppm), Robot 1, which was patrolling area B (with 22% battery remaining), received an emergency response instruction. The intelligent charging strategy unit assessed the urgency of the task (high priority) and allowed Robot 1 to go to area A to handle the situation first. The visual perception unit located the leak point and pushed the data to the park's safety monitoring center. After the leak point was confirmed and the ventilation system was activated, Robot 1's remaining battery dropped to 15%, triggering a low-battery return trip. The multi-robot collaborative scheduling unit showed that charging pile 1 was available, charging pile 2 was occupied by Robot 2 (10 minutes of charging time remaining), and charging pile 3 was occupied by Robot 3 (5 minutes of charging time remaining). Based on the remaining battery levels (Robot 1 15% remaining < Robot 4 18% remaining < Robot 5 25% remaining), Robot 1 was assigned to the charging area of ​​charging pile 3. After 5 minutes, Robot 3 finished charging and left. Robot 1 automatically docked with charging pile 3 and started fast charging. After 30 minutes, the battery level reached 85%, and Robot 1 returned to its patrol route. If robot 4 (with 18% battery remaining) requests charging at this time, the multi-machine collaborative scheduling unit will allocate it to charging pile 1 to achieve efficient utilization of charging resources.

[0127] Furthermore, when robots patrol a 24-hour logistics warehouse, the intelligent charging strategy unit adjusts the charging plan according to the warehouse's peak operating hours (8:00-20:00): during peak hours, all robots are in patrol mode, triggering emergency charging only when the battery level drops below 10%; during off-peak hours (20:00-8:00 the next day), robots sequentially enter charging piles for slow charging, ensuring that the battery level is ≥90% during the next day's peak hours. The multi-robot collaborative scheduling unit also supports automatic detection of charging pile faults: when a charging pile cannot supply power normally, the unit immediately marks it as "faulty" and reassigns the charging queue to other available charging piles, while simultaneously pushing fault information to the mobile devices of maintenance personnel to remind them to repair in a timely manner, ensuring the continuous operation of the robots.

[0128] In one embodiment, the cloud management and data analysis module may include:

[0129] The real-time monitoring platform supports electronic map display, video wall display, alarm list, and remote control.

[0130] The mobile app supports notifications for abnormal events, remote monitoring, voice communication, and playback of past events.

[0131] The data analysis unit is used for statistical analysis of patrol data, distribution of abnormal events, false alarm rate analysis, and trend prediction.

[0132] In this embodiment, the real-time monitoring platform can construct a 3D visualization interface based on WebGL technology. It can integrate a global environment map generated by robot SLAM, on which the location coordinates, power status, current task type, and abnormal event trigger points of each robot are marked in real time. The video wall supports split-screen display of real-time video feeds from more than 16 robots. Clicking on any robot icon switches to the robot's main view, while a dynamic floating window of environmental monitoring data (such as temperature, humidity, gas concentration, and noise levels) is overlaid. Administrators can adjust the screen layout by dragging, or zoom in on key areas (up to 4K resolution) and save screenshots. The platform's alarm list can adopt a hierarchical display mechanism, using different colors (red, orange, and blue) to mark the urgency of abnormal events (e.g., fire and intrusion are level 1 alarms, loitering and leaving items are level 2 alarms, and moving items are level 3 alarms). Each alarm message includes the event type, occurrence time, robot ID, location description, and associated data (such as on-site photos and sensor values). Administrators can click on alarm entries to view event details and trigger remote handling commands (such as voice warnings and robot movement).

[0133] The mobile app supports multiple systems including Android, iOS, and HarmonyOS. Its lightweight design ensures smooth operation in weak network environments. Abnormal event push notifications can use a dual approach of "message push + voice broadcast". After receiving the push notification, the administrator can view real-time video, retrieve video clips 10 seconds before and after the event, or directly initiate two-way voice communication with the robot through the app. The history playback function supports filtering video records by time, robot ID, and event type. Video files can be compressed using H.265 encoding (compression ratio up to 10:1) to save cloud storage resources, while also supporting offline download to local storage.

[0134] The data analysis unit incorporates multi-dimensional statistical models: the patrol data statistics module can generate "daily / weekly / monthly patrol coverage reports," calculating the percentage of patrol time in each area (e.g., the weekly patrol coverage of area B in the shopping mall's underground parking lot reaches 98%), and the average robot movement speed (e.g., the average patrol speed of robots in an industrial park is 0.8 m / s); the abnormal event distribution module uses heat maps to display high-incidence areas of events (e.g., the east side of the school wall is a high-incidence area for climbing behavior, marked as a red heat point), and counts the frequency of occurrence of each type of event (e.g., there were 12 incidents of items being moved in the hospital's inpatient department in a month); false alarm rate is also analyzed. The analysis module can link the judgment results of abnormal events with the records of manual review, calculate the reasons for false alarms of each identification unit (such as false alarms of the fire identification unit are mostly due to steam interference with the sensor), and generate suggestions for optimizing the false alarm rate (such as adjusting the sensor threshold in a steam environment); the trend prediction module is based on the LSTM neural network model, inputting abnormal event data and environmental parameters (such as seasonal changes and holiday traffic) for the past 3 months, and predicting the peak time of events in the next month (such as the peak time for left-behind items in shopping malls on weekends from 14:00 to 16:00), assisting administrators in adjusting the robot patrol routes and frequencies in advance.

[0135] For example, the data analysis unit of a large logistics warehouse discovered through trend prediction that the number of items left behind in warehouse area A was expected to increase by 30% during the "Double 11" period. It then suggested increasing the frequency of robot patrols in this area from once per hour to once every 30 minutes, and at the same time increasing the background model update frequency of the item anomaly identification unit (from once every 5 minutes to once every 2 minutes), which effectively reduced the false negative rate of anomalies during the "Double 11" period.

[0136] In one embodiment, the data analysis unit may further include:

[0137] The patrol heatmap generation function is used to visually display the distribution of patrol frequency.

[0138] The report generation function supports exporting daily, weekly, and monthly reports.

[0139] In this embodiment, the patrol heatmap generation function supports overlay display by time dimension (hour / day / week) and spatial dimension (area / floor). Based on the robot's GPS positioning data and patrol trajectory points, a gradient color scale (e.g., blue for low frequency, yellow for medium frequency, and red for high frequency) is used to render the global map. Simultaneously, markers for abnormal event trigger points can be overlaid (e.g., triangles for intrusion events and circles for left-behind items), intuitively presenting the correlation between "patrol blind spots" and "high-incidence areas of abnormalities." For example, the fire escape area on the basement floor of a shopping mall has a low patrol frequency (blue area), with an average of 2 left-behind incidents per month. Administrators can quickly locate this blind spot using the heatmap, adjust the robot's patrol route, and increase the number of fixed-point patrols to 3 per day. Furthermore, the heatmap supports custom filtering conditions, such as displaying only patrol trajectories from "22:00 to 6:00 the next day," or focusing on the distribution of abnormal events in the "warehouse cold chain area," providing a visual basis for refined patrol strategy adjustments.

[0140] The automatic report generation function has built-in standardized templates and custom configuration options. The standardized daily report includes the total patrol time for the day, the operating status of each robot (such as online time and charging times), statistics on abnormal events (such as 2 first-level alarms and 5 second-level alarms), false alarm rate data (such as a false alarm rate of 1.2% for the day), and environmental monitoring extreme values ​​(such as a maximum temperature of 32℃ and a minimum humidity of 40%). The weekly report adds week-on-week analysis (such as a 15% decrease in intrusion incidents this week compared to last week) and a patrol coverage trend chart (such as an increase in the weekly patrol coverage rate of office building A area from 92% to 96%) based on the daily report. The monthly report integrates key monthly indicators (such as a monthly average patrol coverage rate of 95% and an average false alarm rate of 1.0%), a ranking of high-incidence event types (such as the proportion of abandoned items incidents being 35%), and optimization suggestions (such as adjusting the threshold for identifying people loitering in a certain area). Customizable configuration allows administrators to select the metrics to be included in the report (such as only counting the operation data of a specific robot), set the data display format (such as bar charts and line charts), and export to PDF, Excel, CSV and other formats. The report generation time is ≤10 seconds, and it is automatically pushed to the administrator's email or mobile app to meet the reporting needs of different scenarios.

[0141] In one specific application, a property manager of an office building discovered that the "personnel loitering" level-two alarm on the 10th floor of the building averaged 8 per month when exporting monthly reports using the automatic report generation function, with a false alarm rate as high as 3.5%. Analysis of the false alarm causes in the reports revealed that all were misidentified as cleaning staff organizing tools in the corridor at night. The manager then adjusted the personnel loitering identification rules for that area through the cloud management platform: extending the loitering judgment time from 10:00 PM to 6:00 AM the next day from 15 seconds to 30 seconds, and adding a "cleaning tool recognition" auxiliary condition. After the adjustment, the false alarm rate for that area dropped to 0.8% the following month, significantly improving alarm effectiveness. Simultaneously, the automatic report generation function automatically pushes weekly reports to the property manager's email every Friday at 5:00 PM, allowing the manager to quickly grasp the week's patrol situation without manual data aggregation, greatly improving management efficiency.

[0142] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0143] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0144] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An autonomous patrolling robot system based on multi-sensor fusion and intelligent anomaly recognition, characterized in that, The system includes: The autonomous navigation and path planning module is used to realize environmental mapping, localization, path planning and dynamic obstacle avoidance based on multi-sensor data; The multi-sensor fusion sensing module includes a visual sensing unit, a sound sensing unit, and an environmental monitoring unit, which is used to collect and fuse visual, sound, and environmental data; The AI ​​anomaly recognition and behavior analysis module is used to intelligently analyze the fused multimodal data to identify personnel behavior, abnormal objects, and fire incidents. The two-way voice intercom and remote handling module supports real-time voice communication and remote announcements between the monitoring center and the robot. The automatic charging and energy management module is used for robot power monitoring, automatic navigation charging, and multi-robot collaborative charging scheduling. The cloud-based management and data analysis module is used for robot status monitoring, task scheduling, abnormal event recording, and data analysis.

2. The system of claim 1, wherein, The autonomous navigation and path planning module includes: The mobile chassis features four-wheel differential drive and is equipped with lidar, depth camera, ultrasonic sensor, IMU, and wheeled odometer. SLAM mapping units are used to construct 2D raster maps and 3D semantic maps, and to annotate key points; The positioning unit uses the AMCL algorithm combined with Kalman filtering for multi-sensor fusion positioning. The path planning unit uses the A* algorithm for global path planning and combines it with the DWA algorithm to achieve dynamic obstacle avoidance.

3. The system of claim 1, wherein, The visual perception unit includes: High-definition camera, infrared night vision camera and thermal imaging camera, supporting day and night mode switching; Edge AI chip for real-time video analysis with latency of less than 100ms; It supports personnel detection, behavior recognition, and visual recognition of flames and smoke.

4. The system of claim 1, wherein, The sound sensing unit includes: Microphone arrays are used to pick up ambient sounds and locate sound sources; The audio processing unit supports echo cancellation, noise reduction, and beamforming. An abnormal sound recognition model, based on a 1D-CNN+LSTM model, identifies cries for help, fighting, destruction, explosions, and abnormal machine noises.

5. The system of claim 1, wherein, The environmental monitoring unit includes: Gas sensors are used to detect smoke, combustible gases, carbon monoxide, and toxic gases. Temperature and humidity sensors are used to monitor ambient temperature and humidity. The monitoring strategy unit triggers multi-level alarms based on preset thresholds and pushes abnormal information.

6. The system of claim 1, wherein, The AI ​​anomaly detection and behavior analysis module includes: The personnel behavior recognition unit, based on YOLOv8 and DeepSORT, realizes target detection and tracking, and identifies behaviors such as intrusion, loitering, fighting, falling and climbing. The item anomaly detection unit is used to identify abandoned items, moved items, and damaged facilities; The fire detection unit combines visual flame and smoke recognition with sensor data to achieve multi-source verification.

7. The system of claim 1, wherein, The two-way voice intercom and remote processing module includes: Real-time voice transmission unit supports two-way communication between the monitoring center and the robot; Preset voice warning unit, supports one-click playback of preset warning content; The emergency call button, when triggered, automatically connects to the monitoring center and pushes on-site video and location information.

8. The system of claim 1, wherein, The automatic charging and energy management module includes: The charging station supports magnetic contact docking and fast / slow charging modes. The robot-side charging management unit is equipped with an infrared alignment sensor and a charging management chip. The intelligent charging strategy unit automatically schedules charging tasks based on the battery status. The multi-robot collaborative scheduling unit supports task allocation and charging queuing for multiple robots.

9. The system of claim 1, wherein, The cloud management and data analysis module includes: The real-time monitoring platform supports electronic map display, video wall display, alarm list and remote control; The mobile app supports notifications for abnormal events, remote monitoring, voice intercom, and playback of past events. The data analysis unit is used for statistical analysis of patrol data, distribution of abnormal events, false alarm rate analysis, and trend prediction.

10. The system of claim 9, wherein, The data analysis unit also includes: The patrol heatmap generation function is used to visually display the distribution of patrol frequency; The report generation function supports exporting daily, weekly, and monthly reports.

Images