A multi-modal missing adaptive abnormal behavior detection system, method and robot based on a patrol robot

Abstract

The application discloses a kind of multi-modal missing adaptive abnormal behavior detection system, method and inspection robot based on inspection robot, solve the technical problem that existing multi-modal abnormal behavior detection system leads to poor robustness of abnormal behavior detection.System is connected by multi-modal data acquisition module, data processing module, modal state detection module, missing modal adaptive processing module and multi-modal abnormal behavior detection module in turn communication connection.Constitute.Multi-modal data acquisition module acquires multi-modal data, data processing module is preprocessed to obtain preprocessed multi-modal feature, modal state detection module carries out quality assessment and generates modal state information, missing modal adaptive processing module determines missing type and then executes feature compensation or invalid modal rejection, and multi-modal abnormal behavior detection module dynamically configures fusion weight, calculates behavior confidence score and compares preset abnormality determination threshold to output detection result.

Description

Technical Field

[0001] This invention relates to the field of intelligent inspection and safety monitoring technology, and in particular to a multimodal missing adaptive abnormal behavior detection system, method and inspection robot based on an inspection robot. Background Technology

[0002] With the continuous development of artificial intelligence, mobile robot technology, and intelligent sensing technology, inspection robots have been widely used in complex scenarios such as power line inspection, industrial parks, rail transit, warehousing and logistics, and public safety. They replace manual labor in tasks such as environmental inspection, equipment monitoring, and abnormal behavior identification. Compared to traditional manual inspection methods, inspection robots offer advantages such as all-weather operation, autonomous navigation, real-time monitoring, and strong adaptability to hazardous areas, effectively improving inspection efficiency and safety.

[0003] In actual inspection processes, inspection robots typically utilize various sensing devices such as visual cameras, audio sensors, and robot status sensors to collect and analyze real-time data on personnel behavior, equipment operating status, and environmental information within the inspection area. Among these, abnormal behavior detection, as a crucial component of intelligent inspection systems, is of great significance for public safety early warning, equipment risk identification, and handling of abnormal events.

[0004] Existing multimodal abnormal behavior detection systems generally lack differentiated compensation capabilities for scenarios where different modalities are missing. When local visual information is missing, auxiliary modalities are abnormal, or the primary visual modality fails, the system usually cannot dynamically switch detection strategies, nor can it utilize multi-view visual information and auxiliary modalities for collaborative compensation, resulting in poor robustness of abnormal behavior detection. Summary of the Invention

[0005] This invention provides a multimodal missing adaptive abnormal behavior detection system, method, and inspection robot based on an inspection robot, which solves the technical problem of poor robustness of existing multimodal abnormal behavior detection systems.

[0006] The first aspect of the present invention provides a multimodal missing adaptive abnormal behavior detection system based on an inspection robot, the system comprising a multimodal data acquisition module, a data processing module, a modal state detection module, a missing modality adaptive processing module, and a multimodal abnormal behavior detection module that are sequentially and communicatively connected.

[0007] The multimodal data acquisition module is used to acquire multimodal data;

[0008] The data processing module is used to preprocess the multimodal data to obtain preprocessed multimodal features;

[0009] The modal state detection module is used to perform multimodal data quality assessment on the preprocessed multimodal features and generate modal state information;

[0010] The missing modality adaptive processing module is used to determine the current modality missing type based on the modality state information, and selectively perform feature compensation or invalid modality removal on the preprocessed multimodal features based on the current modality missing type to obtain multimodal features adapted to the current perception state.

[0011] The multimodal abnormal behavior detection module is used to dynamically configure the multimodal feature fusion weights based on the modal state information, calculate the behavior confidence score by combining the multimodal features adapted to the current perception state, compare the behavior confidence score with the preset abnormal judgment threshold, and output the abnormal behavior detection result.

[0012] Optionally, the multimodal data includes visual data, audio data, and robot state data; the multimodal data acquisition module includes a visual acquisition unit, an audio acquisition unit, and a robot state acquisition unit.

[0013] The visual acquisition unit, audio acquisition unit, and robot status acquisition unit are all connected to the data processing module.

[0014] The visual acquisition unit is used to acquire the visual data and transmit it to the data processing module;

[0015] The audio acquisition unit is used to acquire audio data and transmit it to the data processing module;

[0016] The robot status acquisition unit is used to acquire the robot status data and transmit it to the data processing module.

[0017] Optionally, the data processing module is specifically used for:

[0018] The multimodal data is cleaned to obtain cleaned multimodal data.

[0019] The cleaned multimodal data is time-aligned to obtain time-aligned multimodal data.

[0020] Spatially align the time-aligned multimodal data to obtain spatially aligned multimodal data;

[0021] Feature extraction is performed on the spatially aligned multimodal data to obtain preprocessed multimodal features.

[0022] Optionally, the preprocessed multimodal features include visual features, audio features, and human posture features; the current modality missing type includes local visual modality missing, audio-assisted modality missing, human posture-assisted modality missing, and normal modality; the missing modality adaptive processing module is specifically used for:

[0023] Read the modal quality score from the modal state information. The modal quality score includes visual modal quality score, audio modal quality score, and human posture modal quality score. The visual modal quality score includes overall visual quality score and visual sub-regional quality score.

[0024] The overall visual quality score is compared with a preset main modality failure threshold. If the overall visual quality score is greater than or equal to the preset main modality failure threshold, the visual sub-region quality score is compared with a preset local missing threshold one by one. If at least one visual sub-region quality score is less than the preset local missing threshold, it is determined to be a local visual modality missing. The multi-view visual features in the preprocessed multimodal features are used to complement the visual features of the corresponding missing region in the multimodal features, and audio features are fused for auxiliary compensation to obtain multimodal features adapted to the current perception state.

[0025] If the visual sub-region quality scores are all greater than or equal to the preset local missing threshold, the audio modality quality score is compared with the preset auxiliary modality effective threshold; if the audio modality quality score is less than the preset auxiliary modality effective threshold, it is determined that the audio auxiliary modality is missing, and the audio features are removed from the preprocessed multimodal features, while the visual features and human posture features are retained to obtain multimodal features that are adapted to the current perception state.

[0026] If the audio modal quality score is greater than or equal to the preset auxiliary modal effective threshold, the human posture modal quality score is further compared with the preset posture effective threshold; if the human posture modal quality score is less than the preset posture effective threshold, it is determined that the human posture auxiliary modality is missing, and the human posture features are removed from the preprocessed multimodal features, while the visual and audio features are retained to obtain multimodal features that are adapted to the current perception state.

[0027] If the human posture modal quality score is greater than or equal to the preset effective posture threshold, it is determined that the modality is normal, and the preprocessed multimodal features are used as multimodal features to adapt to the current perception state.

[0028] Optionally, the current modality loss type also includes complete loss of the main visual focus; the missing modality adaptive processing module is further configured to:

[0029] If the overall visual quality score is less than the preset master mode failure threshold, it is determined that the master vision is completely missing, abnormal behavior detection is stopped, and a device abnormality alarm message is output.

[0030] Optionally, the system further includes an inspection path planning and dynamic adjustment module;

[0031] The inspection path planning and dynamic adjustment module is communicatively connected to the modal state detection module and the missing modality adaptive processing module, respectively.

[0032] The inspection path planning and dynamic adjustment module is used to calculate the modal quality cost based on the modal quality score in the modal state information when receiving equipment abnormality alarm information. It also combines the path distance parameters and environmental traversal complexity parameters stored in the preset inspection environment map, substitutes them into the cost function of the preset improved A* algorithm to complete the path optimization, generates the optimized inspection path and supplementary acquisition control instructions, and sends the supplementary acquisition control instructions to the motion execution end of the inspection robot to drive the inspection robot to perform supplementary data acquisition.

[0033] The second aspect of this invention provides a multimodal missing adaptive abnormal behavior detection method based on an inspection robot, comprising:

[0034] Collect multimodal data and preprocess the multimodal data to obtain preprocessed multimodal features;

[0035] Based on the preprocessed multimodal features, perform quality assessment of each modality and generate modal state information;

[0036] The current modality missing type is determined based on the modality state information, and feature compensation or invalid modality removal is selectively performed on the preprocessed multimodal features based on the current modality missing type to obtain multimodal features adapted to the current perception state.

[0037] Based on the modal state information, the multimodal feature fusion weights are dynamically configured, and the behavior confidence score is calculated by combining the multimodal features adapted to the current perception state. After comparing the behavior confidence score with the preset anomaly judgment threshold, the abnormal behavior detection result is output.

[0038] The third aspect of the present invention provides an inspection robot, comprising a mobile chassis, a main body, and a robot head connected in sequence;

[0039] The robot's head is equipped with a vision acquisition unit, a microphone array, and an audible and visual alarm. The vision acquisition unit includes an RGB camera, an infrared camera, and a depth camera.

[0040] The bottom of the mobile chassis is equipped with omnidirectional wheels, and the mobile chassis has a built-in lidar and motor status detection component;

[0041] A communication module is provided on the side of the main body of the fuselage, and a control computing unit is built into the main body of the fuselage;

[0042] The control computing unit is electrically connected to the vision acquisition unit, the microphone array, the audible and visual alarm, and the mobile chassis, respectively.

[0043] The control computing unit is used to receive multimodal data and preprocess it, and then upload the preprocessed multimodal features to the edge intelligent processing and decision-making center of the peripheral device through the communication module.

[0044] The peripheral edge intelligence processing and decision-making center is equipped with a multimodal missing adaptive abnormal behavior detection system based on the inspection robot, as described above.

[0045] The fourth aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the multimodal missing adaptive abnormal behavior detection method based on the inspection robot described above.

[0046] The fifth aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, it implements the multimodal missing adaptive abnormal behavior detection method based on an inspection robot as described above.

[0047] As can be seen from the above technical solutions, the present invention has the following advantages:

[0048] The first aspect of the technical solution of the present invention provides a multimodal missing adaptive abnormal behavior detection system based on an inspection robot. The system includes a multimodal data acquisition module, a data processing module, a modality state detection module, a missing modality adaptive processing module, and a multimodal abnormal behavior detection module, which are sequentially and communicatively connected. The multimodal data acquisition module is used to acquire multimodal data. The data processing module is used to preprocess the multimodal data to obtain preprocessed multimodal features. The modality state detection module is used to perform multimodal data quality assessment on the preprocessed multimodal features and generate modality state information. The missing modality adaptive processing module is used to determine the current modality missing type based on the modality state information, and selectively perform feature compensation or invalid modality removal on the preprocessed multimodal features based on the current modality missing type to obtain an adaptive modality. The system employs multimodal features of the current perception state and a multimodal abnormal behavior detection module. This module dynamically configures multimodal feature fusion weights based on modal state information and calculates behavior confidence scores by combining these with the multimodal features adapted to the current perception state. The behavior confidence scores are then compared with preset anomaly detection thresholds to output abnormal behavior detection results. Based on this approach, the modal state detection module perceives the data quality status of each modality in real time. A missing modality adaptive processing module then performs differentiated feature compensation or invalid modality removal for different scenarios such as local visual loss and auxiliary modality failure. Combined with a dynamically weighted multimodal fusion detection mechanism, this system can continuously output stable and reliable abnormal behavior detection results in complex scenarios where modal quality fluctuates or is partially missing due to occlusion, noise interference, sensor anomalies, etc., significantly improving the robustness of abnormal behavior detection.

[0049] The second aspect of the above-mentioned technical solution of the present invention provides a method for detecting multimodal missing adaptive abnormal behavior based on an inspection robot. This method involves collecting multimodal data and preprocessing the data to obtain preprocessed multimodal features; conducting modal quality assessments based on the preprocessed multimodal features to generate modal state information; determining the current modal missing type based on the modal state information; selectively performing feature compensation or invalid modal removal on the preprocessed multimodal features based on the current modal missing type to obtain multimodal features adapted to the current perception state; dynamically configuring multimodal feature fusion weights based on the modal state information; calculating behavior confidence scores by combining the multimodal features adapted to the current perception state; comparing the behavior confidence scores with a preset abnormality judgment threshold; and outputting the abnormal behavior detection result. Based on the above solution, by embedding modal quality assessment and adaptive compensation steps into the detection process, the feature combination and fusion strategy can be dynamically adjusted according to the real-time perception state, enabling abnormal behavior detection without relying on fixed and complete multimodal inputs, thereby improving the robustness of abnormal behavior detection.

[0050] The third aspect of the above-mentioned technical solution of the present invention provides an inspection robot, including a mobile chassis, a main body, and a robot head connected in sequence; the robot head is equipped with a vision acquisition unit, a microphone array, and an audible and visual alarm; the vision acquisition unit includes an RGB camera, an infrared camera, and a depth camera; the bottom of the mobile chassis is provided with omnidirectional wheels, and the mobile chassis has a built-in lidar and motor status detection component; a communication module is provided on the side of the main body, and the main body has a built-in control computing unit; the control computing unit is electrically connected to the vision acquisition unit, the microphone array, the audible and visual alarm, and the mobile chassis; the control computing unit is equipped with a multimodal missing adaptive abnormal behavior detection system; based on the above solution, the inspection robot completes multimodal data acquisition through multiple types of sensing hardware, and relies on the multimodal missing adaptive abnormal behavior detection system on the control computing unit to evaluate the quality of each modality data in real time and adaptively adjust the detection strategy during autonomous inspection, so as to stably complete abnormal behavior detection in complex inspection environments such as changes in lighting, partial occlusion, and signal interference, thereby improving the robustness of abnormal behavior detection. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a schematic diagram of the structure of a multimodal missing adaptive abnormal behavior detection system based on an inspection robot, provided in Embodiment 1 of the present invention.

[0053] Figure 2 This is a flowchart of the missing mode adaptive processing provided in Embodiment 1 of the present invention;

[0054] Figure 3 This is a flowchart of the inspection path planning and AR inspection information display provided in Embodiment 1 of the present invention;

[0055] Figure 4 This is a flowchart illustrating a multimodal missing adaptive abnormal behavior detection method based on an inspection robot, provided in Embodiment 2 of the present invention.

[0056] Figure 5 This is a flowchart of the multimodal missing adaptive abnormal behavior detection method provided in Embodiment 2 of the present invention;

[0057] Figure 6 This is a schematic diagram of a patrol robot based on Embodiment 3 of the present invention;

[0058] Figure 7 This is a diagram of the overall architecture of the software system provided in Embodiment 3 of the present invention;

[0059] Figure 8 This is a diagram of the ROS system interaction provided in Embodiment 3 of the present invention;

[0060] The meanings of the labels in the attached figures are as follows:

[0061] 1. Robot head; 2. Communication module; 3. Main body; 4. Mobile chassis. Detailed Implementation

[0062] This invention provides a multimodal missing adaptive abnormal behavior detection system, method, and inspection robot based on an inspection robot, which solves the technical problem of poor robustness of existing multimodal abnormal behavior detection systems.

[0063] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. It should be noted that in the optional embodiments of the present invention, the object information and other related data involved require the permission or consent of the object when the embodiments of the present invention are applied to specific products or technologies, and the collection, use, and processing of related data must comply with relevant laws, regulations, and standards. That is to say, if the embodiments of the present invention involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of relevant departments, and in compliance with relevant laws, regulations, and standards. If personal information is involved in the embodiments, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject is required, and the embodiments also need to be implemented with the authorization and consent of the object.

[0064] Please see Figure 1 , Figure 1 This is a schematic diagram of the structure of a multimodal missing adaptive abnormal behavior detection system based on an inspection robot, provided in Embodiment 1 of the present invention.

[0065] The present invention provides a multimodal missing adaptive abnormal behavior detection system based on an inspection robot. The system includes a multimodal data acquisition module, a data processing module, a modal state detection module, a missing modality adaptive processing module, and a multimodal abnormal behavior detection module that are connected in sequence via communication.

[0066] A multimodal data acquisition module is used to acquire multimodal data.

[0067] The data processing module is used to preprocess multimodal data to obtain preprocessed multimodal features;

[0068] The modal state detection module is used to perform multimodal data quality assessment on the preprocessed multimodal features and generate modal state information;

[0069] The missing modality adaptive processing module is used to determine the current modality missing type based on the modality state information, and selectively perform feature compensation or invalid modality removal on the preprocessed multimodal features based on the current modality missing type to obtain multimodal features that are adapted to the current perception state.

[0070] The multimodal abnormal behavior detection module is used to dynamically configure the multimodal feature fusion weights based on the modal state information, and calculate the behavior confidence score by combining the multimodal features adapted to the current perception state. After comparing the behavior confidence score with the preset abnormal judgment threshold, the abnormal behavior detection result is output.

[0071] It should be noted that this system mainly consists of the following modules:

[0072] 1) The multimodal sensing and acquisition module is used to collect multi-source data such as images, audio, and equipment operating status in the inspection environment in real time. It includes:

[0073] Visual acquisition unit: used to acquire video images, multi-view target images, and behavioral scene information of the inspection area;

[0074] Audio acquisition unit: used to acquire ambient sounds, abnormal noises, and voice information;

[0075] Robot status acquisition unit: used to acquire robot motion status, motor status, position information and equipment operating parameters;

[0076] 2) The modal state detection and quality assessment module (i.e., the modal state detection module) is used to perform online state detection and quality analysis on different modal data, and to identify situations such as image occlusion, abnormal lighting, audio interference, equipment failure and data loss.

[0077] This module can calculate modal quality scores based on the quality parameters corresponding to different modalities and generate modal state labels, providing a basis for subsequent adaptive processing of missing modalities.

[0078] 3) The missing modality adaptive processing module is used to execute corresponding compensation and degradation processing strategies based on the modality state results.

[0079] When a local visual modality is missing, the system uses other normal visual perspectives to complement multi-view data and combines audio and posture modalities for auxiliary enhancement.

[0080] When the auxiliary modality is abnormal, the system switches to a degraded detection mode based on the visual modality.

[0081] When the main visual modality completely fails, the system stops detecting abnormal behavior and triggers device anomaly alarms and path adjustment mechanisms.

[0082] 4) The multimodal feature fusion and abnormal behavior detection module (i.e., the multimodal abnormal behavior detection module) is used to uniformly fuse multimodal features such as vision, audio, posture and robot state, and perform abnormal behavior recognition.

[0083] The system constructs a unified semantic representation of behavior through modal mapping, gating weighting, and dynamic fusion mechanisms, enabling real-time detection and classification of abnormal behaviors such as smoking, falling, climbing, fighting, and pushing.

[0084] 5) The inspection path dynamic adjustment module is used to dynamically adjust the inspection path according to the modal quality status, environmental access conditions and the location of abnormal events.

[0085] This module constructs a dynamic path planning model based on the improved A* algorithm and introduces modal quality cost and environmental complexity cost to achieve inspection path optimization.

[0086] When a modality is missing or the data quality deteriorates, the system automatically triggers a path replanning mechanism to regenerate a supplementary acquisition path, thereby improving the integrity of data acquisition and the robustness of inspection in complex scenarios.

[0087] 7) The abnormal behavior alarm and result output module is used to generate corresponding alarm outputs and inspection feedback based on the abnormal behavior detection results.

[0088] When the system detects abnormal behavior, it can perform operations such as sound and light alarms, video recording and storage, tracking of key areas, recording of abnormal status and updating of inspection logs, and simultaneously update the inspection status and AR inspection information.

[0089] Each module forms a closed-loop collaborative operation through a unified communication mechanism, realizing the linkage control between multimodal data acquisition, modal quality assessment, missing modality compensation, abnormal behavior detection, and dynamic path adjustment, thereby constructing an intelligent inspection abnormal behavior detection system with real-time performance, adaptive capabilities, and robustness in complex environments.

[0090] It is worth mentioning that the system integrates the inspection robot body based on ROS (Robot Operating System), the edge intelligent processing and decision-making center, and the user management and alarm interaction terminal to build a collaborative security system that integrates multimodal data acquisition, modal state detection, missing modality adaptive processing, abnormal behavior detection, and alarm linkage.

[0091] The system achieves real-time detection of abnormal behavior, adaptive compensation for missing modalities, multimodal fusion analysis, and autonomous path optimization in complex inspection environments through the collaborative work of inspection robots, multimodal sensing devices, abnormal behavior detection modules, and dynamic path adjustment modules.

[0092] The system adopts a closed-loop collaborative architecture of "multimodal perception—state detection—missing data compensation—fusion analysis—anomaly detection—path adjustment" to achieve multi-source data acquisition, modal quality assessment, adaptive processing, and intelligent inspection control in complex environments. The inspection robot itself is responsible for multi-source, multimodal data acquisition and on-site execution; the edge intelligent processing and decision-making center, as the core processing unit, is responsible for multimodal data processing, modal state detection, adaptive processing of missing modalities, abnormal behavior detection, task scheduling, and dynamic path adjustment; the user management and alarm interaction terminal, as the system's interactive output unit, is used to receive anomaly alarms, manage detection results, and monitor equipment operation. All levels collaborate to achieve closed-loop operation of the abnormal behavior detection task.

[0093] The overall system structure mainly includes the following parts.

[0094] (1) ROS-based inspection robot body (execution perception layer):

[0095] The ROS-based inspection robot body serves as the system's underlying execution and perception unit, responsible for performing inspection tasks, collecting multi-source, multi-modal data, and responding to edge-side control commands. This body comprises a multi-source sensor unit, a ROS node module, an embedded computing unit, and an execution mechanism.

[0096] The multi-source sensor unit is used to collect visual data, audio data, and environmental status data. The visual data includes RGB (Red-Green-Blue, RGB camera image data) camera data, infrared camera data, and depth camera data. The audio data includes ambient sound data collected by the microphone array. The environmental status data includes LiDAR data and robot operating status data. The embedded computing unit is used to perform local lightweight processing tasks, including basic data preprocessing and preliminary feature extraction. The ROS node module is used to realize data communication and control management between various functional units within the robot. The actuator is used to complete robot movement, gimbal control, alarm triggering, and key area tracking operations.

[0097] The inspection robot interacts with the edge intelligent processing and decision-making center via the ROS communication mechanism. Specifically, the robot publishes raw data and preliminary feature data to the edge intelligent processing and decision-making center through the ROS Topic mechanism, enabling asynchronous transmission of multimodal data. The edge intelligent processing and decision-making center obtains robot operating status information or sends short-term control requests through the ROS Service mechanism. For tasks such as inspection path adjustment, supplementary data collection in key areas, and continuous target tracking, the edge intelligent processing and decision-making center sends task objectives to the robot through the ROS Action mechanism and receives task execution feedback and results, thereby achieving dynamic control of the robot's task execution process.

[0098] (2) Edge Intelligent Processing and Decision-Making Center (Core Processing Layer):

[0099] The edge intelligent processing and decision-making center is deployed on local edge computing nodes and is used to complete multimodal data processing, modality state detection, missing modality adaptive processing, abnormal behavior detection, and task scheduling. It is the core processing unit of the system. The center includes a modality state detection and quality assessment module, a missing modality adaptive processing module, a multimodal abnormal behavior detection module, a task complexity assessment and scheduling module, an inspection path planning and dynamic adjustment module, and a communication and control interface module.

[0100] The modal state detection and quality assessment module is used to detect the data status of visual modality, audio modality and posture modality in real time, evaluate the online status of the device, data integrity, image clarity, occlusion, audio signal quality and the effectiveness of human key points, and output the available status and quality information of the corresponding modality.

[0101] The missing modality adaptive processing module is used to determine the missing modality status based on modal state information and execute the corresponding adaptive processing strategy according to the missing type. Missing types include partial missing local modalities, missing auxiliary modalities, and complete missing primary modalities.

[0102] In cases where some local modalities are missing, the remaining available visual modalities are used for multi-view visual complementarity, and auxiliary modal information is combined for abnormal behavior detection.

[0103] In the absence of an auxiliary modality, a visual modality is used to perform degraded abnormal behavior detection;

[0104] If the main mode is completely missing, stop abnormal behavior detection and output device abnormality alarm information.

[0105] The missing modality adaptive processing module divides the abnormal behavior detection process into normal detection mode, degraded detection mode and detection termination mode based on modality state information. Under different detection modes, it adaptively combines and dynamically selects available modalities to achieve robust abnormal behavior detection in modality missing scenarios.

[0106] The multimodal abnormal behavior detection module is used to identify abnormal behaviors based on effective multimodal data. Abnormal behaviors include single-person abnormal behaviors and multi-person abnormal behaviors. Among them, single-person abnormal behaviors include smoking, falling, jumping and climbing, while multi-person abnormal behaviors include fighting and pushing.

[0107] The task complexity assessment and scheduling module is used to construct task complexity indicators based on the type of abnormal behavior, the number of targets, and the degree of modality missing, and to dynamically schedule edge computing resources based on task complexity in order to balance the real-time performance and detection accuracy of abnormal behavior detection.

[0108] The inspection path planning and dynamic adjustment module is used to plan inspection paths by combining environmental information of the inspection area, the distribution of areas with high incidence of abnormal behavior, and the quality of real-time modal acquisition. When modal missing or data quality deterioration is detected, the robot's inspection path or observation position is automatically adjusted to supplement data acquisition in the target area.

[0109] The communication and control interface module is used to realize data communication and control interaction between the edge intelligent processing and decision-making center and the inspection robot body. It supports data publishing and subscription mechanisms based on ROS Topic, as well as task control mechanisms based on ROSService and ROS Action.

[0110] (3) User management and alarm interaction terminal (interaction output layer):

[0111] The user management and alarm interaction terminal is used to realize abnormal event management, equipment status monitoring and system operation and maintenance management, including an abnormal alarm receiving unit, a detection result management unit and a remote monitoring unit.

[0112] The abnormal alarm receiving unit is used to receive abnormal behavior alarm information in real time and to highlight high-risk abnormal events; the detection result management unit is used to store abnormal behavior video data, detection results and corresponding time information to support the retrospective analysis of historical abnormal events; the remote monitoring unit is used to visualize the robot's operating status, sensor status and equipment operating status, and to provide equipment status information for operation and maintenance management.

[0113] The user management and alarm interaction terminal interacts with the edge intelligent processing and decision-making center through a standardized communication interface. It is used to receive abnormal behavior detection results, alarm information and device status information, and send remote control commands to the edge intelligent processing and decision-making center to realize remote system management and device operation and maintenance.

[0114] Based on the aforementioned system architecture, this invention constructs a hierarchical collaborative processing mechanism driven by modal state. Multimodal data collected by the inspection robot is transmitted to the edge intelligent processing and decision-making center via the ROS communication mechanism. The edge side performs modal state detection, quality assessment, and adaptive processing of missing modalities before executing abnormal behavior detection. The edge intelligent processing and decision-making center dynamically schedules computing resources based on task complexity and dynamically adjusts the inspection path according to the quality of modal data acquisition to improve the completeness of abnormal behavior detection. When abnormal behavior is detected, the edge intelligent processing and decision-making center sends anomaly alarms and detection results to the user management and alarm interaction terminal, and simultaneously sends control commands to the inspection robot to trigger PTZ tracking, key area recording, and alarm linkage operations, thereby realizing a closed-loop collaborative processing flow of "data acquisition—state assessment—adaptive processing—anomaly detection—control execution—result feedback".

[0115] Through the above-mentioned hierarchical collaborative architecture, this invention realizes the collaborative acquisition of multimodal data, adaptive abnormal behavior detection in scenarios with missing modalities, and abnormal behavior alarm linkage, effectively improving the robustness, real-time performance, and system practicality of abnormal behavior detection in complex scenarios.

[0116] As a further improvement, the multimodal data includes visual data, audio data, and robot state data; the multimodal data acquisition module includes a visual acquisition unit, an audio acquisition unit, and a robot state acquisition unit.

[0117] The vision acquisition unit, audio acquisition unit, and robot status acquisition unit are all connected to the data processing module;

[0118] The visual acquisition unit is used to acquire visual data and transmit it to the data processing module;

[0119] The audio acquisition unit is used to acquire audio data and transmit it to the data processing module;

[0120] The robot status acquisition unit is used to collect robot status data and transmit it to the data processing module.

[0121] It should be noted that the multimodal data acquisition module relies on the inspection robot's hardware to synchronously acquire and output multi-source sensory data, ultimately generating a complete multimodal dataset containing visual data, audio data, and robot status data. The visual acquisition unit, mounted on the robot's rotatable gimbal, consists of an RGB camera, an infrared camera, and a depth camera. During acquisition, a hardware synchronous trigger signal controls the three cameras to simultaneously expose and capture frames, generating visible light color images, thermal imaging temperature images, and spatial depth point cloud images, which together constitute the visual data. Each frame of visual data is stamped with a high-precision hardware timestamp at the acquisition end, encapsulated according to a unified data frame format, and transmitted to the data processing module via the robot's internal high-speed serial bus. The audio acquisition unit uses a multi-microphone array deployed on both sides of the gimbal. During acquisition, it synchronously picks up ambient sound signals, performs analog-to-digital conversion and basic gain calibration, matches the corresponding timestamp to generate continuous digital voiceprint signals, i.e., audio data, and transmits them to the data processing module. The robot state acquisition unit connects to the robot chassis control bus, reading in real time the rotational speed and angle data from the motor encoder, the positioning and mapping data from the LiDAR, and the attitude data from the inertial measurement unit. This data is then aggregated to obtain the robot state data, synchronized with the system's unified time base, and transmitted to the data processing module. All three acquisition units adhere to a unified communication protocol and data encapsulation standard, ensuring that the multi-source data output to the data processing module has an alignable time base and a unified data interface. This multi-unit synchronous acquisition and unified time base encapsulation approach provides consistent raw multimodal input for subsequent data alignment, quality assessment, and adaptive compensation, effectively reducing the processing difficulty of heterogeneous multi-source data, ensuring the spatiotemporal matching accuracy of different modal data, and providing a reliable data foundation for stable abnormal behavior detection in complex environments.

[0122] As a further improvement, the data processing module is specifically used for:

[0123] Data cleaning is performed on the multimodal data to obtain cleaned multimodal data;

[0124] Time alignment is performed on the cleaned multimodal data to obtain time-aligned multimodal data;

[0125] Spatially align the time-aligned multimodal data to obtain spatially aligned multimodal data;

[0126] Feature extraction is performed on the spatially aligned multimodal data to obtain preprocessed multimodal features.

[0127] It should be noted that the system first receives multimodal input data collected by the inspection robot's perception system, including visual images, environmental audio, and robot operating status data. Let the system input data set be:

[0128] ;

[0129] Where: I represents visual image data; A represents environmental audio data; R represents robot operating status data; after performing data cleaning, time alignment, and spatial alignment preprocessing operations, the system extracts features from different modal data respectively:

[0130] ;

[0131] in: Visual features; Audio features; Features of the robot's state; Represents a visual feature encoder; Indicates an audio feature encoder; This represents the robot's state encoder.

[0132] Simultaneously, the system performs human pose estimation based on visual image data, extracting key points and pose features of the human body:

[0133] ;

[0134] in: This represents a function for extracting human pose features. It indicates the characteristics of human posture.

[0135] The final result is a unified set of multimodal features, i.e., the preprocessed multimodal features:

[0136] ;

[0137] It serves as the input for subsequent modal state detection, missing modality adaptive compensation, and abnormal behavior detection modules.

[0138] In this embodiment, the system receives multimodal input data collected by the inspection robot's perception system. Let the system input data set be... Subsequently, a four-level preprocessing process of data cleaning, time alignment, spatial alignment, and feature extraction is executed in sequence. First, data cleaning is performed on the original multimodal dataset D. For the visual image data I, median spatial domain filtering is used to remove salt-and-pepper noise and random noise in the image, and invalid frames that are overexposed or completely black are removed. Neighborhood interpolation is used to repair slightly missing pixel areas. For the environmental audio data A, DC offset and environmental steady-state background noise are filtered out, and silent frames with no effective signal and sudden pulse interference are removed. For the robot running status data R, abnormal jump values ​​that exceed the physical range are removed. The data fluctuations are smoothed by moving average filtering, and linear interpolation is used to complete the temporarily lost sampling points to obtain the cleaned multimodal data. Next, using the hardware timestamps of the visual image data as the baseline time series, time alignment processing is performed on the cleaned multimodal data. Audio data with higher sampling frequencies is resampled using linear interpolation at the baseline time point to match sampling nodes. Robot running state data is matched by pose interpolation based on the corresponding timestamps, ensuring a one-to-one correspondence between the sampling times of the three modalities in the time dimension, resulting in time-aligned multimodal data. Subsequently, based on pre-calibrated hand-eye calibration parameters involving both internal and external camera inputs, spatial alignment processing is performed on the time-aligned multimodal data. The imaging coordinates of the infrared and depth cameras are uniformly mapped to the optical coordinate system of the RGB camera through projection transformation, and then coordinate transformation moments are used to... The system unifies the chassis coordinate system and visual optical coordinate system corresponding to the robot's operational status data, enabling all modal data to share the same spatial reference system, resulting in spatially aligned multimodal data. Finally, modality-specific feature extraction is performed on the spatially aligned multimodal data. A visual feature encoder performs depth encoding on the visual image data to extract visual features; an audio feature encoder performs time-frequency domain transformation and encoding on the environmental audio data to extract audio features; and a robot state encoder normalizes and encodes the robot's operational status data to extract robot state features. Simultaneously, human pose estimation is performed based on the visual image data, using human pose feature extraction. The function performs keypoint detection and skeleton modeling on the human target in the image, extracts human posture features, and then concatenates and integrates the four types of features according to preset dimensional rules to obtain a unified multimodal feature set, i.e., the preprocessed multimodal features, which serve as input for subsequent modal state detection, missing modality adaptive compensation, and abnormal behavior detection modules. Through a four-level progressive standardized preprocessing process combined with modality-specific encoding extraction methods, it can eliminate noise interference, temporal deviation, and spatial misalignment problems of multi-source heterogeneous data, output standardized features with unified dimensions and accurate spatiotemporal matching, and supplement behavioral representation information of human posture dimension, providing a reliable input basis for subsequent modal quality assessment and adaptive compensation.

[0139] Furthermore, the modal state detection module performs online state detection and quality assessment on visual, audio, and pose modal data to identify modal missingness, image occlusion, audio interference, and device malfunctions.

[0140] Let the set of quality features corresponding to the i-th mode be:

[0141] ;

[0142] Where: Ki represents the set of quality assessment parameters corresponding to the i-th mode. This represents the nth quality evaluation index corresponding to the i-th mode;

[0143] Different modes correspond to different quality assessment parameters.

[0144] in:

[0145] For visual modalities, quality parameters include image sharpness, occlusion rate, and brightness stability;

[0146] For audio modalities, quality parameters include signal-to-noise ratio, audio continuity, and noise interference level;

[0147] For attitude modalities, quality parameters include keypoint integrity, skeleton stability, and attitude confidence.

[0148] The system calculates the modal quality score based on the corresponding modal quality parameters:

[0149] ;

[0150] in: This represents the quality evaluation function corresponding to the i-th mode; This represents the quality score for the corresponding mode. The system generates mode state labels based on the quality score results:

[0151] ;

[0152] in: Indicates the modal quality threshold; This indicates that the mode is normal and usable; This indicates that the modality is missing, abnormal, or unavailable.

[0153] Finally, the system outputs a set of modal states:

[0154] ;

[0155] This serves as the input for the subsequent missing modality adaptive compensation module.

[0156] In this embodiment, the modal state detection module receives preprocessed multimodal features and performs online state detection and quality assessment on three modalities: visual, audio, and human posture. This is used to identify modality loss, image occlusion, audio interference, and abnormal feature validity. During the detection process, firstly, for the i-th modality, multi-dimensional quality assessment indicators are extracted from the corresponding features to form the quality assessment parameter set corresponding to that modality. Different modalities correspond to different quality assessment parameters. For the visual modality, image sharpness, occlusion rate, and brightness stability are obtained by calculating the image gradient variance, effective region pixel ratio, and inter-frame brightness mean fluctuation corresponding to visual features. For the audio modality, signal-to-noise ratio, audio continuity, and noise interference level are obtained by performing spectral analysis and effective frame statistics on audio features. For the human posture modality, keypoint integrity rate, skeleton stability, and posture confidence are obtained by statistically analyzing the number of effective keypoints in posture features, inter-frame skeleton coordinate offset, and keypoint confidence mean. Subsequently, the system calls the quality assessment function corresponding to the i-th modality. The quality assessment parameter set for this mode Substituting the values ​​into the function and performing a weighted fusion calculation yields the quality score for the corresponding modality. Satisfying the relation The quality evaluation functions for each modality have built-in appropriate parameter weights, which can be preset and adjusted according to the importance of the modality and the detection requirements; then, the quality scores of each modality are calculated. With preset modal quality threshold Perform a step-by-step comparison to generate the corresponding modal state label. Satisfying the piecewise relation ,when This indicates that the mode is normally available. This indicates that the modality is missing, abnormal, or unavailable. Different modalities can be configured with independent threshold parameters to adapt to their respective quality judgment criteria. Finally, the visual modality status labels are summarized. Audio modal status labels Human posture modal state labels Generate a complete set of modal states. The modal state information, including the quality scores and status labels of each modality, is synchronously output to the missing modality adaptive processing module as the basis for subsequent adaptive compensation processing and detection strategy switching.

[0157] As a further improvement, the preprocessed multimodal features include visual features, audio features, and human pose features; the current modality missing types include local visual modality missing, audio-assisted modality missing, human pose-assisted modality missing, and normal modality; the missing modality adaptive processing module is specifically used for:

[0158] Read the modal quality score from the modal state information. The modal quality score includes visual modal quality score, audio modal quality score and human posture modal quality score; the visual modal quality score includes overall visual quality score and visual sub-regional quality score.

[0159] The overall visual quality score is compared with a preset main modality failure threshold. If the overall visual quality score is greater than or equal to the preset main modality failure threshold, the visual sub-region quality score is compared with a preset local missing threshold one by one. If at least one visual sub-region quality score is less than the preset local missing threshold, it is determined to be a local visual modality missing. The multi-view visual features in the preprocessed multimodal features are used to complement the visual features of the corresponding missing region in the multimodal features, and audio features are fused for auxiliary compensation to obtain multimodal features adapted to the current perception state.

[0160] If the visual sub-region quality scores are all greater than or equal to the preset local missing threshold, the audio modality quality score is compared with the preset auxiliary modality effective threshold; if the audio modality quality score is less than the preset auxiliary modality effective threshold, it is determined that the audio auxiliary modality is missing. The audio features are removed from the preprocessed multimodal features, while the visual features and human posture features are retained to obtain multimodal features that are adapted to the current perception state.

[0161] If the audio modal quality score is greater than or equal to the preset effective threshold for auxiliary modality, the human posture modal quality score is compared with the preset effective threshold for posture. If the human posture modal quality score is less than the preset effective threshold for posture, it is determined that the human posture auxiliary modality is missing. The human posture features are removed from the preprocessed multimodal features, while the visual and audio features are retained to obtain multimodal features that are adapted to the current perception state.

[0162] If the human posture modal quality score is greater than or equal to the preset effective posture threshold, it is determined that the modality is normal, and the preprocessed multimodal features are used as multimodal features to adapt to the current perception state.

[0163] It should be noted that, in order to improve the robustness of abnormal behavior detection in complex inspection environments, this invention executes corresponding adaptive processing strategies based on the modal state results.

[0164] When a local visual modality is missing, the system uses other normal visual perspectives to complement multi-view data and combines audio and pose modalities for auxiliary enhancement in order to recover the behavioral feature expression of the target region.

[0165] The compensation process is expressed as follows:

[0166] ;

[0167] in: Feature representations representing different visual perspectives; Represents audio modal features; Represents attitude modal features; Represents the set of modal states; Represents the visual modality compensation function; This indicates the visual features after compensation.

[0168] When the auxiliary modality is missing, the system switches to a degraded detection mode based on the visual modality, using only the visual modality to perform abnormal behavior detection.

[0169] When the main visual modality is completely missing, the system stops detecting abnormal behavior and triggers device anomaly alarms and path adjustment mechanisms because it cannot obtain effective behavioral image information.

[0170] The final adaptively processed multimodal feature set is obtained as follows:

[0171] ;

[0172] It is then used as input for the subsequent multimodal feature fusion module.

[0173] Specifically, such as Figure 2 As shown, this invention proposes an adaptive processing method for missing modalities in inspection robots. Its core lies in dynamically adjusting the abnormal behavior detection strategy according to the type of missing modality, so as to improve the robustness and system stability of abnormal behavior detection in complex environments.

[0174] The process first receives the modal state detection and quality assessment results, analyzes the data integrity, availability and quality status of visual modality, audio modality and posture modality, and determines whether there are modal missing or data quality degradation issues.

[0175] When the system does not detect modality loss or data quality abnormalities, it directly enters the multimodal feature fusion and abnormal behavior detection process.

[0176] When a modality is detected as missing or data quality is degraded, the system enters the missing modality adaptive processing flow and executes the corresponding processing strategy according to the type of missing modality, specifically including:

[0177] (1) Missing auxiliary modes:

[0178] When the audio modality, posture modality, or other auxiliary modalities experience signal anomalies, data loss, or recognition failures, while the visual modality remains in normal working order, the system executes a visual degradation detection strategy, using only visual modal data to complete the abnormal behavior detection, thereby ensuring the system's continuous detection capability under auxiliary modal anomalies.

[0179] (2) Local visual modality loss:

[0180] When a single or partial visual modality is occluded, blurred, has a limited field of view, or a local device fails, the system executes a multi-view complementary detection strategy. By calling the pan-tilt unit to rotate, adjusting the inspection position, or switching to other visual acquisition devices, it supplements the acquisition and compensates for the missing areas. It also combines auxiliary modal information to enhance the expression of behavioral features and generate compensated multimodal feature data.

[0181] (3) The main visual modality is completely missing:

[0182] When all visual modalities malfunction, are occluded, or have failed, and effective human behavior image information cannot be obtained, the system determines that the main visual modality is completely missing, stops the abnormal behavior detection process, and triggers the equipment abnormal alarm mechanism; at the same time, it sends a safety control command to the robot control module, causing the robot to perform protective actions such as stopping inspection, standing still, or safely docking.

[0183] After completing the adaptive processing of missing modalities, the system performs unified fusion processing on the original or compensated multimodal features to construct a multimodal fusion feature representation, and detects and classifies abnormal behaviors such as smoking, falling, climbing, fighting and pushing based on the multimodal abnormal behavior recognition model.

[0184] When the detection result reaches the preset alarm threshold, the system triggers a series of linked operations such as audible and visual alarms, video storage, PTZ tracking, and recording of key areas, and outputs information on the type, location, and confidence level of the abnormal behavior for abnormal event management and subsequent tracing.

[0185] After completing this round of testing, the system re-enters the next round of data collection and abnormal behavior detection cycle.

[0186] In this embodiment, the missing modality adaptive processing module first reads the modal quality scores from the modal state information, including visual modal quality scores, audio modal quality scores, and human posture modal quality scores. The visual modal quality score is further broken down into an overall visual quality score and a visual sub-region quality score. Then, the overall visual quality score is compared with a preset master modality failure threshold. If the overall visual quality score is less than the preset master modality failure threshold, it is determined that the master vision is completely missing. At this time, the system stops the abnormal behavior detection process and triggers a device abnormality alarm and path adjustment mechanism. If the overall visual quality score is greater than or equal to the preset master modality failure threshold, the system enters the detailed judgment stage of visual sub-region quality. The visual sub-region quality score of each region is compared with a preset local missing threshold one by one. If at least one visual sub-region quality score is less than the preset local missing threshold, it is determined that a local visual modality is missing. At this time, the visual modality compensation function is called to perform multi-view complementarity and auxiliary enhancement processing. The compensation process satisfies the following relation: The processing first utilizes multi-view visual features from other normal perspectives. Through feature space mapping and feature filling of missing regions, the visual features of the corresponding missing regions are complementaryly restored. Then, audio features and human posture features are fused to enhance the behavioral semantic dimension, ultimately obtaining the compensated visual features. The audio modality quality score, combined with other effective modal features, constitutes a multimodal feature adapted to the current perception state. If the quality scores of all visual sub-regions are greater than or equal to a preset local missing threshold, the system enters the step-by-step judgment stage of the auxiliary modality. First, the audio modality quality score is compared with the preset effective threshold for the auxiliary modality. If the audio modality quality score is less than the preset effective threshold for the auxiliary modality, it is determined that the audio auxiliary modality is missing. At this time, the system switches to a downgraded detection mode, removing invalid audio features from the preprocessed multimodal features, retaining complete visual features and human posture features to constitute a multimodal feature adapted to the current perception state. Subsequent abnormal behavior detection is performed only based on the effective visual and posture dimensions. If the audio modality quality score is greater than or equal to a preset threshold, the system enters the step-by-step judgment stage of the auxiliary modality. First, the audio modality quality score is compared with the preset effective threshold for the auxiliary modality. If the audio modality quality score is less than the preset effective threshold for the auxiliary modality, it is determined that the audio auxiliary modality is missing. If the auxiliary modality is not valid, the human posture modality quality score is compared with the preset valid posture threshold. If the human posture modality quality score is less than the preset valid posture threshold, it is determined that the human posture auxiliary modality is missing. Invalid human posture features are removed from the preprocessed multimodal features, and visual and audio features are retained to form multimodal features adapted to the current perception state. Subsequent detection is performed based on visual and audio dual modalities. If the human posture modality quality score is greater than or equal to the preset valid posture threshold, it is determined that the modality is normal, and the complete preprocessed multimodal features are directly used as multimodal features adapted to the current perception state. After the above hierarchical judgment and corresponding processing, the finally adaptively processed multimodal feature set is obtained. It is then transmitted as input to the subsequent multimodal feature fusion and abnormal behavior detection module.

[0187] It is worth mentioning that the aforementioned preset threshold can be set as needed, and this invention does not limit this setting.

[0188] Furthermore, the current modal loss types also include complete loss of the main visual; the missing modality adaptive processing module is also used for:

[0189] If the overall visual quality score is less than the preset primary modality failure threshold, it is determined that the primary vision is completely missing, abnormal behavior detection is stopped, and device abnormality alarm information is output.

[0190] It should be noted that the current modality loss type also includes complete loss of primary vision. When the missing modality adaptive processing module performs the corresponding processing, it first extracts the overall visual quality score from the input modality state information and compares it value by value with the pre-configured preset primary modality failure threshold. When the overall visual quality score is less than the preset primary modality failure threshold, the current modality loss type is determined to be complete loss of primary vision. After the determination, the module first generates a status label for primary vision failure and writes it to the modality state information. After the status update is completed, it sends a detection termination command to the multimodal abnormal behavior detection module, forcibly interrupting the currently executing multimodal feature fusion calculation and abnormal behavior classification reasoning process, preventing completely failed visual features from continuing to participate in the detection calculation, and avoiding serious misjudgments in the detection results due to invalid data. At the same time, the module generates standardized equipment abnormality alarm information, which includes the specific type of failure modality, the actual value of the overall visual quality score, the precise timestamp of the abnormal event trigger, and the robot's... The current spatial coordinates are used to push the alarm information to the system's alarm and control unit, triggering the on-site audible and visual alarm devices to notify on-site and back-end maintenance personnel of equipment failure. Simultaneously, the alarm information is transmitted to the inspection path planning and dynamic adjustment module, triggering emergency path replanning logic. This guides the robot to autonomously move to a pre-set, unobstructed, open-view supplementary data collection point to attempt to re-collect visual data. If the collection results after a pre-set number of consecutive attempts still fail to meet the minimum quality threshold, the robot is controlled to return to the designated standby station to perform equipment self-checks and troubleshooting. Through a hierarchical response mechanism of detection suspension, alarm linkage, and path coordination in scenarios where the main vision is completely missing, unreliable detection processes can be promptly suspended when the core perception dimension is completely ineffective, avoiding false detections and missed detections caused by invalid data from the source. Simultaneously, the linkage between alarm output and path adjustment forms a complete fault response closed loop, effectively improving the system's operational stability and adaptability to complex environments under extreme perception failure scenarios.

[0191] As a further improvement, the multimodal abnormal behavior detection module performs unified fusion of adaptively processed multimodal features to construct a complete behavioral semantic representation.

[0192] The fusion process is represented as:

[0193] ;

[0194] in: , , , represents the feature mapping matrix corresponding to different modalities; F represents the unified multimodal feature representation after fusion.

[0195] To enhance the representation of the importance of different modalities, a gating mechanism is introduced:

[0196] ;

[0197] The final fusion feature is represented as:

[0198] ;

[0199] in: Represents the Sigmoid activation function; ⊙ represents element-wise multiplication; This represents the enhanced fusion feature representation.

[0200] Furthermore, the system performs abnormal behavior recognition based on the fused multimodal features.

[0201] Let the predicted probability of the abnormal behavior category be:

[0202] ;

[0203] in: Represents the classification weight matrix; Indicates the bias term; This represents the probability distribution of abnormal behavior categories.

[0204] The final result of the abnormal behavior identification is as follows:

[0205] ;

[0206] in: Indicates the final category of abnormal behavior.

[0207] The system supports the identification of abnormal behaviors such as smoking, falling, climbing, fighting, and pushing.

[0208] Specifically, firstly, based on the quality score and availability of each modality in the modal state information, the weights of the feature mapping matrices corresponding to different modalities are dynamically configured. Modalities with higher quality scores and stronger availability have larger weight coefficients in their corresponding feature mapping matrices. The weights of low-quality modalities that have been determined to be ineffective are adjusted accordingly, achieving dynamic adaptation of the fusion weights. Subsequently, a unified fusion operation is performed on the multimodal features adapted to the current perception state. The adaptively processed visual features, audio features, and human pose features are multiplied by their corresponding feature mapping matrices and then summed to obtain the fused unified multimodal feature representation F. The fusion process satisfies the following relation: To further enhance the importance representation of different modalities and suppress the interference of redundant features, the module introduces a gating mechanism. First, the gating weight matrix is ​​multiplied by the fused feature F and a gating bias term is added. Then, the gating weight matrix G, with values ​​ranging from 0 to 1, is obtained through a Sigmoid activation function. The operation process satisfies the following relation: Next, element-wise multiplication is performed between the gate weight matrix G and the fused feature F to complete the dynamic weighted calibration of each dimension of the fused feature, resulting in the enhanced fused feature representation. The operation satisfies the relation. After feature enhancement, the module performs abnormal behavior classification and recognition based on the enhanced fused features, and then... With classification weight matrix Multiply and add the classification bias terms The probability distribution of abnormal behavior categories is then obtained through the Softmax function. The operation satisfies the relation. In the probability distribution, each element corresponds to the predicted probability of a type of abnormal behavior; then, the abnormal behavior category with the highest probability is obtained by taking the maximum value. Satisfying the relation Simultaneously, the highest probability value is used as the behavior confidence score of the current detection result. Finally, the calculated behavior confidence score is compared with the preset anomaly judgment threshold. If the behavior confidence score is greater than or equal to the preset anomaly judgment threshold, it is determined that there is an abnormal behavior of the corresponding type, and an abnormal behavior detection result containing the abnormal behavior category, behavior confidence score, event timestamp and robot location information is generated. If the behavior confidence score is less than the preset anomaly judgment threshold, it is determined that the scene state is normal, and a detection result without anomalies is output. Finally, the detection result is synchronously pushed to the downstream alarm and control module.

[0209] Finally, the detection results are synchronously pushed to the downstream alarm and control module. After receiving the detection results, the alarm and control module first parses the data packet, extracting the abnormal behavior category, behavior confidence score, event timestamp, and the robot's current position coordinates. When the detection result determines that there is abnormal behavior, the module immediately generates a local audible and visual alarm command, which is sent to the audible and visual alarm on the top of the inspection robot's pan-tilt unit via the internal control bus. This drives the flashing warning light and the preset decibel voice prompt to start synchronously, completing the safety warning for the on-site area. At the same time, a standardized abnormal alarm event message is generated and pushed to the back-end management terminal via the wireless communication link, displaying the details of the abnormal event and the corresponding real-time image in a pop-up window on the monitoring interface. Subsequently, the module triggers multi-dimensional linkage control logic. On the one hand, it sends a target continuous tracking command to the pan-tilt unit control unit, driving the pan-tilt mechanism to dynamically adjust the shooting angle and focal length according to the spatial position of the abnormal target, ensuring that the abnormal target is always in the center of the image observation. Simultaneously, it starts the multi-modal data recording and storage function, performing local encrypted storage and cloud backup of the visual and audio data within a preset time before and after the abnormal event is triggered, leaving complete evidence for subsequent event tracing and handling. On the other hand, it transmits the location information of the abnormal event to the control unit. The information and the modal quality status of the current area are synchronously transmitted to the inspection path planning and dynamic adjustment module, triggering local inspection path replanning. This guides the robot to prioritize inspection points near the abnormal event to complete supplementary data collection, strengthening the perception coverage density of high-risk areas. When the detection result determines that there is no abnormal behavior, the module synchronously updates the status log of the current inspection point, marks the point as completed and records the corresponding modal quality data, and sends a continue inspection command to the robot motion control unit, driving the robot to proceed to the next collection point along the predetermined path to perform subsequent detection tasks. At the same time, all detection results, corresponding modal status information, and alarm handling records are synchronously written to the abnormal behavior recording module, and structured index storage is completed according to timestamp, area coordinates, and event type. This supports subsequent retrieval, backtracking, and statistical analysis of historical abnormal events based on multi-dimensional conditions. Through the full-link linkage response between detection results, on-site alarms, PTZ tracking, data storage, and path adjustment, a complete closed loop of handling from on-site warning to inspection strategy adjustment can be quickly completed after abnormal behavior identification, effectively improving the response speed and handling efficiency of abnormal events, and further strengthening the stability and closed-loop management capabilities of safety monitoring in unattended scenarios.

[0210] As a further improvement, the system also includes an inspection path planning and dynamic adjustment module;

[0211] The inspection path planning and dynamic adjustment module is communicatively connected to the modal state detection module and the missing modality adaptive processing module, respectively.

[0212] The inspection path planning and dynamic adjustment module is used to calculate the modal quality cost based on the modal quality score in the modal state information when receiving equipment abnormality alarm information. It combines the path distance parameters and environmental traversal complexity parameters stored in the preset inspection environment map, substitutes them into the cost function of the preset improved A* algorithm to complete the path optimization, generates the optimized inspection path and supplementary acquisition control instructions, and sends the supplementary acquisition control instructions to the motion execution end of the inspection robot to drive the inspection robot to perform supplementary data acquisition.

[0213] It should be noted that, in order to enable the inspection robot to navigate autonomously and adjust its dynamic path in complex environments, this invention constructs an inspection path planning model based on an improved A* algorithm.

[0214] Let the inspection environment space be represented by a graph structure:

[0215] ;

[0216] Where: V represents the set of inspection nodes; E represents the set of passable paths between nodes.

[0217] The system uses the A* evaluation function to calculate the path cost:

[0218] ;

[0219] in: This represents the actual path cost from the starting point to the current node; This represents the heuristically estimated cost from the current node to the target node.

[0220] To adapt to the dynamic environmental changes in inspection scenarios, this invention introduces modal quality cost and environmental traversal complexity cost to improve the path cost function:

[0221] ;

[0222] in: Indicates the path distance cost; Indicates the modal quality cost; This represents the cost of environmental traversal complexity; , , These are the corresponding weighting coefficients.

[0223] The system obtains the optimized inspection path by minimizing the path cost function:

[0224] ;

[0225] in: This indicates the optimized inspection path.

[0226] When the system detects a missing modality or a decline in data quality, it dynamically triggers a path replanning mechanism to recalculate the inspection path based on the robot's current position and the supplementary data collection target point, generating an optimized inspection path and a supplementary data collection path.

[0227] The above methods enable the inspection robot to dynamically adjust its path and supplement data collection in modal missing scenarios, thereby improving the robustness of autonomous inspection and the completeness of data collection in complex environments.

[0228] Specifically, such as Figure 3 As shown, this invention integrates inspection task requirements, environmental map information, and multimodal perception feedback to achieve inspection path planning, dynamic adjustment, and real-time overlay display of AR inspection information, providing navigation support and visualization assistance for autonomous robot inspection. The specific process is as follows:

[0229] (1) Loading environmental map and modeling inspection area:

[0230] After the robot is started, the system loads the pre-built inspection environment map, completes the initial positioning and inspection area modeling, and clarifies the scope to be inspected, the location of key equipment, the channel topology and safe passage area, providing basic environmental information support for subsequent path planning and autonomous navigation.

[0231] (2) Inspection path planning and data collection point layout based on A* algorithm:

[0232] The system uses the A* path planning algorithm for global inspection path planning based on the inspection area model, equipment location information, and environmental map data. During the path search process, factors such as travel distance, passage width, pedestrian density, and priority of key areas are comprehensively considered to evaluate the path cost and generate an inspection path that meets both safety and inspection efficiency requirements.

[0233] Meanwhile, the system deploys multimodal data acquisition points at key nodes along the path and sets corresponding robot docking positions, gimbal acquisition angles, and sensor acquisition parameters to improve the perception coverage of key areas. It also reserves supplementary acquisition points for data supplementation in subsequent modality missing scenarios.

[0234] (3) Autonomous inspection and initial navigation of the robot:

[0235] The robot moves autonomously along a preset inspection path using SLAM navigation, sequentially reaching each data collection point. During the inspection process, it performs real-time positioning, path following, and dynamic obstacle avoidance, ensuring the continuity and safety of the inspection process.

[0236] (4) Multimodal data acquisition and environmental perception:

[0237] After the robot arrives at the data collection point, it activates multi-source sensing devices to simultaneously collect visual images, environmental audio, LiDAR point clouds, and robot operating status data, constructing a multimodal dataset containing spatial, acoustic, and status information to provide input for subsequent modal quality assessment and abnormal behavior detection.

[0238] (5) Modal state detection and quality assessment:

[0239] The system performs online status detection and quality scoring on the collected multimodal data, judges the integrity, clarity and usability of each modality, identifies whether there are modality missing, image occlusion, signal loss or data quality degradation, and outputs the modality status assessment results.

[0240] (6) Adaptive processing of missing modes:

[0241] When a modality is missing or data quality is degraded, the system triggers the missing modality adaptive compensation mechanism. Based on different modal states, it executes corresponding data compensation, detection degradation, or device alarm strategies, and dynamically adjusts the PTZ acquisition angle, inspection path, and supplementary acquisition strategy in combination with real-time perception results, providing support for subsequent abnormal behavior detection and navigation adjustment.

[0242] (7) Dynamically adjust the inspection route:

[0243] Based on the results of the adaptive processing of missing modalities, the system adjusts the subsequent inspection path in real time, adds supplementary collection points or optimizes the dwell time and collection angle of the original points, and plans supplementary collection routes to obtain multimodal data that meets quality requirements.

[0244] (8) Supplementing regional data collection and navigation correction:

[0245] After the robot moves to the newly added supplementary collection point, it re-executes the multimodal data collection process to supplement the collection of missing or low-quality modes, and corrects the navigation path based on the real-time environmental perception results to ensure that the robot continues to inspect along a safe and reasonable path.

[0246] (9) Multimodal abnormal behavior detection:

[0247] Based on the verified multimodal data, the system runs an abnormal behavior recognition model based on multimodal fusion to classify and identify abnormal personnel behaviors and abnormal equipment states in the inspection area and calculate confidence levels, and outputs the detection results.

[0248] (10) Abnormal behavior alarm and result output:

[0249] Based on the detection results, the system outputs the abnormal behavior type, location, confidence level, and timestamp information. When the confidence level reaches the preset alarm threshold, the system triggers an audible and visual alarm and synchronizes the alarm information to the backend management terminal.

[0250] (11) AR inspection information generation and real-scene overlay display:

[0251] The system uses AR overlay to display the inspection path, data collection points, and abnormal alarm information with the real-time inspection screen, generating a visual inspection interface for on-site inspection and remote monitoring, thereby improving the visualization and interactivity of the inspection process.

[0252] (12) Inspection ends:

[0253] After a single inspection task is completed, the robot returns to the designated docking area or enters standby mode, waiting for the next inspection task to start.

[0254] Through the above process, this invention realizes closed-loop collaboration of autonomous navigation, multimodal data acquisition, adaptive compensation for missing modalities, and abnormal behavior detection of inspection robots, effectively improving navigation stability, anomaly detection robustness, and inspection visualization level in complex inspection scenarios.

[0255] In this embodiment, the inspection path planning and dynamic adjustment module is communicatively connected to the modal state detection module and the missing modality adaptive processing module, respectively. It is used to perform dynamic path replanning and supplementary data acquisition scheduling based on modal state information when receiving equipment anomaly alarm information or detecting a decline in modal data quality. During module operation, it first abstracts the inspection environment space into a graph structure based on a preset inspection environment map, denoted as […]. Where V represents the set of inspection nodes consisting of regular inspection points, channel nodes, and supplementary data collection target points, and E represents the set of edges corresponding to the traversable paths between nodes, thus completing the topological modeling of the inspection environment and providing a computable spatial basis for path search; subsequently, the module uses an improved A* algorithm to perform path optimization, and the algorithm's basic evaluation function satisfies the relational expression , This represents the total path cost corresponding to the current node. The heuristic cost estimate from the current node to the target node is typically calculated using Euclidean distance or Manhattan distance, used to guide the path search direction and improve optimization efficiency. To adapt to the dynamic perception requirements of inspection scenarios and address the disconnect between traditional path planning and data collection needs, the module estimates the actual path cost. Optimization and improvement are carried out by introducing modal quality cost and environmental traversal complexity cost, and constructing multi-dimensional cost calculation rules that satisfy the relational formula. ;in This represents the path distance cost, calculated from the physical travel distance between the current nodes, reflecting the length cost of the path. The modal quality cost is calculated by the module based on the modal quality scores in the modal state information. The lower the modal quality and the higher the degree of missingness in the target area, the smaller the corresponding modal quality cost, thereby guiding the search path to prioritize covering low-quality perception areas. The environmental passage complexity cost is calculated by weighting parameters such as passage width, obstacle density, and pedestrian activity stored in the pre-set map, reflecting the passage difficulty and safety risks of the corresponding path; , , These are the weight coefficients corresponding to distance cost, modal quality cost, and environmental complexity cost, respectively. These can be preset and adjusted according to the priority requirements of the inspection task to achieve balanced optimization across multiple objectives. After constructing the cost function, the module searches within the graph structure to minimize the total path cost function. The node sequence is used to obtain the optimized inspection path. Satisfying the relation When the system detects modal loss or data quality degradation, or receives an alarm message indicating a complete loss of main vision, the module dynamically triggers a path replanning mechanism. Starting from the robot's current node and targeting the point to be supplemented with low modal quality, the module re-substitutes the improved cost function to complete the path search, generating an optimized inspection path and supplementary acquisition control instructions adapted to the current perception state. The instructions include the coordinates of the target supplementary acquisition point, the recommended travel speed, and the gimbal observation angle parameters. Finally, the module sends the supplementary acquisition control instructions to the motion execution end of the inspection robot through the internal communication bus, driving the robot chassis to move along the optimized path to the target point, and simultaneously coordinating with the gimbal to adjust the acquisition perspective, performing supplementary data acquisition on areas with modal loss or low quality to improve the data input quality for subsequent abnormal behavior detection.

[0256] Furthermore, the system generates alarm outputs based on the abnormal behavior identification results.

[0257] Let the confidence level of the abnormal behavior be S. c The system performs a judgment based on the preset alarm threshold:

[0258] ;

[0259] Wherein: T a This indicates the alarm threshold; Alarm=1 indicates that an alarm for abnormal behavior has been triggered.

[0260] When an alarm is triggered, the system performs operations such as audible and visual alarms, video recording and storage, tracking of key areas and status recording, and simultaneously updates the inspection status and AR inspection information.

[0261] Through the above implementation methods, the present invention realizes closed-loop collaboration of multimodal data acquisition, missing modality adaptive processing, abnormal behavior detection, and dynamic path adjustment for inspection robots in complex environments, thereby improving the robustness, real-time performance, and reliability of abnormal behavior detection in engineering applications.

[0262] For comparison of technical effects, existing technologies can be used as a reference. With the development of multimodal perception technology, more and more inspection systems are beginning to integrate multi-source information such as vision, audio, posture, and robot status to improve the accuracy of abnormal behavior detection and environmental perception capabilities. However, complex inspection environments often present problems such as changes in lighting, target occlusion, environmental noise, equipment vibration, and network fluctuations, which can easily lead to a decrease in the quality or even loss of some modal data, thereby affecting the stability and reliability of multimodal abnormal behavior detection results.

[0263] Secondly, existing systems lack dynamic perception and adaptive processing capabilities based on modal quality. Although they can collect visual, audio, and sensor data, most assume that each modality is always stable and available, and do not establish modal state detection mechanisms for complex inspection environments such as image occlusion, lighting changes, audio interference, and sensor anomalies. This leads to problems such as false detection and missed detection in scenarios where modal quality deteriorates or data is missing.

[0264] Furthermore, existing multimodal abnormal behavior detection methods generally lack the ability to differentiate compensation for scenarios where different modalities are missing. When local visual information is missing, auxiliary modalities are abnormal, or the primary visual modality fails, the system usually cannot dynamically switch detection strategies, nor can it utilize multi-view visual information and auxiliary modalities for collaborative compensation, resulting in poor robustness of abnormal behavior detection.

[0265] Meanwhile, existing multimodal fusion methods typically employ fixed-weight fusion or simple feature splicing, lacking a dynamic weighted fusion mechanism that incorporates modal quality status. They cannot adjust feature contribution in real time based on the importance and availability of different modalities, making them susceptible to interference from low-quality modalities and affecting the accuracy of the final abnormal behavior identification results.

[0266] Furthermore, existing inspection path planning methods primarily focus on path distance and obstacle avoidance requirements, lacking a coordination mechanism with multimodal data acquisition needs. When modalities are missing or data quality deteriorates during inspection, the system cannot dynamically adjust the acquisition points, inspection path, and pan-tilt-zoom (PTZ) perspective based on the modal status, resulting in low efficiency in supplementary data acquisition.

[0267] In addition, the existing abnormal behavior detection system lacks sufficient linkage with the robot navigation control. The abnormal detection results are usually only used for alarm output and do not form a closed-loop collaborative mechanism with the robot chassis movement, gimbal control and supplementary data acquisition strategy, making it difficult to achieve autonomous dynamic response in complex inspection scenarios.

[0268] Finally, most existing AR inspection systems only have basic navigation and information display functions, lacking the ability to integrate and display inspection paths, abnormal alarm locations, equipment status and detection results in real time. They cannot form an integrated "detection-navigation-alarm" visual inspection mechanism, which is not conducive to inspection personnel quickly locating abnormal areas and carrying out remote collaborative handling.

[0269] In summary, although existing technologies have made some progress in areas such as inspection robots, autonomous navigation, and abnormal behavior detection, the following shortcomings still exist:

[0270] 1. There is a lack of a multimodal dynamic weighted fusion mechanism based on modal quality status. Existing methods usually use fixed weights or simple feature splicing, which cannot dynamically adjust the fusion weights according to the data quality and availability of different modalities. They are easily affected by low-quality modal interference, which affects the accuracy of abnormal behavior detection.

[0271] 2. The lack of a modal state detection and missing identification mechanism for complex inspection environments makes it difficult for the existing system to identify modal abnormalities in a timely manner when there are image occlusions, audio noise, sensor malfunctions, or data missing, resulting in unstable subsequent detection results.

[0272] 3. There is a lack of differentiated adaptive compensation strategies for different types of modal loss. Existing methods usually adopt a uniform processing approach, which cannot dynamically switch compensation modes and detection strategies according to different scenarios such as local visual loss, auxiliary modality anomaly or main modality failure.

[0273] 4. Lacking a collaborative mechanism for multi-view visual complementarity and auxiliary modality enhancement, existing systems struggle to recover behavioral feature expression capabilities using other visual perspectives and auxiliary modalities in cases of partial visual occlusion or target absence.

[0274] 5. There is a lack of a dynamic path adjustment mechanism that combines modal quality status with supplementary data collection needs. Existing inspection path planning methods mainly focus on path distance and obstacle avoidance requirements, and cannot dynamically adjust the inspection path and supplementary data collection points according to changes in data quality.

[0275] 6. The lack of a closed-loop collaborative control mechanism of "modal quality assessment - missing data compensation - anomaly detection - path replanning" leads to the independence of abnormal behavior detection, data collection and path navigation, making it difficult to form autonomous linkage capabilities in complex inspection scenarios.

[0276] 7. The system lacks the ability to display anomaly detection results in real time and AR inspection information. The existing system is unable to achieve a visual integration display of anomaly alarm location, inspection path, equipment status and detection results, which is not conducive to on-site inspection and remote collaborative management.

[0277] While existing abnormal behavior detection systems for inspection robots have incorporated technologies such as multimodal perception, mobile robot navigation, and artificial intelligence detection, several significant shortcomings remain, making it difficult to meet the demands for real-time abnormal behavior detection, autonomous path adjustment, and robust handling of multimodal missing features in complex inspection environments. By analyzing the problems of existing technologies, this invention proposes a modality-quality-driven, multimodal missing feature adaptive abnormal behavior detection method, achieving synergistic optimization among abnormal behavior detection, modality compensation, dynamic path adjustment, and inspection control. Therefore, the main drawbacks of existing technologies are as follows:

[0278] 1. Insufficient dynamic sensing capability of modal quality:

[0279] Existing inspection systems typically assume that visual, audio, and sensor data are always stable and available. They lack modal state detection and quality assessment mechanisms for situations such as image occlusion, lighting changes, audio noise, and equipment malfunctions, which makes the system prone to false detections and missed detections in complex environments.

[0280] 2. Insufficient dynamic fusion capability of multimodal information:

[0281] Most existing multimodal abnormal behavior detection methods use fixed-weight fusion or simple feature splicing, which cannot dynamically adjust the fusion weights according to the data quality, importance and availability of different modalities. This makes it easy for low-quality modalities to interfere with the detection results and affect the accuracy of abnormal behavior recognition.

[0282] 3. Insufficient adaptive compensation capability for missing modes:

[0283] When local visual information is missing, auxiliary modality is abnormal, or the main visual modality fails, existing systems usually lack differentiated processing mechanisms for different missing scenarios, cannot dynamically switch detection modes, and find it difficult to use multi-view visual information and auxiliary modality for collaborative compensation.

[0284] 4. Insufficient multi-view visual collaboration capability:

[0285] Existing inspection systems typically rely on a single visual perspective for abnormal behavior detection. When the target is occluded or some areas cannot be captured effectively, the system struggles to compensate for the target area using other visual perspectives, resulting in incomplete representation of behavioral characteristics.

[0286] 5. Insufficient coordination between inspection route planning and data collection:

[0287] Existing inspection path planning methods mainly focus on path length and obstacle avoidance requirements, lacking the ability to dynamically adjust paths by combining modal quality status and supplementary data collection requirements. When modal anomalies are detected, they cannot autonomously adjust the inspection path and data collection location.

[0288] 6. Insufficient linkage between anomaly detection and robot control:

[0289] Existing abnormal behavior detection results are usually only used for alarm output, lacking real-time linkage with robot movement control, gimbal adjustment and supplementary data collection mechanisms, making it difficult to form autonomous response capabilities in complex inspection scenarios.

[0290] 7. Insufficient visualization capabilities for AR inspection information:

[0291] Most existing AR inspection systems only have basic navigation and information display functions, and cannot integrate abnormal alarm locations, inspection paths, equipment status and detection results in real time, making it difficult to achieve integrated visual inspection of "detection-navigation-alarm".

[0292] Therefore, this invention aims to solve the following technical problems (multiple technical problems existing in the current process of intelligent museum navigation and cultural relic security supervision):

[0293] 1. Insufficient heterogeneity of multimodal data and lack of unified modeling capabilities:

[0294] Inspection robots need to process multi-source heterogeneous data such as visual images, environmental audio, human posture and robot status simultaneously during operation. However, existing systems usually adopt simple feature splicing or single-modal independent processing methods, lacking unified feature representation, cross-modal alignment and dynamic fusion mechanisms, which makes it difficult for different modalities to form effective collaboration, affecting the accuracy and stability of abnormal behavior detection.

[0295] 2. Dynamic fluctuations in modal quality lead to insufficient robustness in detection:

[0296] Complex inspection environments are prone to issues such as lighting variations, equipment vibration, signal interference, target occlusion, and data loss, leading to continuous fluctuations in the quality of visual, audio, and other modal data. Existing abnormal behavior detection methods lack dynamic evaluation and adaptive processing capabilities based on modal quality, making them susceptible to false positives, false negatives, and unstable detection results.

[0297] 3. Insufficient adaptive compensation capability in scenarios with missing modalities:

[0298] When some modal data is missing or abnormal, existing systems are usually unable to implement differentiated processing strategies for different types of missing data. They lack multi-view visual complementarity, auxiliary modal enhancement and degradation detection mechanisms, resulting in weak adaptability of the system to modal missing scenarios in complex inspection environments.

[0299] 4. Insufficient dynamic weight allocation capability during multimodal feature fusion:

[0300] Traditional multimodal fusion methods mostly adopt a fixed weight fusion mechanism, which does not dynamically adjust the contribution of each modality according to modal quality, availability, and environmental changes. This results in low-quality modalities interfering with the detection results, and the advantages of high-quality modalities cannot be fully utilized, thus affecting the accuracy of abnormal behavior recognition.

[0301] 5. Insufficient coordination between inspection path planning and multimodal data acquisition requirements:

[0302] Existing inspection path planning methods mainly focus on path length and obstacle avoidance requirements, lacking a dynamic path planning mechanism that combines modal quality status, supplementary data collection requirements, and environmental complexity. This results in some inspection areas failing to acquire high-quality data, affecting the subsequent abnormal behavior detection effect and inspection efficiency.

[0303] 6. The abnormal behavior detection and dynamic path adjustment lack a closed-loop linkage mechanism:

[0304] Existing inspection systems typically treat anomaly detection, modal compensation, and path planning as independent modules, lacking a closed-loop collaborative mechanism of "modal quality assessment - missing data compensation - anomaly detection - path replanning," and are unable to dynamically adjust the data collection points, inspection paths, and sensor parameters based on real-time detection results.

[0305] 7. Insufficient ability to integrate and display AR inspection information with anomaly detection results:

[0306] The existing AR inspection system does not realize the real-time linkage display of inspection path, abnormal alarm location, equipment status and detection results, making it difficult to form an integrated visual inspection mechanism of "detection-navigation-alarm", which is not conducive to inspection personnel quickly locating abnormal areas and carrying out remote collaborative handling.

[0307] To address the aforementioned technical problems, the present invention proposes:

[0308] 1. A unified modeling method for multimodal data in complex inspection scenarios, which realizes collaborative modeling and unified fusion of different modal data by uniformly encoding and representing multi-source heterogeneous data such as visual images, environmental audio, human posture and robot state, thereby improving the multimodal perception capability in complex environments.

[0309] 2. An abnormal behavior detection method based on dynamic evaluation of modal quality, which improves the robustness and stability of abnormal behavior detection in complex inspection environments by dynamically identifying modal missingness, occlusion and noise interference by real-time detection of the integrity, stability and availability of each modal data.

[0310] 3. An adaptive compensation method for different modality loss types, which dynamically executes multi-view visual complementarity, auxiliary modality enhancement and degradation detection strategies according to different scenarios such as visual modality loss, auxiliary modality anomaly and main modality failure, so as to improve the system's continuous detection capability in modality loss scenarios.

[0311] 4. A dynamic multimodal feature fusion method based on modality quality drive, which adaptively adjusts the fusion weight of each modality feature according to the quality status and availability of different modalities, reduces the interference of low-quality modalities on the detection results, and improves the accuracy of abnormal behavior recognition.

[0312] 5. An inspection path planning method that combines modal quality status and data acquisition requirements, which realizes dynamic path optimization and supplementary acquisition point planning in complex inspection environments by introducing modal quality cost, environmental access complexity and supplementary acquisition requirements into the path cost function.

[0313] 6. A closed-loop collaborative control method of "modal quality assessment - missing data compensation - anomaly detection - path replanning" is proposed. When modal anomalies or data quality degradation are detected, modal compensation, anomaly detection and path adjustment are dynamically executed in a coordinated manner to improve the autonomous inspection capability and adaptability of the inspection robot to complex environments.

[0314] 7. An AR inspection information linkage display method, which overlays the inspection path, abnormal alarm location, equipment status and test results with the real-time on-site scene information to realize an integrated visual inspection service for inspection navigation, abnormal location and status monitoring, thereby improving the real-time interactive capability and engineering application efficiency of the inspection process.

[0315] In summary, the main objectives of this invention include:

[0316] 1. Provide a dynamic detection mechanism based on modal quality state:

[0317] By constructing a modal state detection and quality assessment model, real-time quality analysis of multimodal data such as vision, audio, posture, and robot state can be achieved, thereby improving the ability to identify modal anomalies in complex environments.

[0318] 2. Provides multimodal dynamic fusion and robust abnormal behavior detection capabilities:

[0319] By combining modal quality states with a dynamic gating fusion mechanism, adaptive weighted fusion of multimodal features is achieved, improving the accuracy and stability of abnormal behavior detection.

[0320] 3. Provide differentiated compensation mechanisms for different modality loss scenarios:

[0321] For different scenarios such as local visual loss, auxiliary modality anomaly, and main visual modality failure, the corresponding compensation mode and detection strategy are dynamically switched to improve the system robustness in complex inspection environments.

[0322] 4. Provides behavioral feature recovery capabilities based on multi-view visual complementarity:

[0323] By fusing information from different visual perspectives and combining audio and posture modalities for auxiliary enhancement, behavioral feature compensation and recovery can be achieved in scenarios with local visual deficiencies.

[0324] 5. Provides dynamic path adjustment capabilities based on modal quality:

[0325] By combining modal quality status, environmental accessibility, and supplementary data collection needs, the inspection path is dynamically replanned to improve the integrity of data collection and inspection efficiency in complex environments.

[0326] 6. Provide a closed-loop collaborative mechanism for anomaly detection and robot autonomous control:

[0327] It enables coordinated control between abnormal behavior detection, modal compensation, path adjustment, and supplementary data collection, thereby improving the autonomous response capability of the inspection robot in complex scenarios.

[0328] 7. Provides the capability to integrate and display anomaly alarms with AR inspection information:

[0329] It enables real-time integrated display of abnormal behavior detection results, inspection paths, equipment status, and alarm information, improving the visualization management capabilities and efficiency of abnormal handling during the inspection process.

[0330] Specifically, the system consists of an inspection robot, a multimodal sensor unit, a data processing unit, a modal state detection and adaptive compensation unit, a multimodal abnormal behavior detection unit, and an alarm and control unit. The inspection robot is equipped with an RGB camera, an infrared camera, a depth camera or LiDAR, an audio sensor, and a motion sensor to collect data on human behavior, image information, audio information, and spatial location information in the environment.

[0331] This invention constructs a multimodal data processing and intelligent decision-making framework for modality-deficient scenarios. Through multimodal data alignment and unified modeling, it achieves the fusion of visual, audio, and posture information. Furthermore, it introduces a modality state detection and quality assessment mechanism to determine the availability of each modality's data in real time. Based on this, it designs adaptive compensation and degradation strategies for missing modalities. Through multi-view visual complementarity, primary and secondary modality collaboration, and modality switching mechanisms, it improves the robustness of abnormal behavior detection in occlusion, noise interference, and complex environments. Simultaneously, combined with an inspection path planning and dynamic adjustment mechanism, it achieves proactive data compensation by adjusting the inspection path and observation perspective when modality deficiencies or data quality degradation are detected.

[0332] Based on this, the system can detect abnormal behavior in both single and multiple individuals, and supports stable identification even when multimodal information is incomplete. Each module interacts with data and schedules tasks through a unified communication mechanism, and, combined with control strategies, drives the robot to perform motion adjustments and device linkages, thereby achieving integrated operation of abnormal behavior detection, real-time alarms, and system status feedback.

[0333] This invention is applicable to scenarios such as unattended inspection, public safety monitoring, and intelligent security. It can significantly improve the accuracy of abnormal behavior detection and the stability of system operation under conditions of missing or interfered modal information.

[0334] Through the above-mentioned technical means, this invention constructs a unified intelligent inspection system with modal quality driving as the core, integrating multimodal state detection, adaptive missing compensation, multimodal dynamic fusion, and dynamic adjustment of inspection path. It realizes the collaborative optimization of abnormal behavior detection, data acquisition, and robot autonomous inspection control, and improves the robustness, real-time performance, and engineering application reliability of the system in complex inspection environments.

[0335] Furthermore, the key point of this invention is:

[0336] 1. A multimodal anomaly behavior detection mechanism based on modal quality-driven approach:

[0337] This invention enables real-time detection of modal missingness, occlusion, noise interference, and data anomalies by performing online quality assessment of multimodal data such as vision, audio, posture, and robot status. It also dynamically executes corresponding adaptive processing strategies based on different missing states, thereby improving the robustness and stability of abnormal behavior detection in complex inspection environments.

[0338] 2. Missing mode adaptive compensation mechanism:

[0339] Differentiated compensation strategies are developed for scenarios with different modalities missing, including:

[0340] Multi-view visual complementarity when local visual modalities are missing;

[0341] Visual degradation detection when auxiliary modalities are missing;

[0342] A safety stop and anomaly alarm mechanism when the main visual modality is completely missing.

[0343] The system's continuous detection capability in complex environments is improved by using a tiered processing approach.

[0344] 3. Multimodal feature fusion and abnormal behavior recognition mechanism:

[0345] This invention integrates visual features, audio features, posture features, and robot state features, and combines them with a gating enhancement mechanism to construct a complete behavioral semantic representation, enabling accurate identification of abnormal behaviors such as smoking, falling, climbing, fighting, and pushing.

[0346] 4. Dynamic inspection path adjustment mechanism based on improved A* algorithm:

[0347] This invention, based on the traditional A* path planning algorithm, introduces modal quality cost and environmental complexity cost to dynamically optimize the inspection path. When a modality is missing or data quality deterioration is detected, the system automatically triggers path replanning and supplementary data collection, improving the integrity of data collection and the robustness of inspection in complex scenarios.

[0348] 5. Closed-loop collaborative mechanism for inspection navigation and anomaly detection:

[0349] This invention realizes a closed-loop linkage of "data acquisition - modal quality assessment - missing modality compensation - abnormal behavior detection - dynamic path adjustment", enabling the inspection robot to adjust its perception and navigation strategies in real time according to environmental changes, thereby improving the level of intelligent autonomous inspection.

[0350] The point to be protected by this invention is:

[0351] 1. A modal quality-driven multimodal missing adaptive abnormal behavior detection method, which achieves robust identification of abnormal behavior in complex inspection environments by performing quality assessment and state detection on multimodal data such as vision, audio, posture and robot operation status.

[0352] 2. A compensation mechanism that performs hierarchical adaptive processing for different types of modality loss, dynamically switching the corresponding compensation strategy, degradation detection strategy and security alarm strategy according to different scenarios such as local visual modality loss, auxiliary modality loss and complete loss of main visual modality.

[0353] 3. An abnormal behavior detection method that combines multi-view visual complementarity and auxiliary modality enhancement. By fusing information from different visual perspectives and combining audio modality and human posture modality for auxiliary enhancement, it can realize the recovery of behavioral features and recognition of abnormal behavior in occluded, low-light and partially missing scenes.

[0354] 4. An inspection navigation method based on modal quality assessment results triggering dynamic path adjustment. When modal missingness or data quality degradation is detected, the robot inspection path, supplementary collection points and gimbal collection angle are automatically adjusted to improve the data collection integrity and inspection reliability in complex scenarios.

[0355] 5. A multimodal abnormal behavior recognition method that integrates visual, audio, human posture and robot operating status information, which realizes the classification, recognition and confidence calculation of abnormal behaviors such as smoking, falling, climbing, fighting and pushing by constructing a unified multimodal feature representation.

[0356] 6. An inspection path planning method based on an improved A∗ algorithm that integrates modal quality cost and environmental travel complexity cost. During the path search process, the method comprehensively considers path distance, modal quality risk and environmental dynamic state to achieve autonomous navigation and dynamic path replanning in complex inspection environments.

[0357] 7. A closed-loop collaborative control method for inspection robots, which integrates "modal perception, quality assessment, missing component compensation, anomaly detection, and path adjustment," achieves dynamic feedback between perception, compensation, detection, and navigation through multi-module linkage, thereby improving the real-time performance, stability, and engineering application reliability of the inspection system in complex environments.

[0358] 8. A robust multimodal missing abnormal behavior detection system suitable for complex inspection scenarios, comprising a multimodal perception module, a modal state detection module, an adaptive compensation module, a multimodal fusion module, an abnormal behavior detection module, and a dynamic path adjustment module, for realizing autonomous inspection and intelligent detection of abnormal behavior in complex environments.

[0359] 9. A multimodal feature fusion method based on dynamic adjustment of modal quality weights, which adaptively adjusts the weight allocation of each modality in the fusion process according to the real-time quality scores and available status of different modalities such as vision, audio, posture and robot state, so as to increase the feature contribution ratio of high-quality modalities, suppress the interference of low-quality modalities on the detection results, and improve the accuracy and robustness of abnormal behavior recognition in complex inspection environments.

[0360] 10. A hardware control method for an inspection robot based on modal quality feedback, which automatically adjusts the robot chassis movement speed, gimbal rotation angle and sensor acquisition parameters according to the modal data quality assessment results, so as to achieve precise matching between hardware equipment and inspection requirements and improve the stability and adaptability of inspection robot hardware operation.

[0361] Compared with existing technologies, this invention, based on existing abnormal behavior detection technologies for inspection robots, introduces a modal quality-driven mechanism, a multimodal missing adaptive processing mechanism, a multi-view visual complementarity mechanism, and a dynamic path adjustment mechanism to achieve an integrated intelligent system for abnormal behavior detection and autonomous inspection control in complex inspection environments. Compared with existing technologies, this invention has the following significant advantages:

[0362] 1. Stronger detection robustness in multimodal missing scenarios:

[0363] Compared to traditional anomaly detection methods that assume all modalities are always valid, this invention can perform real-time quality assessment and state detection on multimodal data such as vision, audio, posture, and robot state, and dynamically adjust the detection strategy according to the modal state, thereby improving the stability and reliability of anomaly behavior detection in complex environments.

[0364] 2. Possesses adaptive handling capability for missing modes:

[0365] Compared to existing fixed detection processes, this invention can automatically switch between corresponding compensation modes, downgraded detection modes, and safety alarm modes for different situations such as missing local visual modalities, abnormal auxiliary modalities, and failure of the main visual modalities, thereby achieving continuous detection and autonomous adaptation in complex inspection scenarios.

[0366] 3. Enhanced multi-view visual complementarity:

[0367] Compared to single-view visual detection methods, this invention can supplement the target area by using other normal visual perspectives under conditions of image occlusion, partial loss, or low illumination, and combine audio modality and human posture modality for auxiliary enhancement, thereby improving the ability to recover behavioral features and the accuracy of abnormal behavior recognition.

[0368] 4. The multimodal fusion method is more dynamic and intelligent:

[0369] Compared to traditional fixed-weight fusion methods, this invention dynamically adjusts the fusion weights based on the real-time quality status of different modalities, increasing the contribution of high-quality modalities to the anomaly detection results while reducing the interference of low-quality modalities, thereby improving the accuracy of anomaly behavior recognition in complex scenarios.

[0370] 5. Enhanced integration of anomaly detection and path planning:

[0371] Compared to traditional standalone path planning systems, this invention incorporates modal quality status and anomaly detection results into the inspection path planning process. When the system detects modal missingness or data quality degradation, it can automatically trigger a path replanning mechanism to dynamically adjust the robot's inspection path, supplement data collection points, and adjust the gimbal's acquisition angle, thereby improving the integrity of inspection data collection and the efficiency of autonomous inspection.

[0372] 6. Stronger adaptability to complex environments:

[0373] Compared to existing inspection systems, this invention can dynamically adjust detection strategies and inspection behaviors based on complex situations such as obstruction, low light, changes in pedestrian flow, equipment malfunctions, and dynamic environmental changes, thereby improving the system's stable operation and reliability in complex inspection environments.

[0374] 7. Enhanced ability to respond to abnormal events:

[0375] This invention can link abnormal behavior detection results with the robot control module to achieve autonomous response operations such as gimbal tracking, supplementary data collection in key areas, video recording and storage, audible and visual alarms, and path adjustment, thereby improving the efficiency of abnormal event location and on-site handling capabilities.

[0376] 8. The system's closed-loop collaborative capability is more complete:

[0377] This invention realizes a closed-loop collaborative control mechanism of "modal perception - quality assessment - missing data compensation - feature fusion - anomaly detection - path adjustment", which has stronger real-time performance, adaptability and system collaboration capabilities compared with traditional single detection systems.

[0378] In summary, this invention outperforms existing technologies in terms of multimodal missing robustness, adaptive compensation capability, multi-view collaborative detection capability, dynamic path planning capability, and adaptability to complex environments. It achieves an integrated intelligent enhancement of abnormal behavior detection and autonomous inspection control, and has good engineering application prospects and promotional value.

[0379] Furthermore, the technical solution of this invention can be modified or replaced in various ways without departing from the basic principles of this invention, in order to adapt to different application scenarios or technical conditions. The following are some modified designs and extended applications of this invention.

[0380] 1) Design changes and alternative solutions:

[0381] 1. Alternatives to multimodal information modeling and fusion mechanisms:

[0382] In this invention, multimodal data is fused through unified encoding, cross-modal alignment, and dynamic modeling mechanisms. In specific implementations, this process can be carried out in different ways, for example:

[0383] Different types of feature encoding models can be used to learn representations of visual, speech, text, and environmental data;

[0384] Cross-modal alignment can be achieved using linear mapping, nonlinear mapping, or attention-based alignment methods.

[0385] The multimodal fusion process can be replaced by sequence modeling structures, attention mechanism models, or other dynamic modeling methods;

[0386] In resource-constrained scenarios, model compression, parameter quantization, or lightweight network structures can be used to reduce computational overhead.

[0387] The above-mentioned different implementation methods are all equivalent substitutes for the multimodal unified modeling and fusion mechanism of this invention.

[0388] 2. Alternatives to semantic enhancement and intent reasoning mechanisms:

[0389] This invention achieves the process from semantic understanding to decision generation through semantic completion and intent reasoning. In different embodiments, this part can take the following alternative forms:

[0390] The semantic completion process can be implemented based on different types of knowledge bases, knowledge graphs, or external knowledge models;

[0391] Common sense reasoning mechanisms can be implemented using generative models, retrieval-enhanced models, or rule-based reasoning methods.

[0392] User intent recognition can be achieved using classification models, sequence decision models, or dialogue context-based reasoning models.

[0393] The strategic decision-making process can be implemented based on rule-based strategies, learning strategies, or hybrid decision-making mechanisms;

[0394] All of the above alternative methods can achieve the basic functions of semantic enhancement and intent-driven decision-making.

[0395] 3. Alternative solutions for task scheduling and system architecture:

[0396] This invention uses the ROS framework to implement robot control and task scheduling. In other embodiments, a system architecture with similar functionality can be used, for example:

[0397] Node communication and task scheduling are achieved using ROS2 or other distributed robot operation frameworks.

[0398] Implement multi-module collaboration and data flow management using custom middleware or microservice architecture;

[0399] Based on the system scale and real-time requirements, the task scheduling strategy can be optimized in a centralized or distributed manner.

[0400] All of the above alternative solutions can achieve the same system collaborative control effect as the present invention.

[0401] 4. Alternatives to edge computing and cloud-based collaborative mechanisms:

[0402] This invention achieves hierarchical task processing through the collaboration of edge computing and cloud-based intelligent inference. In other embodiments, adjustments can be made according to actual needs, for example:

[0403] Migrate some cloud functions to the edge to achieve localized inference;

[0404] Adopt a pure cloud-based centralized processing or multi-edge node collaborative processing mode;

[0405] The allocation strategy for computing tasks is dynamically adjusted based on network conditions.

[0406] The above methods are all equivalent implementations of the edge-cloud collaborative computing mechanism.

[0407] 5. Extensions or alternatives to the sensor system:

[0408] In this invention, the robot acquires environmental and cultural relic information through multiple sensors. In different implementations, the following adjustments can be made:

[0409] Replace existing cameras with visual sensing devices of different types or precision;

[0410] Add sensing devices such as infrared, depth, thermal imaging, or millimeter-wave radar;

[0411] Expand the types of environmental monitoring sensors, such as air quality, light intensity, or vibration monitoring;

[0412] The above-mentioned substitutions and extensions are all equivalent implementations of the sensing system of this invention.

[0413] 6. Expanding the tour guide and inspection functions:

[0414] Based on the system of this invention, the following functions can be further expanded:

[0415] Supports multilingual interaction, personalized recommendations, or cross-device collaborative navigation;

[0416] Expand augmented reality display formats to achieve immersive interactive experiences;

[0417] Added functions for inspection plan management, abnormal event recording, and historical data analysis;

[0418] Based on the data analysis results, provide a decision support interface for the management system;

[0419] The above extensions do not change the core technical principles of this invention.

[0420] 1. Alternatives to multimodal sensing devices and data acquisition methods:

[0421] In this invention, the multimodal perception data mainly includes visual images, environmental audio, human posture, and robot operating status information. In specific implementation, the perception devices and acquisition methods can be replaced or expanded according to different inspection scenarios. For example, in addition to ordinary RGB cameras, visual acquisition devices can also use infrared cameras, thermal imaging devices, depth cameras, panoramic cameras, or LiDAR for environmental perception.

[0422] Audio acquisition equipment can employ array microphones, directional pickup devices, or ambient sound sensors;

[0423] Robot operating status information can be collected through inertial navigation modules, encoders, positioning modules, motor status detection modules, or environmental sensors;

[0424] Human posture information can be obtained through key point detection equipment, depth vision equipment, or skeleton recognition modules.

[0425] 2. Alternative solutions for multimodal feature processing and fusion methods:

[0426] In this invention, multimodal data is used to detect abnormal behavior through a unified feature representation and dynamic fusion mechanism. In specific implementations, this process can be implemented in different ways, for example:

[0427] The process of extracting features from different modalities can be implemented using convolutional neural networks, Transformers, state-space models, or other deep learning models;

[0428] Cross-modal feature alignment can be achieved using linear mapping, nonlinear mapping, gating mechanisms, or dynamic fusion methods based on attention mechanisms.

[0429] The fusion weights can be dynamically adjusted based on modal quality scores, environmental conditions, or task requirements.

[0430] In resource-constrained scenarios, lightweight models, edge computing devices, or distributed inference methods can also be used for deployment.

[0431] 3. Alternative solutions to missing modality compensation and anomaly handling methods:

[0432] In this invention, when some modalities are missing or data quality degrades, the system executes corresponding compensation and degradation processing strategies. In specific implementations, this part can be implemented in different ways, for example:

[0433] When a local visual area is obstructed, multi-view supplementary acquisition, gimbal turning supplementary acquisition, or cross-camera collaborative methods can be used to restore the target area information;

[0434] Missing mode compensation can be achieved through feature prediction, historical data-assisted analysis, cross-modal mapping, or generative compensation methods.

[0435] When the auxiliary modality is abnormal, the visual-dominated detection mode or the dynamic degradation detection mode can be used to continue the abnormal behavior recognition.

[0436] When the primary visual modality fails, emergency measures such as safe shutdown, remote takeover, or autonomous return can be used for emergency handling.

[0437] 4. Alternative solutions to inspection path planning and robot control methods:

[0438] In this invention, autonomous inspection and supplementary data collection in complex environments are achieved through a dynamic path planning mechanism. In specific implementation, this part can be implemented in different ways, for example:

[0439] In addition to the improved A* algorithm, the path planning process can also be implemented using Dijkstra's algorithm, D* algorithm, artificial potential field method or reinforcement learning path planning method;

[0440] In addition to considering path distance, the path cost function can also incorporate factors such as modal quality risk, obstacle density, environmental complexity, and task priority.

[0441] 5. Alternatives to inspection robot platforms:

[0442] This invention is not only applicable to wheeled inspection robots, but also to tracked robots, quadruped robots, drone inspection platforms, or multi-robot collaborative inspection systems.

[0443] The robot's movement mechanism, gimbal structure, and navigation module can all be adjusted according to the inspection environment.

[0444] In addition to wheeled inspection robots, robot platforms can also use tracked robots, quadruped robots, drone inspection platforms, or multi-robot collaborative inspection systems.

[0445] The robot control process can also be expanded by combining gimbal-linked control, automatic tracking of key areas, and dynamic re-sampling mechanisms.

[0446] 2) Other uses and application scenarios:

[0447] Although this invention is primarily applied to scenarios involving abnormal behavior detection and autonomous inspection of inspection robots, its technical solutions are also applicable to other scenarios requiring multimodal perception, missing modality compensation, and intelligent analysis of dynamic environments, including but not limited to:

[0448] 1. Smart Park and Public Safety Inspection:

[0449] It is suitable for park security, campus safety, shopping mall inspection and public area safety supervision, realizing abnormal behavior detection, dangerous event early warning and autonomous inspection management.

[0450] 2. Industrial equipment inspection and fault monitoring:

[0451] It is suitable for industrial scenarios such as factory workshops, power equipment, petrochemical parks and smart mines, and can realize equipment anomaly detection, dangerous behavior identification and unmanned inspection in complex environments.

[0452] 3. Supervision of rail transit and transportation hubs:

[0453] It is suitable for subway stations, high-speed rail stations, airports and transportation hubs to realize abnormal human behavior detection, key area monitoring and dynamic inspection navigation.

[0454] 4. Warehousing and Logistics and Unmanned Warehouse Inspection:

[0455] It is suitable for smart warehousing, logistics parks and unmanned warehouse environments, enabling cargo area inspection, abnormal event detection and multi-robot collaborative navigation.

[0456] 5. Dangerous areas are not inspected:

[0457] It is suitable for high temperature, high pressure, toxic, harmful or disaster environments, and can realize autonomous inspection and intelligent identification of abnormal events in complex and dangerous areas.

[0458] 6. Smart Cities and Emergency Safety Management:

[0459] It is suitable for urban security, emergency rescue and large-scale event monitoring scenarios, and can realize abnormal behavior detection, dynamic path planning and on-site auxiliary decision-making in complex environments.

[0460] In summary, this invention is not only applicable to the field of abnormal behavior detection of inspection robots, but can also be extended to various complex environment intelligent perception and autonomous inspection scenarios, and has good scalability, engineering adaptability and industrial application value.

[0461] In this embodiment of the invention, a multimodal missing adaptive abnormal behavior detection system based on an inspection robot is provided. The system includes a multimodal data acquisition module, a data processing module, a modality state detection module, a missing modality adaptive processing module, and a multimodal abnormal behavior detection module, which are sequentially connected in communication. The multimodal data acquisition module is used to acquire multimodal data. The data processing module is used to preprocess the multimodal data to obtain preprocessed multimodal features. The modality state detection module is used to perform multimodal data quality assessment on the preprocessed multimodal features and generate modality state information. The missing modality adaptive processing module is used to determine the current modality missing type based on the modality state information, and selectively perform feature compensation or invalid modality removal on the preprocessed multimodal features based on the current modality missing type to obtain an adaptive modality. The system employs multimodal features of the pre-perception state and a multimodal abnormal behavior detection module. This module dynamically configures multimodal feature fusion weights based on modal state information and calculates behavior confidence scores by combining them with multimodal features adapted to the current perception state. The behavior confidence scores are then compared with preset anomaly detection thresholds to output abnormal behavior detection results. Based on this approach, the modal state detection module perceives the data quality status of each modality in real time. A missing modality adaptive processing module then performs differentiated feature compensation or invalid modality removal for different scenarios such as local visual loss and auxiliary modality failure. Combined with a dynamically weighted multimodal fusion detection mechanism, this system can continuously output stable and reliable abnormal behavior detection results in complex scenarios where modal quality fluctuates or is partially missing due to occlusion, noise interference, sensor anomalies, etc., significantly improving the robustness of abnormal behavior detection.

[0462] Please see Figure 4 , Figure 4 This is a flowchart illustrating a multimodal missing adaptive abnormal behavior detection method based on an inspection robot, as provided in Embodiment 2 of the present invention.

[0463] This invention provides a multimodal missing adaptive abnormal behavior detection method based on an inspection robot, comprising:

[0464] Step 401: Collect multimodal data and preprocess the multimodal data to obtain preprocessed multimodal features;

[0465] Step 402: Based on the preprocessed multimodal features, conduct modal quality assessments and generate modal state information;

[0466] Step 403: Determine the current modality missing type based on the modality state information, and selectively perform feature compensation or invalid modality removal on the preprocessed multimodal features based on the current modality missing type to obtain multimodal features that are adapted to the current perception state;

[0467] Step 404: Dynamically configure the multimodal feature fusion weights based on the modal state information, and calculate the behavior confidence score by combining the multimodal features adapted to the current perception state. After comparing the behavior confidence score with the preset anomaly judgment threshold, output the abnormal behavior detection result.

[0468] It should be noted that, as Figure 5 As shown, this invention achieves stable detection and adaptive processing of abnormal behavior in complex environments by real-time evaluation of multimodal data quality and dynamic adjustment of detection strategies based on modality missing conditions.

[0469] (1) Inspection route planning and dynamic adjustment

[0470] Based on the environmental information of the inspection area, areas with high incidence of abnormal behavior, and multimodal data collection requirements, the robot inspection path is planned, and multimodal data collection points are set. When a decrease in modal quality or a lack of modality is detected, the inspection path and the gimbal collection angle are dynamically adjusted to supplement the data collection of the target area.

[0471] (2) Multimodal data acquisition

[0472] Visual data, spatial location data, human posture data, and environmental audio data are acquired through RGB cameras, infrared cameras, depth cameras, LiDAR, and audio acquisition devices to form a multimodal data set.

[0473] (3) Multimodal data preprocessing and spatiotemporal alignment

[0474] The collected multimodal data is filtered, denoised, format-converted, and feature-extracted. Multimodal data alignment is completed based on timestamps and spatial coordinates to construct a unified multimodal feature representation.

[0475] (4) Modal state detection and quality assessment

[0476] The device performs device status detection and data quality assessment on visual, audio, and posture modal data, determines the integrity, clarity, availability, and signal quality of each modal data, and outputs modal status information.

[0477] (5) Adaptive processing of missing modes

[0478] Based on the modality state assessment results, the modality missing situation is determined, and corresponding adaptive processing strategies are executed, including multi-view data complementation, auxiliary modality enhancement, downgraded detection, path adjustment and supplementary acquisition, etc., to improve the detection stability in modality missing scenarios.

[0479] (6) Multimodal feature fusion

[0480] The original or compensated multimodal features are fused to construct a unified multimodal fusion feature representation.

[0481] (7) Abnormal behavior detection

[0482] Based on the fused multimodal features, a deep learning-based multimodal abnormal behavior recognition model is used to identify and classify abnormal behaviors such as smoking, falling, climbing, fighting, and pushing.

[0483] (8) Abnormal alarms and result output

[0484] When abnormal behavior is detected, the system outputs the type, location, and confidence level of the abnormal behavior, and triggers linked operations such as audible and visual alarms, video storage, PTZ tracking, and recording of key areas to achieve real-time response and recording of abnormal events.

[0485] In this embodiment of the invention, a multimodal missing adaptive abnormal behavior detection method based on an inspection robot is provided. The method involves collecting multimodal data and preprocessing it to obtain preprocessed multimodal features; performing modal quality assessment based on the preprocessed multimodal features to generate modal state information; determining the current modal missing type based on the modal state information; selectively performing feature compensation or invalid modal removal on the preprocessed multimodal features based on the current modal missing type to obtain multimodal features adapted to the current perception state; dynamically configuring multimodal feature fusion weights based on the modal state information; calculating behavior confidence scores by combining the multimodal features adapted to the current perception state; comparing the behavior confidence scores with a preset abnormality judgment threshold; and outputting the abnormal behavior detection result. Based on the above scheme, by embedding modal quality assessment and adaptive compensation steps into the detection process, the feature combination and fusion strategy can be dynamically adjusted according to the real-time perception state, enabling abnormal behavior detection without relying on fixed and complete multimodal inputs, thereby improving the robustness of abnormal behavior detection.

[0486] Please see Figure 6 , Figure 6 This is a schematic diagram of a patrol robot based on Embodiment 3 of the present invention.

[0487] The present invention provides an inspection robot, comprising a mobile chassis 4, a main body 3, and a robot head 1 connected in sequence;

[0488] The robot's head 1 is equipped with a vision acquisition unit, a microphone array, and an audible and visual alarm. The vision acquisition unit includes an RGB camera, an infrared camera, and a depth camera.

[0489] The bottom of the mobile chassis 4 is equipped with omnidirectional wheels, and the mobile chassis 4 has a built-in lidar and motor status detection component;

[0490] A communication module 2 is provided on the side of the main body 3, and a control computing unit is built into the main body 3;

[0491] The control computing unit is electrically connected to the vision acquisition unit, microphone array, audible and visual alarm, and mobile chassis 4, respectively.

[0492] The control computing unit is used to receive multimodal data and preprocess it, and then upload the preprocessed multimodal features to the peripheral edge intelligent processing and decision-making center through the communication module 2.

[0493] The peripheral edge intelligence processing and decision-making center is equipped with a multimodal missing adaptive abnormal behavior detection system based on inspection robots.

[0494] It should be noted that the inspection robot is responsible for inspection task execution, environmental perception, and path movement control. It includes: a mobility system consisting of a wheeled chassis, drive motors, a motion controller, and an autonomous navigation module, used to enable the inspection robot's autonomous movement and path execution;

[0495] The PTZ control system is used to control the camera's attitude adjustment, enabling video acquisition and target tracking from different areas and multiple angles.

[0496] Power management system: Used to provide stable power to the various functional modules of the robot;

[0497] Communication module: Used to enable data communication between the robot and the monitoring platform.

[0498] like Figure 6 As shown, the navigation and inspection robot of this invention is the core execution terminal of the system, used to perform tasks such as intelligent inspection, multimodal data acquisition, and abnormal human behavior detection within the inspection scenario. The robot as a whole is composed of multiple hardware units, which operate collaboratively under the unified scheduling of the control and computing unit to realize environmental perception, inspection navigation, multimodal data processing, abnormal behavior detection, and alarm linkage functions, and form a collaborative working relationship with the edge intelligent processing and scheduling center.

[0499] Based on functional division, robot hardware mainly includes motion-related units, perception-related units, computing and control-related units, interaction and alarm-related units, and communication-related units.

[0500] (1) Motion-related unit: The motion-related unit is used to realize the robot's autonomous movement and path planning in the internal space of the inspection scene. It includes a differential wheel mobile chassis, a motor drive module and a power supply module, and also integrates a lidar device. The mobile chassis, as the robot's motion execution mechanism, receives motion control commands under the control of the control computing unit to realize actions such as forward movement, turning and following preset paths; the lidar is used to acquire environmental spatial structure information, supporting the robot to realize map building, autonomous localization, navigation and obstacle avoidance and dynamic path adjustment. During operation, the motion control combines the multimodal data quality assessment results from the perception-related unit to form a closed-loop control mechanism. The control computing unit can dynamically adjust the motion commands according to the real-time perception results and modality missing conditions. When a decrease in data quality or modality missing is detected, the inspection path is automatically adjusted and supplementary data is collected in the relevant areas, thereby ensuring the stable inspection operation of the robot in complex scene environments.

[0501] (2) Perception-related unit The perception-related unit is used to acquire multi-source environmental information and personnel behavior information in the inspection scene. It is integrated into the robot gimbal mechanism and the main body of the robot, including visual perception equipment, audio perception equipment and posture perception support module.

[0502] Visual perception devices include RGB cameras, infrared cameras, and depth cameras. RGB cameras are used to collect visual information about human movements and capture behavioral characteristics such as smoking and climbing. Infrared cameras are used to collect information on changes in human body temperature to assist in the identification of abnormal behavior. Depth cameras are used to acquire information on the spatial location and movement trajectory of people, providing spatial dimension support for judging behavioral status.

[0503] The audio sensing device is a microphone array integrated on the side of the pan-tilt unit, used to collect environmental audio signals and capture abnormal sound features such as arguments and falls, serving as an auxiliary modality in multimodal detection. After preliminary processing by the control and computing unit, the timestamps and spatial coordinates of the multimodal data are aligned to form a unified multimodal data structure, providing a data foundation for subsequent modal state detection, quality assessment, and abnormal behavior detection.

[0504] (3) The built-in control computing unit only undertakes local data acquisition and hardware scheduling functions, does not carry AI inference hardware, and does not perform intelligent operations such as modal quality inspection, feature compensation, and anomaly recognition. The control computing unit receives raw data from sensors such as vision, microphone array, lidar, and chassis motor, completes preprocessing, generates a preliminary feature data stream (preprocessed multimodal features), and then uploads it to the edge intelligent processing and decision-making center through the communication module of the fuselage. At the same time, it receives control commands for gimbal adjustment, chassis movement, and audible and visual alarms issued from the edge, and drives the corresponding hardware to complete the inspection execution actions. High-computing-power operations such as modal quality assessment, missing modality adaptive compensation, multimodal feature fusion, and abnormal behavior inference are all uniformly completed by the edge intelligent processing and decision-making center.

[0505] (4) Interaction and Alarm Related Unit The interaction and alarm related unit is used to realize abnormal behavior alarm and result feedback, including an audible and visual alarm and a display interaction device. The audible and visual alarm is integrated into the top of the PTZ mechanism. When the AI ​​computing unit determines that the confidence level of the abnormal behavior reaches the alarm threshold, it triggers the audible and visual alarm and realizes on-site warning through light and sound signals. At the same time, the display interaction device can simultaneously display alarm information, inspection status and AR navigation marks, and supports local storage of abnormal behavior video data, detection results and time information, which is convenient for subsequent traceability and management.

[0506] (5) Communication-related unit The communication-related unit is used to realize internal data interaction of the robot and data linkage with external systems. It is integrated into the upper part of the robot body and adopts wireless communication. The robot realizes two-way data interaction with the edge intelligent processing and scheduling center based on the communication module, including the issuance of inspection instructions, uploading of multimodal perception data, feedback of abnormal detection results, push of alarm information and feedback of equipment status; at the same time, it supports the system linkage triggered by the communication mechanism to realize camera linkage, automatic recording and key area tracking control when abnormal behavior occurs.

[0507] Through the above-mentioned robot hardware structure design, this invention realizes multi-source multimodal perception, adaptive processing of missing modalities, intelligent detection of abnormal behavior, and edge-robot collaborative execution capabilities, providing stable and reliable execution terminal support for intelligent inspection and personnel safety management in inspection scenarios.

[0508] Furthermore, such as Figure 7 As shown, the tour guide and inspection robot software system of the present invention adopts a layered architecture design, consisting of a perception layer, a data processing layer, an intelligent decision-making layer, and an application service layer from bottom to top. Each layer interacts with the other through standardized interfaces to achieve data exchange and business collaboration, forming a complete closed loop from multimodal data acquisition to abnormal behavior alarm and linkage control.

[0509] (1) Perception layer

[0510] The perception layer is used to collect multi-source heterogeneous data required for system operation and serves as the data input foundation for the software system. The data sources for this layer primarily come from the multimodal perception devices of the navigation and inspection robots, including various data generated by vision, audio, LiDAR, and motion control units. Specifically, the robot side provides not only personnel movement, body temperature, and spatial position information collected by RGB / infrared / depth cameras, but also environmental audio data collected by microphone arrays, spatial structure and positioning data output by LiDAR, and motion state and inspection path data generated by the mobile chassis.

[0511] The aforementioned multi-source data is uniformly encapsulated before entering the data processing layer, and includes timestamps and spatial coordinate information to ensure the alignment and traceability of different modal data during subsequent processing. All types of data are uploaded to the data processing layer via the ROS communication mechanism and standardized data interfaces, forming a continuously input multimodal data stream.

[0512] (2) Data processing layer

[0513] The data processing layer is used for preprocessing, quality assessment, and feature extraction of the multimodal data input from the perception layer. It serves as an intermediate processing unit connecting the perception layer and the intelligent decision-making layer. The core functions of this layer include usability detection, quality assessment, spatiotemporal alignment, and feature extraction of each modality's data. Specifically, the modality status detection and quality assessment module is responsible for detecting the online status and data quality of each device, identifying modality anomalies such as image occlusion and audio interference; the multimodal data spatiotemporal alignment module is responsible for aligning visual, audio, and posture data according to a unified temporal and spatial reference; and the human posture feature extraction module identifies key human points and extracts posture features based on visual data.

[0514] The processed, standardized data is transmitted upwards to the intelligent decision-making layer through an internal interface, providing reliable data input support for subsequent inspection planning, modal compensation, and abnormal behavior detection.

[0515] (3) Intelligent decision-making layer

[0516] The intelligent decision-making layer is the core logic control hub of the system, used to complete inspection path planning, modal adaptive compensation, and abnormal behavior detection decisions based on processed multimodal data. The core modules of this layer include an inspection path planning and decision-making module, a missing modality adaptive compensation module, a multimodal abnormal behavior detection module, and an execution control interface. Specifically, the inspection path planning module plans or dynamically adjusts the inspection path based on environmental information and modal acquisition status; the missing modality adaptive compensation module executes multi-view complementarity, auxiliary modality enhancement, or fault alarm strategies based on the type of missing modality; the multimodal abnormal behavior detection module identifies various abnormal behaviors of personnel based on multimodal feature reasoning; and the execution control interface is responsible for translating decision commands into robot motion and gimbal control actions.

[0517] This layer also receives alarm feedback information from the application service layer, enabling dynamic updates of the inspection strategy and forming a control closed loop.

[0518] (4) Application service layer

[0519] The application service layer provides business output and interaction services to users and operational scenarios, and is the final presentation unit of system functions. The core services of this layer include abnormal behavior alarm service, abnormal event recording and storage service, and system linkage and PTZ tracking service. Specifically, the abnormal behavior alarm service is responsible for triggering audible and visual alarms and pushing alarm information; the abnormal event recording and storage service is responsible for storing the video, detection results, and time information corresponding to abnormal behavior; and the system linkage and PTZ tracking service is responsible for triggering camera linkage, automatic recording, and tracking and control of key areas.

[0520] This layer interacts with the upper-layer system and terminal devices through standardized interfaces to realize business functions such as alarm processing, data traceability and operation and maintenance management, and feeds back the alarm processing results to the intelligent decision-making layer to support the continuous optimization of inspection strategies.

[0521] like Figure 8 As shown, this invention adopts an edge-edge collaborative interaction architecture consisting of a "robot local ROS control layer + edge intelligent processing layer". The robot local ROS control layer is responsible for real-time execution tasks such as multi-source data acquisition, motion control, and local alarms; the edge intelligent processing layer is responsible for highly complex intelligent computing tasks such as multimodal data processing, modal state evaluation, adaptive handling of missing modalities, and abnormal behavior detection. The two layers achieve data interaction and control collaboration through the ROS communication mechanism.

[0522] (1) Data acquisition function node

[0523] The data acquisition function node is deployed on the robot's local ROS control layer to collect visual data, audio data, LiDAR data, and robot operating status data, and publishes them to the edge intelligent processing layer via the topic / raw_data.

[0524] (2) Multimodal data processing module

[0525] The multimodal data processing module is deployed in the edge intelligent processing layer. It is used to preprocess the raw multimodal data, extract features and construct a unified data representation, and send the processed data to the modal state detection and quality assessment module and the missing modality adaptive processing module.

[0526] (3) Modal state detection and quality assessment module

[0527] The modality state detection and quality assessment module is used to detect and assess the state of visual modality, audio modality and posture modality, and send the modality state information to the missing modality adaptive processing module through the topic / modality_status.

[0528] (4) Missing mode adaptive processing module

[0529] The missing modality adaptive processing module is used to perform modal compensation, adaptive detection and path adjustment decisions based on modal state information, and send the optimized feature data to the multimodal abnormal behavior detection module, while sending path adjustment and gimbal control commands to the robot's local ROS control layer.

[0530] (5) Multimodal abnormal behavior detection module

[0531] The multimodal abnormal behavior detection module is used to identify abnormal behaviors based on multimodal feature data, and publishes the abnormal behavior detection results to the system status update and alarm function node through the topic / abnormal_behavior_results.

[0532] (6) ROS control and execution function nodes

[0533] The ROS control and execution function node is deployed in the robot's local ROS control layer. It is used to receive path control and gimbal control commands sent by the edge intelligent processing layer, drive the robot chassis, gimbal and alarm devices to complete the corresponding actions, and publish robot operation status information.

[0534] (7) System status update and alarm function node

[0535] The system status update and alarm function node is used to receive abnormal behavior detection results and robot operation status information, and to complete abnormal alarms, status updates, data recording and AR navigation information display.

[0536] Through the aforementioned ROS-based edge-to-edge collaborative interaction mechanism, this invention achieves collaborative linkage between multimodal data acquisition, modality missing adaptive processing, abnormal behavior detection, and robot control execution, effectively improving the real-time performance, robustness, and intelligence level of abnormal behavior detection in inspection robots under complex scenarios.

[0537] In this embodiment, the inspection robot of the present invention is the front-end execution terminal of the multimodal missing adaptive abnormal behavior detection system, which is used to perform autonomous movement, multi-source data acquisition and front-end preprocessing in the inspection scenario. The robot includes a robot head 1, a communication module 2, a body 3 and a mobile chassis 4 from top to bottom. All components work together under the unified scheduling of the control and computing unit, and realize data interaction and collaborative control with the edge intelligent processing and decision-making center of the peripheral device through the communication module 2.

[0538] The robot head 1 integrates a rotatable gimbal mechanism, equipped with a vision acquisition unit, a microphone array, and an audible and visual alarm. The vision acquisition unit includes an RGB camera, an infrared camera, and a depth camera, which can adjust the observation angle with the gimbal mechanism to achieve multi-view scene data acquisition from different areas and multiple angles. The RGB camera is used to acquire visible light images of the inspection scene, capturing personnel actions and environmental status information. The infrared camera is used to acquire temperature distribution information within the scene, assisting in the identification and judgment of abnormal behavior and equipment status. The depth camera is used to acquire the spatial position and motion trajectory information of the target, providing spatial dimension support for behavior status judgment. The microphone array is deployed on the side of the robot head 1 to acquire environmental audio signals, capturing acoustic features such as abnormal sounds and personnel voices, and participating in multimodal detection as an auxiliary modality. The audible and visual alarm is integrated at the top of the robot head 1, used to provide on-site safety warnings through flashing lights and voice prompts when receiving abnormal alarm commands.

[0539] The communication module 2 is located on the side of the main body 3 and uses wireless communication. On the one hand, it realizes data interaction between the internal hardware units of the robot and the control computing unit. On the other hand, it undertakes the function of bidirectional transmission between the end and the edge. It uploads the multimodal feature data collected and preprocessed by the front end to the edge intelligent processing and decision-making center of the peripheral device, and also receives inspection instructions, path adjustment instructions and alarm control instructions issued by the edge side, so as to ensure real-time information synchronization and collaborative control between the front-end robot and the back-end detection system.

[0540] The main body 3 serves as the robot's main load-bearing structure, housing a built-in control and computing unit. This unit is electrically connected to the vision acquisition unit, microphone array, audible and visual alarm, and mobile chassis 4, undertaking front-end data preprocessing and overall machine operation scheduling functions. Specifically, the control and computing unit cleans, spatiotemporally aligns, and performs preliminary feature extraction on the raw multimodal data collected by each sensing unit, generating standardized preprocessed multimodal features. These features are then uploaded to the peripheral edge intelligent processing and decision-making center via communication module 2. The multimodal missing adaptive abnormal behavior detection system deployed at this center performs modal quality assessment, missing adaptive compensation, and abnormal behavior recognition calculations. Simultaneously, the control and computing unit receives control commands from the edge side, synchronously driving gimbal angle adjustment, alarm triggering, and chassis movement to ensure the orderly execution of inspection tasks.

[0541] The mobile chassis 4 is located at the bottom of the robot, with omnidirectional wheels installed on the bottom. It integrates a LiDAR and motor status detection component. The omnidirectional wheels, in conjunction with the motor drive mechanism, enable the robot to move in all directions, turn, and follow paths, supporting the robot to autonomously complete full-scene inspection tasks along a planned path. The LiDAR is used to scan and acquire spatial structure information of the inspection environment, supporting the robot in environmental map construction, autonomous localization, navigation and obstacle avoidance, and dynamic path adjustment. The motor status detection component is used to collect parameters such as motor speed and operating current in real time, which are used as part of the robot's status data in multimodal data acquisition. During operation, the mobile chassis 4 receives motion control commands from the control and computing unit and dynamically adjusts its movement status based on the real-time environmental perception data from the LiDAR. When a decrease in modal data quality or a lack of data is detected, it can respond to a supplementary acquisition command and move to the target location to complete supplementary data acquisition for the corresponding area.

[0542] Through the above hardware structure design, the inspection robot realizes multi-source perception data acquisition, front-end standardized preprocessing and edge collaborative interaction, which can adapt to the multimodal data acquisition needs in complex inspection environments and provide stable and reliable front-end execution terminal support for multimodal missing adaptive abnormal behavior detection systems.

[0543] In this embodiment of the invention, an inspection robot is provided, comprising a mobile chassis, a main body, and a robot head connected in sequence. The robot head is equipped with a vision acquisition unit, a microphone array, and an audible and visual alarm. The vision acquisition unit includes an RGB camera, an infrared camera, and a depth camera. The bottom of the mobile chassis is equipped with omnidirectional wheels, and the mobile chassis has a built-in lidar and motor status detection component. A communication module is located on the side of the main body, and the main body has a built-in control and computing unit. The control and computing unit is electrically connected to the vision acquisition unit, the microphone array, the audible and visual alarm, and the mobile chassis. The control and computing unit is equipped with a multimodal missing adaptive abnormal behavior detection system. Based on the above scheme, the inspection robot completes multimodal data acquisition through multiple types of sensing hardware. Relying on the multimodal missing adaptive abnormal behavior detection system on the control and computing unit, it evaluates the quality of each modality of data in real time and adaptively adjusts the detection strategy during autonomous inspection. It can stably complete abnormal behavior detection in complex inspection environments such as changes in lighting, partial occlusion, and signal interference, thereby improving the robustness of abnormal behavior detection.

[0544] This invention also provides a computer device, including a memory and a processor, wherein the memory stores a computer program; when the computer program is executed by the processor, the processor performs the steps of the multimodal missing adaptive abnormal behavior detection method as described in the above embodiments.

[0545] This invention also provides a computer-readable storage medium storing a computer program / instructions thereon, which, when executed by a processor, implements the steps of the multimodal missing adaptive abnormal behavior detection method as described in the above embodiments.

[0546] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal missing adaptive abnormal behavior detection system based on an inspection robot, characterized in that, The system includes a multimodal data acquisition module, a data processing module, a modal state detection module, a missing modality adaptive processing module, and a multimodal abnormal behavior detection module, which are connected in sequence via communication. The multimodal data acquisition module is used to acquire multimodal data; The data processing module is used to preprocess the multimodal data to obtain preprocessed multimodal features; The modal state detection module is used to perform multimodal data quality assessment on the preprocessed multimodal features and generate modal state information; The missing modality adaptive processing module is used to determine the current modality missing type based on the modality state information, and selectively perform feature compensation or invalid modality removal on the preprocessed multimodal features based on the current modality missing type to obtain multimodal features adapted to the current perception state. The multimodal abnormal behavior detection module is used to dynamically configure the multimodal feature fusion weights based on the modal state information, calculate the behavior confidence score by combining the multimodal features adapted to the current perception state, compare the behavior confidence score with the preset abnormal judgment threshold, and output the abnormal behavior detection result.

2. The multimodal missing adaptive abnormal behavior detection system based on an inspection robot according to claim 1, characterized in that, The multimodal data includes visual data, audio data, and robot state data; the multimodal data acquisition module includes a visual acquisition unit, an audio acquisition unit, and a robot state acquisition unit. The visual acquisition unit, audio acquisition unit, and robot status acquisition unit are all connected to the data processing module. The visual acquisition unit is used to acquire the visual data and transmit it to the data processing module; The audio acquisition unit is used to acquire audio data and transmit it to the data processing module; The robot status acquisition unit is used to acquire the robot status data and transmit it to the data processing module.

3. The multimodal missing adaptive abnormal behavior detection system based on an inspection robot according to claim 1, characterized in that, The data processing module is specifically used for: The multimodal data is cleaned to obtain cleaned multimodal data. The cleaned multimodal data is time-aligned to obtain time-aligned multimodal data. Spatially align the time-aligned multimodal data to obtain spatially aligned multimodal data; Feature extraction is performed on the spatially aligned multimodal data to obtain preprocessed multimodal features.

4. The multimodal missing adaptive abnormal behavior detection system based on an inspection robot according to claim 2, characterized in that, The preprocessed multimodal features include visual features, audio features, and human posture features; the current modality missing type includes local visual modality missing, audio-assisted modality missing, human posture-assisted modality missing, and normal modality; the missing modality adaptive processing module is specifically used for: Read the modal quality score from the modal state information. The modal quality score includes visual modal quality score, audio modal quality score, and human posture modal quality score. The visual modal quality score includes overall visual quality score and visual sub-regional quality score. The overall visual quality score is compared with a preset main modality failure threshold. If the overall visual quality score is greater than or equal to the preset main modality failure threshold, the visual sub-region quality score is compared with a preset local missing threshold one by one. If at least one visual sub-region quality score is less than the preset local missing threshold, it is determined to be a local visual modality missing. The multi-view visual features in the preprocessed multimodal features are used to complement the visual features of the corresponding missing region in the multimodal features, and audio features are fused for auxiliary compensation to obtain multimodal features adapted to the current perception state. If the visual sub-region quality scores are all greater than or equal to the preset local missing threshold, the audio modality quality score is compared with the preset auxiliary modality effective threshold; if the audio modality quality score is less than the preset auxiliary modality effective threshold, it is determined that the audio auxiliary modality is missing, and the audio features are removed from the preprocessed multimodal features, while the visual features and human posture features are retained to obtain multimodal features that are adapted to the current perception state. If the audio modal quality score is greater than or equal to the preset auxiliary modal effective threshold, the human posture modal quality score is further compared with the preset posture effective threshold; if the human posture modal quality score is less than the preset posture effective threshold, it is determined that the human posture auxiliary modality is missing, and the human posture features are removed from the preprocessed multimodal features, while the visual and audio features are retained to obtain multimodal features that are adapted to the current perception state. If the human posture modal quality score is greater than or equal to the preset effective posture threshold, it is determined that the modality is normal, and the preprocessed multimodal features are used as multimodal features to adapt to the current perception state.

5. The multimodal missing adaptive abnormal behavior detection system based on an inspection robot according to claim 1, characterized in that, The current modality loss type also includes complete loss of the main visual focus; the missing modality adaptive processing module is further used for: If the overall visual quality score is less than the preset master mode failure threshold, it is determined that the master vision is completely missing, abnormal behavior detection is stopped, and a device abnormality alarm message is output.

6. The multimodal missing adaptive abnormal behavior detection system based on an inspection robot according to claim 5, characterized in that, The system also includes an inspection path planning and dynamic adjustment module; The inspection path planning and dynamic adjustment module is communicatively connected to the modal state detection module and the missing modality adaptive processing module, respectively. The inspection path planning and dynamic adjustment module is used to calculate the modal quality cost based on the modal quality score in the modal state information when receiving equipment abnormality alarm information. It also combines the path distance parameters and environmental traversal complexity parameters stored in the preset inspection environment map, substitutes them into the cost function of the preset improved A* algorithm to complete the path optimization, generates the optimized inspection path and supplementary acquisition control instructions, and sends the supplementary acquisition control instructions to the motion execution end of the inspection robot to drive the inspection robot to perform supplementary data acquisition.

7. A multimodal missing adaptive abnormal behavior detection method based on an inspection robot, characterized in that, include: Collect multimodal data and preprocess the multimodal data to obtain preprocessed multimodal features; Based on the preprocessed multimodal features, perform quality assessment of each modality and generate modal state information; The current modality missing type is determined based on the modality state information, and feature compensation or invalid modality removal is selectively performed on the preprocessed multimodal features based on the current modality missing type to obtain multimodal features adapted to the current perception state. Based on the modal state information, the multimodal feature fusion weights are dynamically configured, and the behavior confidence score is calculated by combining the multimodal features adapted to the current perception state. After comparing the behavior confidence score with the preset anomaly judgment threshold, the abnormal behavior detection result is output.

8. An inspection robot, characterized in that, Including a mobile chassis, main body, and robot head connected in sequence; The robot's head is equipped with a vision acquisition unit, a microphone array, and an audible and visual alarm. The vision acquisition unit includes an RGB camera, an infrared camera, and a depth camera. The bottom of the mobile chassis is equipped with omnidirectional wheels, and the mobile chassis has a built-in lidar and motor status detection component; A communication module is provided on the side of the main body of the fuselage, and a control computing unit is built into the main body of the fuselage; The control computing unit is electrically connected to the vision acquisition unit, the microphone array, the audible and visual alarm, and the mobile chassis, respectively. The control computing unit is used to receive multimodal data and preprocess it, and then upload the preprocessed multimodal features to the edge intelligent processing and decision-making center of the peripheral device through the communication module. The peripheral edge intelligent processing and decision-making center is equipped with a multimodal missing adaptive abnormal behavior detection system based on an inspection robot as described in any one of claims 1-7.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the multimodal missing adaptive abnormal behavior detection method based on inspection robot as described in claim 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the multimodal missing adaptive abnormal behavior detection method based on inspection robot as described in claim 7.

Images