Robot inspection method, device, equipment and storage medium

Abstract

The application provides a robot inspection method, device, equipment and storage medium. The method comprises: acquiring multi-modal data of a scene to be inspected, including visual data and positioning data; performing event identification processing on the multi-modal data according to an end-side multi-modal large model configured for the robot to obtain an event identification result; the event identification processing refers to a processing process of performing abnormal feature detection on the visual data, performing position correlation verification on the positioning data, combining a preset inspection event rule to identify an event in the scene to be inspected, and obtaining the event identification result; the event identification result includes an event identifier, an event type and an event-associated multi-modal data index; generating a structured event report according to the event identification result through the end-side multi-modal large model; wherein the structured event report includes an event time, positioning data corresponding to the event, visual data corresponding to the event, an event type, an event record and text data corresponding to the event. The real-time inspection is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a robot inspection method, apparatus, equipment and storage medium. Background Technology

[0002] Robotic inspection technology is widely used in industrial, security, and municipal inspection scenarios that require high real-time performance, high precision, and adaptability to complex environments, in order to improve the real-time performance, accuracy, and scenario adaptability of inspections.

[0003] In existing technologies, robot inspection modes primarily rely on centralized cloud processing. This involves the robot terminal only collecting data, uploading the raw data to a cloud server via the network, where a large-scale model identifies and analyzes the data before transmitting the inspection results back. However, this mode is constrained by network bandwidth, transmission stability, and cloud queuing times. Data transmission and cloud processing result in end-to-end delays of several seconds to tens of seconds, failing to meet the real-time response requirements for emergency events.

[0004] Therefore, existing technologies suffer from excessively long response times to events, failing to meet the real-time requirements of emergency scenarios. Summary of the Invention

[0005] This application provides a robot inspection method, apparatus, equipment, and storage medium to improve the real-time performance of inspections.

[0006] In a first aspect, embodiments of this application provide a robot inspection method, comprising:

[0007] Acquire multimodal data of the scene to be inspected; the multimodal data includes visual data and positioning data.

[0008] Based on the multimodal large model configured for the robot, event recognition processing is performed on the multimodal data to obtain event recognition results. The event recognition processing refers to the process of detecting abnormal features in visual data, verifying the positional correlation of positioning data, and identifying events in the scene to be inspected by combining preset inspection event rules, thereby obtaining event recognition results. The event recognition results include event identifiers, event types, and multimodal data indexes associated with the events.

[0009] Using a multimodal large model on the edge, a structured event report is generated based on the event recognition results. The structured event report includes the event time, the corresponding location data, the corresponding visual data, the event type, the event record, and the corresponding text data.

[0010] In one possible implementation, the end-side multimodal large model includes depthwise separable convolutional layers and lightweight convolutional layers;

[0011] Accordingly, based on the end-side multimodal large model configured for the robot, event recognition processing is performed on the multimodal data to obtain event recognition results, including:

[0012] Based on the lightweight convolutional layer, preliminary feature extraction is performed on the multimodal data to obtain preliminary multimodal features;

[0013] Based on the depthwise separable convolutional layer, depth features are extracted from the preliminary multimodal features to obtain depth features; among them, depth features include scene features and abnormal target features corresponding to visual data, as well as location coordinate features and location association features corresponding to localization data;

[0014] Based on the preset inspection event rules, the deep features are classified and identified to obtain the event identification results.

[0015] In one possible implementation, before performing event recognition processing on the multimodal data based on the end-side multimodal large model configured for the robot to obtain the event recognition result, the method further includes:

[0016] Obtain the training dataset; the training dataset includes visual data, location data, and text data;

[0017] Based on the cross-modal training mechanism, visual data, localization data, and text data are fused to obtain fused multimodal data;

[0018] The edge-side multimodal large model is trained based on the fused multimodal data.

[0019] In one possible implementation, acquiring multimodal data of the scene to be inspected includes:

[0020] Acquire the scene terrain data, facility distribution data, and control area data of the scene to be inspected;

[0021] Based on the scene terrain data, facility distribution data, and control area data, a 3D map of the scene to be inspected is generated;

[0022] Obtain inspection tasks;

[0023] Based on the inspection task and the 3D map of the scene to be inspected, a multi-objective optimization algorithm is used to generate the inspection route.

[0024] Inspections are conducted according to the inspection route to obtain multimodal data of the scene to be inspected.

[0025] In one possible implementation, after generating the inspection route using a multi-objective optimization algorithm based on the inspection task and a 3D map of the scene to be inspected, the method further includes:

[0026] Real-time multimodal data is acquired based on the sensor cluster;

[0027] Based on the extended Kalman filter algorithm, real-time multimodal data is calibrated to obtain calibrated real-time multimodal data;

[0028] A local raster map is generated based on the calibrated real-time multimodal data;

[0029] Obstacle identification is performed based on the local grid map to obtain the obstacle identification results;

[0030] The inspection route is dynamically adjusted based on the obstacle identification results.

[0031] In one possible implementation, after generating a structured event report based on the event identification results, the method further includes:

[0032] Based on the structured event report, identify the risk area corresponding to the event;

[0033] Obtain verification data for the risk area;

[0034] Based on the verification data, determine the risk level corresponding to the event;

[0035] Based on the risk level, execute the corresponding early warning action.

[0036] In one possible implementation, after determining the risk level corresponding to the event, the process includes:

[0037] Generate risk warning messages based on risk areas and risk levels;

[0038] Based on the preset priority queue scheduling mechanism, the risk warning messages are pushed and processed using the message queue telemetry transmission protocol.

[0039] In one possible implementation, after generating a structured event report based on the event identification results, the method further includes:

[0040] Based on the structured event report, a visual output is generated, which includes an inspection trajectory map, an event distribution histogram, and robot status data.

[0041] Secondly, embodiments of this application provide a robot inspection device, comprising:

[0042] The acquisition module is used to acquire multimodal data of the scene to be inspected; the multimodal data includes visual data and positioning data.

[0043] The recognition module is used to perform event recognition processing on multimodal data based on the end-side multimodal large model configured for the robot, and obtain event recognition results. The event recognition processing refers to the process of detecting abnormal features in visual data, verifying the position correlation of positioning data, and identifying events in the scene to be inspected by combining preset inspection event rules, and obtaining event recognition results. The event recognition results include event identifiers, event types, and multimodal data indexes associated with the events.

[0044] The generation module is used to generate structured event reports based on the event recognition results using a multimodal large model on the edge. The structured event report includes the event time, the corresponding location data, the corresponding visual data, the event type, the event record, and the corresponding text data.

[0045] In one possible implementation, the end-side multimodal large model includes depthwise separable convolutional layers and lightweight convolutional layers;

[0046] Accordingly, the recognition module can also be used for:

[0047] Based on the lightweight convolutional layer, preliminary feature extraction is performed on the multimodal data to obtain preliminary multimodal features;

[0048] Based on the depthwise separable convolutional layer, depth features are extracted from the preliminary multimodal features to obtain depth features; among them, depth features include scene features and abnormal target features corresponding to visual data, as well as location coordinate features and location association features corresponding to localization data;

[0049] Based on the preset inspection event rules, the deep features are classified and identified to obtain the event identification results.

[0050] In one possible implementation, before performing event recognition processing on the multimodal data based on the end-side multimodal large model configured for the robot to obtain the event recognition result, the robot's inspection device further includes a training module, which can specifically be used for:

[0051] Obtain the training dataset; the training dataset includes visual data, location data, and text data;

[0052] Based on the cross-modal training mechanism, visual data, localization data, and text data are fused to obtain fused multimodal data;

[0053] The edge-side multimodal large model is trained based on the fused multimodal data.

[0054] In one possible implementation, the acquisition module can also be used for:

[0055] Acquire the scene terrain data, facility distribution data, and control area data of the scene to be inspected;

[0056] Based on the scene terrain data, facility distribution data, and control area data, a 3D map of the scene to be inspected is generated;

[0057] Obtain inspection tasks;

[0058] Based on the inspection task and the 3D map of the scene to be inspected, a multi-objective optimization algorithm is used to generate the inspection route.

[0059] Inspections are conducted according to the inspection route to obtain multimodal data of the scene to be inspected.

[0060] In one possible implementation, after generating the inspection route using a multi-objective optimization algorithm based on the inspection task and a 3D map of the scene to be inspected, the robot's inspection device further includes an adjustment module, which can specifically be used for:

[0061] Real-time multimodal data is acquired based on the sensor cluster;

[0062] Based on the extended Kalman filter algorithm, real-time multimodal data is calibrated to obtain calibrated real-time multimodal data;

[0063] A local raster map is generated based on the calibrated real-time multimodal data;

[0064] Obstacle identification is performed based on the local grid map to obtain the obstacle identification results;

[0065] The inspection route is dynamically adjusted based on the obstacle identification results.

[0066] In one possible implementation, after generating a structured event report based on the event identification results, the robot's inspection device further includes an early warning module, which can specifically be used for:

[0067] Based on the structured event report, identify the risk area corresponding to the event;

[0068] Obtain verification data for the risk area;

[0069] Based on the verification data, determine the risk level corresponding to the event;

[0070] Based on the risk level, execute the corresponding early warning action.

[0071] In one possible implementation, after determining the risk level corresponding to the event, the robot's inspection device includes a push module, which can specifically be used for:

[0072] Generate risk warning messages based on risk areas and risk levels;

[0073] Based on the preset priority queue scheduling mechanism, the risk warning messages are pushed and processed using the message queue telemetry transmission protocol.

[0074] In one possible implementation, after generating a structured event report based on the event recognition results, the robot's inspection device further includes a display module, which can specifically be used for:

[0075] Based on the structured event report, a visual output is generated, which includes an inspection trajectory map, an event distribution histogram, and robot status data.

[0076] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0077] The memory stores instructions that the computer executes;

[0078] The processor executes computer execution instructions stored in memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0079] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0080] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0081] This application provides a robot inspection method, apparatus, device, and storage medium. It acquires multimodal data of the scene to be inspected, including visual data and positioning data. Based on a multimodal large-scale model configured for the robot, event recognition processing is performed on the multimodal data to obtain event recognition results. This event recognition processing involves detecting abnormal features in the visual data, verifying the positional correlation of the positioning data, and combining preset inspection event rules to identify events in the scene to be inspected, thus obtaining event recognition results. The event recognition results include event identifiers, event types, and multimodal data indexes associated with the events. Based on the event recognition results, a structured event report is generated using the multimodal large-scale model. The structured event report includes event time, corresponding positioning data, corresponding visual data, event type, event record, and corresponding text data. Compared to existing technologies, this application's method deploys the multimodal large-scale model on the robot's terminal side, allowing event recognition and decision-making to be completely decentralized to the robot terminal, eliminating the time overhead of cloud transmission and queuing, thereby improving the real-time performance of the inspection. Attached Figure Description

[0082] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0083] Figure 1 A schematic diagram of a robot inspection system architecture provided in this application;

[0084] Figure 2 Flowchart of the robot inspection method provided in this application Figure 1 ;

[0085] Figure 3 Flowchart of the robot inspection method provided in this application Figure 2 ;

[0086] Figure 4 A schematic diagram of the inspection device for the robot provided in this application;

[0087] Figure 5 A schematic diagram of the structure of the electronic device provided in this application.

[0088] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0089] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0090] It should be noted that all data involved in this application are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0091] In industrial, security, and municipal inspection scenarios, there are extremely high requirements for real-time inspection, scenario generalization ability, multi-task processing ability, and edge autonomous decision-making ability.

[0092] In existing technologies, inspection is usually carried out using robot inspection technology based on cloud-based multimodal large models or technology based on robot terminal data collection.

[0093] Among them, the robot inspection technology based on cloud-based multimodal large model usually transmits the raw data such as pictures and videos collected by the robot to the cloud data center through the network, where the large model in the cloud performs recognition and analysis, and then sends the results back.

[0094] However, the continuous transmission of massive amounts of video data and the concentrated consumption of cloud computing power result in huge network bandwidth and cloud computing costs. Meanwhile, the cloud requires professional personnel to maintain and adjust models and early warning rules, leading to slow response times and high reliance on human intervention. Furthermore, the transmission process is constrained by network bandwidth, transmission stability, and cloud queuing times, resulting in end-to-end latency of several seconds or even tens of seconds. For emergencies discovered during inspections, immediate on-site alarms and rapid responses are impossible, leading to high inspection costs and insufficient real-time performance.

[0095] In existing technologies based on robot terminal data collection, inspection robots are only equipped with lightweight traditional computer vision algorithms, but lack the ability to run large and complex models on terminal devices. They are easily affected by factors such as light, weather, and occlusion, resulting in high false alarm and false negative rates. They cannot cope with complex and ever-changing scene requirements, leading to insufficient recognition accuracy and generalization ability of lightweight models.

[0096] Furthermore, existing inspection robots can only patrol along preset fixed routes and cannot dynamically adjust their routes according to actual on-site conditions. Moreover, existing inspection robots are only responsible for collecting and transmitting data; subsequent event trajectory reconstruction, analysis, and decision-making rely on manual processing in the cloud, lacking edge-side intelligent analysis capabilities, which further exacerbates inspection delays and cost issues.

[0097] To address the aforementioned issues, the core concept of this application is to deploy a multimodal large model on the robot terminal side, thereby enabling event recognition and decision-making to be fully decentralized to the robot terminal, completely eliminating the time overhead of cloud transmission and queuing, and improving the real-time performance of inspections.

[0098] Optionally, Figure 1 This is a schematic diagram of a robot inspection system architecture provided in this application. Figure 1 As shown, the robot's inspection system architecture includes at least one of a data acquisition device 101, a processing device 102, and a display device 103.

[0099] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the above architecture. In other feasible embodiments of this application, the above architecture may include more or fewer components than illustrated, or combine some components, or split some components, or arrange different components, which can be determined according to the actual application scenario and is not limited here. Figure 1 The components shown can be implemented in hardware, software, or a combination of both.

[0100] In the specific implementation process, the data acquisition device 101 may include an input / output interface or a communication interface. The data acquisition device 101 can be connected to the processing device through the input / output interface or the communication interface. The data acquisition device 101 is used to acquire multimodal data of the scene to be inspected. The multimodal data includes visual data and positioning data.

[0101] The processing device 102 can be used to perform event recognition processing on multimodal data based on the edge-side multimodal large model configured for the robot, and obtain event recognition results. Event recognition processing refers to the process of detecting abnormal features in visual data, verifying the positional correlation of positioning data, and combining preset inspection event rules to identify events in the scene to be inspected, thereby obtaining event recognition results. The event recognition results include event identifiers, event types, and multimodal data indexes associated with the events. Based on the event recognition results, the edge-side multimodal large model generates a structured event report. The structured event report includes event time, corresponding positioning data, corresponding visual data, event type, event record, and corresponding text data.

[0102] The display device 103 can also be a touch screen or the screen of a terminal device, used to receive user commands while displaying the above-mentioned content, so as to realize interaction with the user.

[0103] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0104] Figure 2 Flowchart of the robot inspection method provided in this application Figure 1 ,like Figure 2 As shown, the method includes:

[0105] S201. Obtain multimodal data of the scene to be inspected; where multimodal data includes visual data and positioning data.

[0106] In this embodiment, the robot uses a cluster of sensors to collect multimodal data of the scene to be inspected in a comprehensive and real-time manner. The data collection process relies on the multi-source fusion positioning technology and perception fusion capability of the robot's autonomous navigation patrol system to ensure the accuracy and integrity of the data.

[0107] S202. Based on the multimodal large model configured for the robot, perform event recognition processing on the multimodal data to obtain event recognition results. Event recognition processing refers to the process of detecting abnormal features in visual data, verifying the position correlation of positioning data, and identifying events in the scene to be inspected by combining preset inspection event rules to obtain event recognition results. Event recognition results include event identifiers, event types, and multimodal data indexes associated with events.

[0108] In this embodiment, based on the edge-side multimodal large model configured for the robot, event recognition is performed on the multimodal data, visual data features are extracted and anomalies are detected, the spatial location correlation of the positioning data is verified, scene event matching is completed by combining preset inspection event rules, and false alarms are filtered through spatiotemporal correlation reasoning. Finally, an event recognition result including event identifier, event type, and multimodal data index is generated. The entire event recognition process is implemented in real time on the edge side.

[0109] S203. Using the edge-side multimodal large model, generate a structured event report based on the event recognition results; wherein, the structured event report includes event time, corresponding location data, corresponding visual data, event type, event record, and corresponding text data.

[0110] In this embodiment, relying on the natural language processing and multimodal data fusion capabilities of the edge multimodal large model, combined with the structured data processing technology of the fully automatic report generation system, the generation of structured event reports is completed directly on the robot edge without cloud intervention.

[0111] The robot inspection method provided in this application acquires multimodal data of the scene to be inspected; performs event recognition processing on the multimodal data according to the end-side multimodal large model configured for the robot to obtain event recognition results; generates a structured event report based on the event recognition results through the end-side multimodal large model; and deploys the multimodal large model on the robot terminal side so that event recognition and decision-making are completely decentralized to the robot terminal, completely eliminating the time overhead of cloud transmission and queuing, thereby improving the real-time performance of inspection.

[0112] Figure 3 Flowchart of the robot inspection method provided in this application Figure 2 ,like Figure 3 As shown, in this embodiment... Figure 2 Based on the embodiments, the inspection method of the robot is described in detail, which includes:

[0113] S301. Obtain the scene terrain data, facility distribution data, and control area data of the scene to be inspected.

[0114] In this embodiment, a drone equipped with a dynamic differential global positioning system is used to perform oblique photography at a scale of 1:1000 to obtain scene terrain data, facility distribution data, and control area data of the scene to be inspected.

[0115] S302. Generate a 3D map of the scene to be inspected based on the scene terrain data, facility distribution data, and control area data.

[0116] In this embodiment, by combining ground robot synchronous localization and mapping technology, a 3D map of the scene to be inspected, including semantic annotations, is generated based on scene terrain data, facility distribution data, and control area data, so that the accuracy of the 3D map of the scene to be inspected is controlled within ±5 cm; wherein, the semantic annotations include roads, fences, and control area boundaries.

[0117] For example, the scene terrain data includes slope; if the slope is greater than 30°, the corresponding area will be marked as a high-risk communication zone; the facility distribution data includes border fences and camera locations; and the controlled area data includes detention center cordon lines and cultural relic protection zone boundaries.

[0118] In some embodiments, an enhanced adaptive storage technology format is used to store the 3D map of the scene to be inspected, and dynamic loading of the 3D map is supported to ensure that the loading time of the 3D map of a single area does not exceed 2 seconds.

[0119] In some embodiments, a built-in safe operation design domain boundary is included based on the boundary of the controlled area. For example, a red zone is set 5 m away from the boundary of the controlled area to delineate the scope of the robot's inspection. The safe operation design domain boundary includes three preset operation design domains: the blue zone allows full-function operation (e.g., full-speed movement and robotic arm operation); the yellow zone limits the speed to 0.8 m / s and prohibits physical contact actions; the red zone triggers emergency braking and only allows the camera to scan around.

[0120] S303, Obtain inspection tasks.

[0121] In this embodiment, the robot's inspection tasks include safety monitoring, equipment status inspection, and behavior analysis, thus expanding the application scenarios.

[0122] For example, inspection tasks include at least one of the following:

[0123] Intrusion monitoring and inspection are conducted on perimeter fences and restricted area boundaries to identify illegal intrusions in real time; fire hazard investigation and inspection are carried out in key areas within the inspection area to promptly detect fires and abnormal smoke; instrument readings and equipment status of damaged equipment are inspected in power distribution rooms and equipment rooms; and behavioral analysis and inspection are conducted in areas where personnel are active to identify abnormal behavior.

[0124] S304. Based on the inspection task and the 3D map of the scene to be inspected, a multi-objective optimization algorithm is used to generate the inspection route.

[0125] In this embodiment, the multi-objective optimization algorithm is a hybrid multi-objective optimization algorithm of Dijkstra-PSO (Dijkstra-Particle Swarm Optimization) improved from the A* algorithm (a heuristic path search and graph traversal algorithm).

[0126] Based on the inspection task, predefined inspection parameters are defined. These predefined parameters include the coverage rate of key areas, the duration of a single inspection, the inspection speed, and the inspection frequency of key areas. For example, the coverage rate of key areas is greater than or equal to 95%, the duration of a single inspection is less than or equal to 120 minutes, the inspection speed is 0.5 m / s to 2 m / s, and the inspection frequency of key areas is once per hour.

[0127] By combining predefined inspection parameters and a 3D map of the scene to be inspected, a multi-objective optimization algorithm is used to generate an inspection route; the inspection route is a Bézier curve trajectory carrying a velocity gradient.

[0128] In some embodiments, the generated inspection route can be adjusted by dragging and dropping nodes, setting speed gradients, and importing contingency plans for emergencies through a visual route editing tool in the human-computer collaborative interaction interface, so that the generated inspection route is more in line with the actual scenario.

[0129] For example, speed gradient settings include automatically limiting speed to 0.5 m / s on muddy roads, and emergency scenario contingency plans include automatically switching to preset backup inspection routes during rainstorms, and automatically generating off-peak inspection routes based on tidal flow data in border defense scenarios; where tidal flow data includes peak periods for port clearance.

[0130] S305. Conduct inspections according to the inspection route to obtain multimodal data of the scene to be inspected.

[0131] In this embodiment, during the inspection process of the robot according to the inspection route, the driving strategy is dynamically adjusted based on the model predictive control algorithm and the scene terrain data in the 3D map.

[0132] For example, the drive strategy includes a 40% improvement in the climbing stability of the quadruped robot on a 30° slope and a 65% reduction in the slippage frequency of the wheeled robot on gravel roads. It supports three modes: ±180° in-situ turning, 0.2 m / s low-speed precision movement, and 2 m / s high-speed cruising, to meet the speed control needs of different scenarios.

[0133] In some embodiments, the process of the robot performing inspections according to the inspection route includes:

[0134] Real-time multimodal data is acquired based on the sensor cluster.

[0135] In this embodiment, the robot uses a sensor cluster to collect multimodal data of the scene to be inspected from all directions, providing complete and accurate real-time multimodal data for subsequent edge-side multimodal large-scale models. The sensor cluster includes a vision system, radar, environmental sensors, and a positioning system. The environmental sensors include temperature and humidity sensors, barometers, and gas sensors. The positioning system integrates the BeiDou satellite navigation system, ultra-wideband positioning, and inertial navigation system. The real-time multimodal data includes images, videos, positioning coordinates, temperature and humidity, gas concentration, and equipment status images.

[0136] For example, the vision system consists of three global shutter cameras with a 120° wide-angle view, 12MP resolution, and support for automatic white balance and strong light resistance; the three global shutter cameras are installed in the front, left and right directions of the robot's body, respectively, covering a 270° field of view.

[0137] The environmental sensors include a temperature and humidity sensor with an accuracy of ±2% relative humidity, a barometer with an altitude accuracy of ±0.5 m, and a gas sensor that can detect the concentration of carbon monoxide or methane gas.

[0138] The radar includes a 16-line lidar with a 360° scanning range and an accuracy of ±2 cm.

[0139] Based on the extended Kalman filter algorithm, real-time multimodal data is calibrated to obtain calibrated real-time multimodal data.

[0140] In this embodiment, the robot relies on an autonomous navigation patrol system to conduct inspections along the inspection route and achieves centimeter-level real-time calibration through multi-source fusion positioning technology. The multi-source fusion positioning technology includes at least one of the following: integrated ultra-wideband positioning, dual-mode positioning of BeiDou satellite navigation system and Global Positioning System, and inertial navigation system. Real-time calibration is performed using an extended Kalman filter algorithm to eliminate positioning errors.

[0141] For example, every time the robot moves 10 m, it uses a Beidou satellite navigation system with a real-time dynamic differential positioning accuracy of ±5 cm, an integrated ultra-wideband positioning system with an indoor accuracy of ±10 cm, and a 9-axis inertial navigation system with angular velocity noise less than or equal to 0.01° / s to perform fusion positioning to correct trajectory deviations, so as to control the positioning error within 10 cm.

[0142] A local raster map is generated based on the calibrated real-time multimodal data.

[0143] Obstacle identification is performed based on the local grid map to obtain the obstacle identification results.

[0144] In this embodiment, the obstacle recognition result includes the obstacle type, the relative distance between the obstacle and the robot, the spatial dimensions of the obstacle, and the real-time motion status of the obstacle.

[0145] For example, the obstacle can be a dynamic obstacle, and the real-time motion state of the obstacle includes speed and direction; the response delay of the obstacle recognition process is less than 0.2 seconds.

[0146] The inspection route is dynamically adjusted based on the obstacle identification results.

[0147] In this embodiment, based on the model predictive control algorithm and the obstacle recognition results, the influence range of the obstacle is marked so as to dynamically adjust the inspection route and ensure the normal execution of the inspection process.

[0148] The robot inspection method provided in this application achieves real-time perception and autonomous adaptation during the inspection process through multi-dimensional acquisition by a sensor cluster, data calibration, obstacle recognition, and dynamic adjustment of the inspection route. This provides a complete and accurate data source for large-scale model recognition at the edge. Multi-source fusion positioning calibration using an extended Kalman filter algorithm controls the positioning error within 10cm, ensuring the accuracy of spatial correlation of the data. Based on the calibrated real-time multimodal data, a local grid map is generated, enabling multi-dimensional obstacle information recognition within 0.2 seconds, providing a precise basis for inspection route adjustment. Combined with a model predictive control algorithm, the inspection route is dynamically adjusted, accurately marking the obstacle's influence range and planning the optimal detour path, effectively avoiding various static and dynamic obstacles, ensuring continuous and uninterrupted inspection, and improving the real-time performance of robot inspection.

[0149] In some embodiments, before performing event recognition processing on the multimodal data based on an end-side multimodal large model configured for the robot to obtain the event recognition result, the method further includes:

[0150] Obtain the training dataset; the training dataset includes visual data, location data, and text data.

[0151] In this embodiment, for example, in a security patrol scenario, visual data, positioning data, and text data are collected by a low-angle drone and a high-angle drone. The visual data includes 500,000 20-second video clips of real security scenes as a training dataset. Among these, 100,000 video clips are positive samples of abnormal events, and 400,000 are negative samples without abnormalities. Abnormal events include intrusion, damage, and fighting. The positioning data includes the viewing parameters, spatial location, and imaging scale of the drone or drone. The text data includes the event type and event description text. The event types include abnormal events and normal events.

[0152] Based on the cross-modal training mechanism, visual data, localization data, and text data are fused to obtain fused multimodal data.

[0153] In this embodiment, the fusion process includes visual data augmentation and expansion, text data augmentation and alignment, and cross-view compensation. Visual data, localization data, and text data are carefully fused to obtain enhanced visual features, localization features, and text semantic features.

[0154] For example, visual data augmentation includes one or more of the following methods:

[0155] 1. Apply random cropping to each video slice independently; the cropping ratio is 5% to 25%.

[0156] 2. A segmented resampling strategy is adopted to divide the video into N segments and randomly adjust the playback speed between segments; where N is 3 to 8 and the playback speed between segments is 0.5x to 2.0x.

[0157] 3. Randomly discard 15% of video frames.

[0158] 4. Zoom in and out: Zoom in and out of the entire video slice by 50% to 200%.

[0159] 5. Add Gaussian blur and jitter to reduce video quality.

[0160] 6. Color jitter; where the hue change is less than or equal to 0.1 and the saturation is less than or equal to 0.3.

[0161] Text data augmentation and alignment include: using the Tongyi Qianwen large model to expand and rewrite the event type and event description text of relevant abnormal events, and output more than 5 kinds of description text for abnormal events.

[0162] Cross-view compensation includes: adding a view transformation matrix during training for images acquired from the low view of the unmanned dog and the high view of the drone to improve the robustness of cross-view recognition; wherein, the view transformation matrix T∈R^(2×2).

[0163] The enhanced visual features, location features, and text semantic features are encoded and aligned to generate fused multimodal data.

[0164] The edge-side multimodal large model is trained based on the fused multimodal data.

[0165] In this embodiment, 5 million video slices are used to perform multi-task pre-training on the edge multimodal large model; the multi-task includes determining whether there are abnormal events and determining the type of abnormal events, and a total of 2 rounds of training are conducted.

[0166] Using 500,000 aligned data points, the visual encoder was frozen, and only the language model and adapter were fine-tuned. The training task was to describe event details, and the training was conducted in two rounds. The aligned data included 100,000 samples of anomalous events and five types of descriptive text.

[0167] In some embodiments, optical flow information from adjacent frames is used to filter false detection events, reducing the false alarm rate from 5 times / hour to 0.3 times / hour, thereby enabling spatiotemporal correlation reasoning.

[0168] In some embodiments, for small targets in a drone with a pixel area less than or equal to 20×20, shallow feature fusion is added to the feature pyramid to enhance the small target.

[0169] The robot inspection method provided in this application improves the robustness of cross-view recognition through multimodal data fusion processing and edge-side multimodal large model training, effectively reduces the false alarm rate through spatiotemporal correlation reasoning, enhances the small target recognition capability of UAVs through shallow feature fusion, effectively improves the accuracy of abnormal event recognition, adapts to security inspection needs, and ensures efficient and reliable recognition results.

[0170] In some embodiments, the edge-side multimodal large model includes depthwise separable convolutional layers and lightweight convolutional layers.

[0171] In this embodiment, the depth-separable convolutional layer splits the traditional convolutional layer into a depthwise convolutional layer and a pointwise convolutional layer, which greatly reduces the amount of computation; the lightweight convolutional layer generates core features with a small number of convolutions, and then completes the features through linear transformation, which significantly reduces the number of parameters.

[0172] Accordingly, based on the end-side multimodal large model configured for the robot, event recognition processing is performed on the multimodal data to obtain event recognition results, including:

[0173] S306. Based on the lightweight convolutional layer, perform preliminary feature extraction on the multimodal data to obtain preliminary multimodal features.

[0174] In this embodiment, a lightweight convolutional layer is deployed on the robot's edge computing platform to perform preliminary feature extraction on the real-time multimodal data collected by the robot's sensor cluster. The lightweight convolutional layer reduces the amount of computation through feature reuse and inexpensive operations. Under the premise that the robot's edge computing platform can simultaneously perform intelligent recognition and inference on three 1080P video streams and the inference processing frame rate of each video stream is stably maintained at 18 FPS, the shallow features of the multimodal data are extracted to form preliminary multimodal features, thereby reducing the computational power consumption for edge inference and compressing the parameter size to 4.3MB.

[0175] In some embodiments, spatiotemporal information anchoring technology is used to automatically embed triple positioning information into the real-time multimodal data collected by the robot sensor cluster. The triple positioning information includes GPS latitude and longitude coordinates with a positioning accuracy of ±5 m, a unique identifier of the ultra-wideband indoor positioning tag, and a timestamp information with a time accuracy of milliseconds.

[0176] S307. Based on the depth-separable convolutional layer, perform depth feature extraction on the preliminary multimodal features to obtain depth features; among which, the depth features include scene features and abnormal target features corresponding to the visual data, as well as location coordinate features and location association features corresponding to the localization data.

[0177] In this embodiment, based on the preliminary multimodal feature extraction performed by the lightweight convolutional layer, a depth-separable convolutional layer is used to perform depth feature extraction for dimensional decoupling of the preliminary multimodal features, separating channel convolution and spatial convolution to improve computational efficiency and adapt to the limited computing power of the edge.

[0178] For visual data, scene features and abnormal target features are extracted. For example, scene features include fence scenes at border crossings, detention center supervision areas, and substation distribution rooms, including environmental context features such as lighting, terrain, and facility layout. Abnormal target features include the outline and movement of unauthorized intruders, the fuselage features of drones, the appearance features of damaged equipment, and the shape features of smoke and fire.

[0179] For the location data, location coordinate features and location association features are extracted. For example, location coordinate features include the latitude and longitude of the event and the indoor location tag, while location association features include the relative distance between the target and the fence or controlled area, the spatial orientation of the robot and the abnormal target, and the trajectory association features of multi-frame location data.

[0180] After all deep features are fused, a structured feature vector is formed, providing a core basis for subsequent event classification and recognition. This process is completed on the device side without cloud transmission delay.

[0181] S308. Based on the preset inspection event rules, classify and identify the deep features to obtain the event identification results.

[0182] In this embodiment, the preset inspection event rules include event type classification rules and risk level association rules; wherein, event types include safety monitoring, equipment status and behavior analysis; and risk levels include level one risk, level two risk and level three risk.

[0183] The event identification results include event identifier, event type, event location coordinates, confidence level, and risk level.

[0184] The robot inspection method provided in this application reduces the computational complexity on the edge by introducing depth-separable convolutional layers and lightweight convolutional layers for event recognition processing, thereby improving the utilization rate of edge computing power, eliminating the time overhead of cloud transmission and queuing, and improving the real-time performance of inspection.

[0185] In some embodiments, after generating a structured event report based on the event identification results, the method further includes:

[0186] Based on the structured event report, identify the risk area corresponding to the event.

[0187] In this embodiment, based on the event time, corresponding location data, corresponding visual data, event type, event record, and corresponding text data in the structured event report, combined with the 3D map of the scene to be inspected and the built-in safety operation design domain boundary, the location data corresponding to the event is mapped to a geospatial grid using a map matching algorithm. The surrounding geographical features, facility distribution, and preset control area attributes of the grid where the event is located are automatically identified. At the same time, the scope of influence is delineated in conjunction with the event type, thereby determining the risk area corresponding to the event. The risk area is marked as a blue, yellow, or red risk zone according to the three-level operation design domain division standard, and the boundary coordinates, coverage area, and surrounding key facility information of the risk area are clearly defined, providing a spatial basis for subsequent risk verification and early warning.

[0188] Obtain verification data for the risk areas.

[0189] In this embodiment, based on the identified risk area, a multi-angle evidence collection strategy is automatically initiated to collect data from all angles of the risk area to obtain verification data. The verification data includes multi-dimensional collection content, including: 8-channel visual data that moves along a 2m radius circumference centered on the event point and collects a panoramic image every 45°; temperature distribution data of the risk area collected by infrared thermal imaging equipment; ambient sound data of the risk area recorded synchronously by microphone array; 3D point cloud data of the risk area generated by LiDAR scanning; and environmental parameters of the risk area, including temperature, humidity, and gas concentration, collected by environmental sensors. At the same time, historical inspection data and similar event records around the risk area are integrated to form a complete verification dataset, ensuring that the data can comprehensively verify the authenticity, severity, and scope of impact of the event and eliminate false alarm factors.

[0190] Based on the verification data, determine the risk level corresponding to the event.

[0191] In this embodiment, verification data from the risk area is input into the risk assessment model. Feature extraction and comprehensive analysis are performed on the verification data. Combined with preset inspection event risk judgment rules, quantitative scoring is conducted from four dimensions: event type, scope of impact, degree of harm, and occurrence scenario. Based on the scoring results, events are divided into three risk levels. The judgment criteria include: Level 1 (red) risk, such as armed fights, suspected explosives, fire spread, and toxic gas leaks, with a model confidence level greater than or equal to 95%; Level 2 (yellow) risk, such as drones entering restricted areas, facility damage, and illegal fence crossings, with a model confidence level greater than or equal to 90%; and Level 3 (blue) risk, such as loitering, minor road blockages, and minor equipment malfunctions, with a model confidence level greater than or equal to 85%. This risk assessment model has an accuracy rate of 94.3% in judging risk levels and a false alarm rate as low as 0.7 times per thousand hours, ensuring the accuracy of risk level judgment.

[0192] Based on the risk level, execute the corresponding early warning action.

[0193] In some embodiments, after determining the risk level corresponding to the event, the process includes:

[0194] Risk warning messages are generated based on the risk area and risk level.

[0195] Based on the preset priority queue scheduling mechanism, the risk warning messages are pushed and processed using the message queue telemetry transmission protocol.

[0196] In this embodiment, the risk level, the corresponding warning operation, and the corresponding risk warning message include:

[0197] Level 1 (Red) Risk: Triggers an immediate shutdown alarm. The device immediately activates a 120dB siren with an effective warning distance of 500m and turns on a directional high-intensity searchlight with an illumination distance of 300m to deter people on site. The remote terminal immediately pushes an emergency notification containing a real-time video stream through the message queue telemetry transmission protocol. Within 30 seconds, it dials a voice call to the Level 3 duty personnel. The command center platform pops up a window and marks the risk area in red.

[0198] Level 2 (Yellow) Risk: Triggers regional alarms and remote notifications. The device broadcasts a warning on-site via voice broadcast, and the remote terminal sends a message with an event snapshot to the on-duty personnel within 5 seconds. The command center platform updates the event heat map and marks the risk area.

[0199] Level 3 (blue) risk: Only triggers local voice warnings. The device plays a preset warning message (e.g., "You have entered a restricted area, please leave immediately") through the speaker to remind users of abnormal behavior on site, without sending emergency notifications to the remote end.

[0200] In some embodiments, after generating a structured event report based on the event identification results, the method further includes:

[0201] Based on the structured event report, a visual output is generated, including an inspection trajectory map, an event distribution histogram, and robot status data. In this embodiment, the structured event report supports PDF (Portable Document Format) and HTML (Hypertext Markup Language) formats. The inspection trajectory map includes a heat map with velocity; the event distribution histogram includes the event type, the corresponding time period, and the risk area; and the robot status data includes the status of the core computing module, the power supply and cooling system, the sensor cluster operating status, motion control parameters, and equipment health indicators.

[0202] The robot inspection method provided in this application generates a 3D map with semantic annotations, which is then calibrated using an extended Kalman filter to avoid positioning errors. It generates inspection routes adapted to different scenarios through a multi-objective optimization algorithm, and combines model predictive control to achieve obstacle recognition and dynamic adjustment of the inspection route, ensuring continuous inspection. It employs a lightweight edge-side model to reduce computational consumption and eliminate cloud transmission latency. It integrates multimodal data to improve recognition robustness, significantly reduce false alarm rates, and enhance small target recognition capabilities. It achieves accurate risk classification and efficient early warning response. It generates visual reports, adapting to multiple inspection scenarios, comprehensively improving the real-time performance, accuracy, and reliability of inspections.

[0203] Figure 4 The schematic diagram of the inspection device for the robot provided in this application is as follows: Figure 4 As shown, the robot inspection device provided in this embodiment includes:

[0204] The acquisition module 401 is used to acquire multimodal data of the scene to be inspected; the multimodal data includes visual data and positioning data.

[0205] The recognition module 402 is used to perform event recognition processing on multimodal data according to the end-side multimodal large model configured for the robot, and obtain event recognition results. The event recognition processing refers to the process of detecting abnormal features in visual data, verifying the position correlation of positioning data, and combining preset inspection event rules to identify events in the scene to be inspected and obtain event recognition results. The event recognition results include event identifier, event type, and multimodal data index of event association.

[0206] The generation module 403 is used to generate a structured event report based on the event recognition results using the edge multimodal large model. The structured event report includes the event time, the corresponding location data, the corresponding visual data, the event type, the event record, and the corresponding text data.

[0207] In one possible implementation, the edge-side multimodal large model includes depthwise separable convolutional layers and lightweight convolutional layers; correspondingly, the recognition module 402 can also be specifically used for:

[0208] Based on the lightweight convolutional layer, preliminary feature extraction is performed on the multimodal data to obtain preliminary multimodal features;

[0209] Based on the depthwise separable convolutional layer, depth features are extracted from the preliminary multimodal features to obtain depth features; among them, depth features include scene features and abnormal target features corresponding to visual data, as well as location coordinate features and location association features corresponding to localization data;

[0210] Based on the preset inspection event rules, the deep features are classified and identified to obtain the event identification results.

[0211] In one possible implementation, before performing event recognition processing on the multimodal data based on the end-side multimodal large model configured for the robot to obtain the event recognition result, the robot's inspection device also includes a training module, which can specifically be used for:

[0212] Obtain the training dataset; the training dataset includes visual data, location data, and text data;

[0213] Based on the cross-modal training mechanism, visual data, localization data, and text data are fused to obtain fused multimodal data;

[0214] The edge-side multimodal large model is trained based on the fused multimodal data.

[0215] In one possible implementation, the acquisition module 401 can also be used for:

[0216] Acquire the scene terrain data, facility distribution data, and control area data of the scene to be inspected;

[0217] Based on the scene terrain data, facility distribution data, and control area data, a 3D map of the scene to be inspected is generated;

[0218] Obtain inspection tasks;

[0219] Based on the inspection task and the 3D map of the scene to be inspected, a multi-objective optimization algorithm is used to generate the inspection route.

[0220] Inspections are conducted according to the inspection route to obtain multimodal data of the scene to be inspected.

[0221] In one possible implementation, after generating the inspection route using a multi-objective optimization algorithm based on the inspection task and a 3D map of the scene to be inspected, the robot's inspection device also includes an adjustment module, which can be specifically used for:

[0222] Real-time multimodal data is acquired based on the sensor cluster;

[0223] Based on the extended Kalman filter algorithm, real-time multimodal data is calibrated to obtain calibrated real-time multimodal data;

[0224] A local raster map is generated based on the calibrated real-time multimodal data;

[0225] Obstacle identification is performed based on the local grid map to obtain the obstacle identification results;

[0226] The inspection route is dynamically adjusted based on the obstacle identification results.

[0227] In one possible implementation, after generating a structured event report based on the event recognition results, the robot's inspection device also includes an early warning module, which can specifically be used for:

[0228] Based on the structured event report, identify the risk area corresponding to the event;

[0229] Obtain verification data for the risk area;

[0230] Based on the verification data, determine the risk level corresponding to the event;

[0231] Based on the risk level, execute the corresponding early warning action.

[0232] In one possible implementation, after determining the risk level corresponding to the event, the robot's inspection device includes a push module, which can specifically be used for:

[0233] Generate risk warning messages based on risk areas and risk levels;

[0234] Based on the preset priority queue scheduling mechanism, the risk warning messages are pushed and processed using the message queue telemetry transmission protocol.

[0235] In one possible implementation, after generating a structured event report based on the event recognition results, the robot's inspection device also includes a display module, which can specifically be used for:

[0236] Based on the structured event report, a visual output is generated, which includes an inspection trajectory map, an event distribution histogram, and robot status data.

[0237] The robot inspection device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and will not be described in detail here.

[0238] Figure 5 A schematic diagram of the structure of the electronic device provided in this application. Figure 5 As shown, the electronic device provided in this embodiment includes at least one processor 501 and a memory 502. Optionally, the electronic device further includes a communication component 503. The processor 501, memory 502, and communication component 503 are connected via a bus 504.

[0239] In a specific implementation, at least one processor 501 executes computer execution instructions stored in memory 502, causing at least one processor 501 to perform the above-described method.

[0240] The specific implementation process of processor 501 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0241] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0242] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0243] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0244] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0245] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0246] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0247] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0248] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0249] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0250] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0251] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0252] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0253] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A robot inspection method, characterized in that, include: Acquire multimodal data of the scene to be inspected; wherein, the multimodal data includes visual data and positioning data; Based on the end-side multimodal large model configured for the robot, event recognition processing is performed on the multimodal data to obtain event recognition results. The event recognition processing refers to the process of detecting abnormal features in the visual data, verifying the positional correlation of the positioning data, and combining preset inspection event rules to identify events in the scene to be inspected, thereby obtaining event recognition results. The event recognition results include event identifiers, event types, and multimodal data indexes associated with the events. Based on the event recognition results, the edge multimodal large model generates a structured event report; wherein the structured event report includes event time, corresponding location data, corresponding visual data, event type, event record, and corresponding text data.

2. The method according to claim 1, characterized in that, The end-side multimodal large model includes depthwise separable convolutional layers and lightweight convolutional layers; Accordingly, the step of performing event recognition processing on the multimodal data based on the end-side multimodal large model configured for the robot to obtain event recognition results includes: Based on the lightweight convolutional layer, preliminary feature extraction is performed on the multimodal data to obtain preliminary multimodal features; Based on the depth-separable convolutional layer, depth features are extracted from the preliminary multimodal features to obtain depth features; wherein, the depth features include scene features and abnormal target features corresponding to the visual data, as well as position coordinate features and position association features corresponding to the positioning data; Based on the preset inspection event rules, the deep features are classified and identified to obtain the event identification results.

3. The method according to claim 1, characterized in that, Before performing event recognition processing on the multimodal data based on the end-side multimodal large model configured for the robot to obtain the event recognition result, the method further includes: Obtain a training dataset; wherein the training dataset includes the visual data, the positioning data, and the text data; According to the cross-modal training mechanism, the visual data, the positioning data, and the text data are fused to obtain fused multimodal data; The edge-side multimodal large model is trained based on the fused multimodal data.

4. The method according to claim 1, characterized in that, The acquisition of multimodal data of the scene to be inspected includes: Acquire the scene terrain data, facility distribution data, and control area data of the scene to be inspected; Based on the scene terrain data, the facility distribution data, and the control area data, a 3D map of the scene to be inspected is generated; Obtain inspection tasks; Based on the inspection task and the 3D map of the scene to be inspected, a multi-objective optimization algorithm is used to generate the inspection route. The inspection is carried out according to the inspection route to obtain multimodal data of the scene to be inspected.

5. The method according to claim 4, characterized in that, After generating the inspection route using a multi-objective optimization algorithm based on the inspection task and the 3D map of the scene to be inspected, the process further includes: Real-time multimodal data is acquired based on the sensor cluster; The real-time multimodal data is calibrated according to the extended Kalman filter algorithm to obtain calibrated real-time multimodal data. A local raster map is generated based on the calibrated real-time multimodal data; Based on the local grid map, obstacle identification is performed to obtain the obstacle identification result; The inspection route is dynamically adjusted based on the obstacle identification results.

6. The method according to claim 1, characterized in that, After generating a structured event report based on the event identification result, the method further includes: Based on the structured event report, the risk area corresponding to the event is determined; Obtain the verification data for the risk area; Based on the verification data, the risk level corresponding to the event is determined; Based on the risk level, execute the corresponding early warning operation.

7. The method according to claim 6, characterized in that, After determining the risk level corresponding to the event, the following is included: Based on the risk area and the risk level, a risk warning message is generated; Based on a preset priority queue scheduling mechanism, the risk warning message is pushed out using a message queue telemetry transmission protocol.

8. The method according to claim 1, characterized in that, After generating a structured event report based on the event identification result, the method further includes: Based on the structured event report, a visual output is generated; wherein, the visual output includes an inspection trajectory diagram, an event distribution histogram, and robot state data.

9. An inspection device for a robot, characterized in that, include: The acquisition module is used to acquire multimodal data of the scene to be inspected; wherein, the multimodal data includes visual data and positioning data; The recognition module is used to perform event recognition processing on the multimodal data according to the end-side multimodal large model configured for the robot, and obtain event recognition results. The event recognition processing refers to the process of detecting abnormal features in the visual data, verifying the positional correlation of the positioning data, and combining preset inspection event rules to identify events in the scene to be inspected, thereby obtaining event recognition results. The event recognition results include event identifiers, event types, and multimodal data indexes associated with the events. The generation module is used to generate a structured event report based on the event recognition results using the edge multimodal large model; wherein the structured event report includes event time, location data corresponding to the event, visual data corresponding to the event, event type, event record, and text data corresponding to the event.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1-8.

Images