A mobile event detection system and method with laser radar and vision fusion

Abstract

The application discloses a kind of mobile event detection systems and methods of laser radar and vision fusion, the system uses layered decoupling architecture, including laser radar data processing module, vision data processing module, fusion decision module, data storage module and interactive interface module.Laser radar data processing module exports dynamic point cloud data and spatial position information, and vision data processing module exports vision detection result and semantic information;Fusion decision module establishes the spatial mapping of three-dimensional space and image pixel, realizes the cross verification and comprehensive detection of two-way triggering;Data storage module realizes layered storage and detection rule self-learning optimization, and interactive interface module realizes data interaction and result output.The application significantly improves the real-time and accuracy of mobile event detection, adapts to the dynamic target detection demand in complex scene.

Description

Technical Field

[0001] This invention relates to the field of intelligent sensing and target detection technology, specifically to a mobile event detection system and method that integrates lidar and vision. Background Technology

[0002] With the rapid development of unmanned power plant operations, artificial intelligence, autonomous driving, and intelligent robotics, environmental perception technology has become a key core technology in related industries. LiDAR, as an important perception sensor, can provide point cloud data in three-dimensional space and has important applications in environmental perception; visual sensors can provide rich semantic information, enabling accurate classification and recognition of objects.

[0003] Traditional single-sensor detection methods have significant limitations: pure LiDAR geometric detection methods are insufficient in object classification and fine contour segmentation, and are prone to false detections; pure vision detection methods are greatly affected by environmental factors such as lighting and weather, and lack depth information, resulting in low reliability in complex scenes. Furthermore, traditional point cloud processing methods often employ frame-by-frame processing, leading to high detection latency, which is insufficient to meet the real-time requirements of high-speed mobile platforms.

[0004] Existing fusion detection technologies mostly employ simple feature stitching methods, failing to achieve complementary and cross-validation of depth information from LiDAR and visual sensors. When handling rapid, sudden movements or complex occlusion scenarios, detection accuracy and system robustness still need improvement. Therefore, there is an urgent need for a motion event detection technology that can achieve deep fusion of LiDAR and visual information, with low latency and high reliability. Summary of the Invention

[0005] The purpose of this invention is to provide a mobile event detection system and method that integrates lidar and vision to meet the application requirements of dynamic target detection in complex environments.

[0006] To address the aforementioned technical problems, this invention provides a mobile event detection system that integrates lidar and vision, comprising a lidar data processing module, a vision data processing module, a fusion decision module, a data storage module, and an interactive interface module, with each module adopting a layered and decoupled architecture design. The lidar data processing module is used to acquire point cloud data streams from lidar sensors, perform real-time dynamic target detection on the point cloud data streams based on a streaming point processing mechanism, and output dynamic point cloud data and spatial location information. The visual data processing module is used to acquire video data streams from visual sensors, identify and locate specific object categories in the video data streams based on object detection algorithms, and output visual detection results and semantic information. The fusion decision module is communicatively connected to the lidar data processing module and the visual data processing module, respectively, and is used to establish the spatial mapping relationship between the lidar three-dimensional spatial coordinates and the visual image pixel coordinates. The fusion decision module triggers the visual data processing module to perform targeted analysis of the corresponding area based on dynamic point cloud data and spatial location information, and guides the lidar data processing module to perform key detection of the corresponding three-dimensional space based on visual detection results and semantic information, and comprehensively judges and outputs the final motion event detection result. The data storage module is used to store various types of data in layers during system operation and to build a closed-loop feedback link from the mobile event detection results to the feature library, thereby realizing the self-learning optimization of the system detection rules. The interactive interface module is used to provide a data interaction interface between the system and external systems, as well as a visualization display and alarm output of the motion event detection results.

[0007] According to the above scheme, the lidar data processing module adopts a streaming point processing mechanism to complete the motion state judgment when a single point cloud data arrives.

[0008] According to the above scheme, the lidar data processing module constructs an angular domain pixel-level depth memory model; the angular domain pixel-level depth memory model maps the three-dimensional point cloud data stream to the angle space, maintains a limited historical depth sample for each angle pixel and constructs a local static environment model, providing a static environment discrimination benchmark for real-time dynamic target detection.

[0009] According to the above scheme, the lidar data processing module realizes dynamic point recognition based on the occlusion consistency detection theory. By comparing the relationship between the current observation depth value and the historical depth front, it determines whether the target violates the static occlusion law and completes the dynamic point recognition.

[0010] According to the above scheme, the visual data processing module is based on the YOLOv8 target detection algorithm to realize the target recognition, positioning and target tracking of pedestrians, foreign objects, safety helmets and inspection equipment, and output visual detection results including object category, position and motion state.

[0011] According to the above scheme, when the lidar data processing module detects a moving point in a specified spatial area, the fusion decision module triggers the visual data processing module to perform targeted key analysis on the video frames of the corresponding area in the image; when the visual data processing module identifies a moving object in the image, it guides the lidar data processing module to perform key detection on the point cloud data of the corresponding three-dimensional spatial area, thus completing the cross-validation and false detection elimination of the two data streams.

[0012] According to the above scheme, the data storage module adopts a storage architecture that combines MySQL relational database and Redis in-memory database, and sets up a hierarchical storage structure of sample database, result database and log database. The sample database stores point cloud data stream and video data stream, the result database stores motion event detection results, and the log database stores system operation log data.

[0013] According to the above scheme, the interactive interface module provides an interface with the HTTP Hypertext Transfer Protocol; the MQTT message middleware interface is used for real-time data interaction with external intelligent devices and systems; and the HTTP Hypertext Transfer Protocol interface is used for remote access, visualization, and alarm output.

[0014] This invention also provides a method for motion event detection by fusion of lidar and vision, comprising the following steps: S1. Acquire 3D point cloud data streams through lidar sensors and video image data streams through vision sensors; S2. Real-time dynamic target detection is performed on the point cloud data stream based on the streaming point processing mechanism, and dynamic point cloud data and spatial location information are output. S3. Based on the object detection algorithm, identify and locate specific object categories in the video data stream, and output visual detection results and semantic information; S4. Establish the spatial mapping relationship between the three-dimensional spatial coordinates of the LiDAR and the pixel coordinates of the visual image; the fusion decision module triggers the visual data processing module to perform targeted analysis of the corresponding area based on dynamic point cloud data and spatial location information, and guides the LiDAR data processing module to perform key detection of the corresponding three-dimensional space based on visual detection results and semantic information, and comprehensively judges and outputs the final motion event detection result. S5. Output the motion event detection results to the outside world through the interface, and at the same time feed the motion event detection results back to the feature library to complete the iterative optimization of the detection rules. All kinds of data in the above steps are stored in layers synchronously.

[0015] According to the above scheme, in step S2, the three-dimensional point cloud data is first mapped to the angle space to construct the angle domain pixel-level depth memory model and the local static environment model. Then, through the occlusion consistency detection theory, the relationship between the current observed depth value and the historical depth front is compared to identify dynamic points and complete the output of dynamic point cloud data.

[0016] Beneficial effects This invention employs a layered, decoupled architecture to design a motion event detection system that integrates LiDAR and vision. The clear boundaries of responsibility for each functional module effectively improve the system's maintainability and scalability. The LiDAR data processing module performs real-time dynamic target detection on the point cloud data stream based on a streaming point processing mechanism, rapidly outputting dynamic point cloud data and spatial location information, ensuring the real-time nature of motion event detection. In conjunction with the vision data processing module, it identifies and locates targets in the video data stream, outputting visual detection results carrying semantic information, achieving effective complementarity between 3D spatial geometric information and image semantic information. This invention establishes a spatial mapping relationship between the LiDAR's 3D spatial coordinates and the visual image's pixel coordinates through a fusion decision module, achieving deep linkage between the two sensing data streams. This allows the LiDAR detection results to trigger targeted analysis of corresponding areas by the vision system, and the vision detection results to guide the LiDAR to focus on key areas in the corresponding 3D space. A comprehensive judgment is then made to output the final detection result, effectively overcoming the inherent limitations of single-sensor detection and significantly improving the accuracy and reliability of motion event detection in complex scenarios. Meanwhile, through the hierarchical storage and closed-loop feedback design of the data storage module, the system's detection rules can be self-learned and optimized, continuously improving the system's operating performance. Combined with the interactive interface module, the detection results can be interacted with externally, visualized, and alarmed, which can fully adapt to the mobile event detection needs of various intelligent sensing scenarios. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the hierarchical structure of a motion event detection system that fuses lidar and vision according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the data flow of a mobile event detection system that fuses lidar and vision according to an embodiment of the present invention; Figure 3 This is a flowchart of a motion event detection method that fuses lidar and vision according to an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0019] See Figure 1 , Figure 2This embodiment discloses a mobile event detection system that fuses LiDAR and vision, including a LiDAR data processing module, a vision data processing module, a fusion decision module, a data storage module, and an interactive interface module. Each module adopts a layered and decoupled architecture design. The lidar data processing module is used to acquire point cloud data streams from lidar sensors, perform real-time dynamic target detection on the point cloud data streams based on a streaming point processing mechanism, and output dynamic point cloud data and spatial location information. The visual data processing module is used to acquire video data streams from visual sensors, identify and locate specific object categories in the video data streams based on object detection algorithms, and output visual detection results and semantic information. The fusion decision module is communicatively connected to the lidar data processing module and the visual data processing module, respectively, and is used to establish the spatial mapping relationship between the lidar three-dimensional spatial coordinates and the visual image pixel coordinates. The fusion decision module triggers the visual data processing module to perform targeted analysis of the corresponding area based on dynamic point cloud data and spatial location information, and guides the lidar data processing module to perform key detection of the corresponding three-dimensional space based on visual detection results and semantic information, and comprehensively judges and outputs the final motion event detection result. The data storage module is used to store various types of data in layers during system operation and to build a closed-loop feedback link from the mobile event detection results to the feature library, thereby realizing the self-learning optimization of the system detection rules. The interactive interface module is used to provide a data interaction interface between the system and external systems, as well as a visualization display and alarm output of the motion event detection results.

[0020] Furthermore, the lidar data processing module adopts a streaming point processing mechanism to determine the motion state when a single point cloud data arrives.

[0021] Specifically, the streaming point processing mechanism does not require waiting for complete frame data, and reduces the detection granularity from the frame level to the point level.

[0022] Furthermore, the lidar data processing module constructs an angular domain pixel-level depth memory model; the angular domain pixel-level depth memory model maps the three-dimensional point cloud data stream to the angle space, maintains a limited historical depth sample for each angle pixel and constructs a local static environment model, providing a static environment discrimination benchmark for real-time dynamic target detection.

[0023] Furthermore, the lidar data processing module realizes dynamic point recognition based on the occlusion consistency detection theory. By comparing the relationship between the current observation depth value and the historical depth front, it determines whether the target violates the static occlusion law and completes the dynamic point recognition.

[0024] Furthermore, the visual data processing module is based on the YOLOv8 target detection algorithm to realize the identification, localization and tracking of targets such as pedestrians, foreign objects, safety helmets and inspection equipment, and outputs visual detection results including object category, position and motion state.

[0025] Furthermore, when the lidar data processing module detects a moving point in a specified spatial area, the fusion decision module triggers the visual data processing module to perform targeted key analysis on the video frames of the corresponding area in the image; when the visual data processing module identifies a moving object in the image, it guides the lidar data processing module to perform key detection on the point cloud data of the corresponding three-dimensional spatial area, thus completing the cross-validation and false detection elimination of the two data streams.

[0026] Furthermore, the data storage module adopts a storage architecture that combines a MySQL relational database and a Redis in-memory database, and sets up a hierarchical storage structure of a sample database, a result database, and a log database. The sample database stores point cloud data streams and video data streams, the result database stores motion event detection results, and the log database stores system operation log data.

[0027] Specifically, the data storage module supports data traceability and analysis.

[0028] Furthermore, the interactive interface module provides an interface with the HTTP Hypertext Transfer Protocol; the MQTT message middleware interface is used for real-time data interaction with external intelligent devices and systems; and the HTTP Hypertext Transfer Protocol interface is used for remote access, visualization, and alarm output.

[0029] join Figure 3 This embodiment also provides a method for motion event detection by fusion of lidar and vision, including the following steps: S1. Acquire 3D point cloud data streams through lidar sensors and video image data streams through vision sensors; S2. Real-time dynamic target detection is performed on the point cloud data stream based on the streaming point processing mechanism, and dynamic point cloud data and spatial location information are output. S3. Based on the object detection algorithm, identify and locate specific object categories in the video data stream, and output visual detection results and semantic information; S4. Establish the spatial mapping relationship between the three-dimensional spatial coordinates of the LiDAR and the pixel coordinates of the visual image; the fusion decision module triggers the visual data processing module to perform targeted analysis of the corresponding area based on dynamic point cloud data and spatial location information, and guides the LiDAR data processing module to perform key detection of the corresponding three-dimensional space based on visual detection results and semantic information, and comprehensively judges and outputs the final motion event detection result. S5. Output the motion event detection results to the outside world through the interface, and at the same time feed the motion event detection results back to the feature library to complete the iterative optimization of the detection rules. All kinds of data in the above steps are stored in layers synchronously.

[0030] Specifically, this method, based on point-level dynamic detection, generates instance-level representations of dynamic targets through spatial clustering, achieving structured output from dynamic points to dynamic objects, and supporting multi-target tracking and SLAM dynamic object filtering. Furthermore, in step S2, the three-dimensional point cloud data is first mapped to the angle space to construct an angle domain pixel-level depth memory model and a local static environment model. Then, through the occlusion consistency detection theory, the relationship between the current observed depth value and the historical depth front is compared to identify dynamic points and complete the output of dynamic point cloud data.

[0031] The mobile event detection system based on the fusion of lidar and vision in this invention is built and runs on ROS, facilitating the integration of other ROS packages. It employs PCL (Point Cloud Library) to process point cloud data, the Eigen library for linear algebra, matrix, and vector operations, and TBB (Intel Threading Building Blocks) for multi-threaded parallel computing, ensuring system real-time performance and computational efficiency. The system adopts a layered architecture: interaction layer → core control layer → business logic layer → basic service layer → dependency layer. Each module has a single responsibility and low coupling, facilitating independent development, testing, and upgrades. It supports real-time ROS operation and is easily embedded into SLAM systems. The system's hardware environment includes a LiDAR sensor, a vision sensor (camera), and an embedded computing platform. The software environment is based on the Linux operating system. MQTT is used as a message middleware to enable communication between modules and between systems, and HTTP is used as a hypertext transfer protocol to provide a web access interface.

[0032] The specific implementation of the LiDAR data processing module involves: acquiring streaming point cloud data from the LiDAR sensor; employing a dynamic real-time point detection mechanism based on streaming point processing, processing each point cloud data point immediately upon arrival, without waiting for a complete frame; mapping the 3D point cloud data to angle space to construct a pixel-level depth memory model in the angle domain, maintaining a limited set of historical depth samples for each pixel location to form a local static environment model; and using an occlusion consistency detection algorithm to compare the currently detected depth value with the historical depth front. If the current depth value violates the static occlusion rule, the point is identified as a dynamic point, achieving rapid identification of dynamic points with a detection latency down to the microsecond level. LiDAR data processing utilizes multi-threaded parallel computing technology, accelerating computation based on TBB, processing point cloud data using PCL, and performing linear algebra operations using the Eigen library. The visual data processing module is implemented as follows: It trains and deploys models based on the YOLOv8 object detection algorithm, and trains the model to identify specific object categories such as pedestrians, vehicles, foreign objects, safety helmets, and equipment for application scenarios such as autonomous driving and intelligent inspection; it interfaces with the video data stream acquired by the visual sensor to perform real-time object detection and tracking, and outputs information such as the object category, pixel-level position, movement speed and direction. The detection speed is fast and can meet the real-time requirements.

[0033] The specific implementation of the fusion decision module: As the core scheduling module of the system, it receives the output data from the LiDAR data processing module and the vision data processing module, establishes a spatial coordinate mapping relationship, and realizes the mutual conversion between the LiDAR 3D spatial coordinates and the visual image pixel coordinates. When the LiDAR data processing module detects a moving point in a certain spatial area, the fusion decision module triggers the vision data processing module to perform targeted key analysis on the video frames of the corresponding area to verify whether there is a known moving object in the area. Conversely, when the vision data processing module identifies a fast-moving object in the image, the fusion decision module guides the LiDAR data processing module to focus on the point cloud data in the corresponding 3D space to confirm its motion state. The fusion decision module performs cross-validation and comprehensive judgment on the detection results of the two data sources, eliminates false detection information from a single sensor, and outputs the final motion event detection result.

[0034] Data storage module implementation: A hierarchical storage design is adopted. The sample library stores the original point cloud data and video data, the result library stores the core detection result data, and the log library stores the system's operation process data, which facilitates data traceability and analysis. At the same time, a data closed loop is constructed to feed the detection results back to the feature library, continuously optimize the detection rules and model parameters, and realize the system's self-learning and iterative upgrade.

[0035] The interactive interface module provides multiple interface types: the MQTT interface is used for real-time data interaction with other smart devices and systems, the HTTP interface is used for web visualization and remote access, and it also supports alarm output of detection results. When an abnormal movement event is detected, it can issue alarm information in a timely manner.

[0036] The system of this invention can operate 24 / 7 without failure, and can automatically restart within 3 seconds when a failure occurs, ensuring the stability and reliability of the system. The system architecture has good scalability and can be connected to other systems by adding API interfaces, or the detection modules can be expanded to support new object types and detection requirements.

[0037] The present invention has at least the following beneficial effects: Compared with the prior art, the present invention has the following advantages: 1. High real-time detection performance: It adopts a dynamic point real-time detection mechanism based on streaming point processing, which reduces the detection granularity from the frame-level to the point-level, achieving microsecond-level detection latency, significantly reducing data processing time, and is suitable for high-speed mobile platforms; 2. High detection accuracy: It achieves deep fusion and cross-validation of LiDAR geometric motion information and visual semantic information, which effectively reduces the false detection and false detection rate of a single sensor, especially when dealing with rapid sudden motion or complex occlusion scenarios. 3. Strong environmental adaptability: LiDAR's geometric detection does not rely on semantic information and training data, and is effective for unknown objects; visual detection provides rich semantic information. The combination of the two enables the system to adapt to different environments and various types of sensors. 4. Structured Output: Based on point-level dynamic detection, a dynamic target instance-level representation is generated through spatial clustering, realizing a structured output from dynamic points to dynamic objects, which facilitates subsequent multi-target tracking and decision analysis. 5. Self-learning optimization: Construct a data closed-loop design, and the detection results can be fed back to the feature library to continuously optimize the detection rules, realize the system's self-learning and iterative upgrade, and improve the system's long-term operating performance.

[0038] It should be noted that, depending on the implementation needs, the various steps / components described in this application can be broken down into more steps / components, or two or more steps / components or parts of the operation of steps / components can be combined into new steps / components to achieve the purpose of this invention.

[0039] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A motion event detection system that fuses lidar and vision, characterized in that, It includes a lidar data processing module, a visual data processing module, a fusion decision module, a data storage module, and an interactive interface module. Each module adopts a layered and decoupled architecture design. The lidar data processing module is used to acquire point cloud data streams from lidar sensors, perform real-time dynamic target detection on the point cloud data streams based on a streaming point processing mechanism, and output dynamic point cloud data and spatial location information. The visual data processing module is used to acquire video data streams from visual sensors, identify and locate specific object categories in the video data streams based on object detection algorithms, and output visual detection results and semantic information. The fusion decision module is communicatively connected to the lidar data processing module and the visual data processing module, respectively, and is used to establish the spatial mapping relationship between the lidar three-dimensional spatial coordinates and the visual image pixel coordinates. The fusion decision module triggers the visual data processing module to perform targeted analysis of the corresponding area based on dynamic point cloud data and spatial location information, and guides the lidar data processing module to perform key detection of the corresponding three-dimensional space based on visual detection results and semantic information, and comprehensively judges and outputs the final motion event detection result. The data storage module is used to store various types of data in layers during system operation and to build a closed-loop feedback link from the mobile event detection results to the feature library, thereby realizing the self-learning optimization of the system detection rules. The interactive interface module is used to provide a data interaction interface between the system and external systems, as well as a visualization display and alarm output of the motion event detection results.

2. The motion event detection system integrating lidar and vision according to claim 1, characterized in that, The lidar data processing module adopts a streaming point processing mechanism to determine the motion state when a single point cloud data arrives.

3. The motion event detection system integrating lidar and vision according to claim 1, characterized in that, The lidar data processing module constructs an angular domain pixel-level depth memory model; the angular domain pixel-level depth memory model maps the three-dimensional point cloud data stream to the angle space, maintains a limited historical depth sample for each angle pixel and constructs a local static environment model, providing a static environment discrimination benchmark for real-time dynamic target detection.

4. The motion event detection system integrating lidar and vision according to claim 1, characterized in that, The lidar data processing module achieves dynamic point identification based on the occlusion consistency detection theory. By comparing the current observation depth value with the historical depth front, it determines whether the target violates the static occlusion rule and completes the identification of dynamic points.

5. The motion event detection system integrating lidar and vision according to claim 1, characterized in that, The visual data processing module is based on the YOLOv8 target detection algorithm to realize the identification, localization and tracking of targets such as pedestrians, foreign objects, safety helmets and inspection equipment, and outputs visual detection results including object category, position and motion state.

6. The motion event detection system integrating lidar and vision according to claim 1, characterized in that, When the lidar data processing module detects a moving point in a specified spatial area, the fusion decision module triggers the visual data processing module to perform targeted key analysis on the video frames of the corresponding area in the image; when the visual data processing module identifies a moving object in the image, it guides the lidar data processing module to perform key detection on the point cloud data of the corresponding three-dimensional spatial area, completing the cross-validation and false detection elimination of the two data streams.

7. The motion event detection system integrating lidar and vision according to claim 1, characterized in that, The data storage module adopts a storage architecture that combines a MySQL relational database and a Redis in-memory database. It sets up a hierarchical storage structure of a sample database, a result database, and a log database. The sample database stores point cloud data streams and video data streams, the result database stores motion event detection results, and the log database stores system operation log data.

8. The motion event detection system integrating lidar and vision according to claim 1, characterized in that, The interactive interface module provides an interface with the HTTP Hypertext Transfer Protocol; the MQTT message middleware interface is used for real-time data interaction with external intelligent devices and systems; and the HTTP Hypertext Transfer Protocol interface is used for remote access, visualization, and alarm output.

9. A method for motion event detection by fusing lidar and vision, characterized in that, Includes the following steps: S1. Acquire 3D point cloud data streams through lidar sensors and video image data streams through vision sensors; S2. Real-time dynamic target detection is performed on the point cloud data stream based on the streaming point processing mechanism, and dynamic point cloud data and spatial location information are output. S3. Based on the object detection algorithm, identify and locate specific object categories in the video data stream, and output visual detection results and semantic information; S4. Establish the spatial mapping relationship between the three-dimensional spatial coordinates of the lidar and the pixel coordinates of the visual image; The fusion decision module triggers the visual data processing module to perform targeted analysis of the corresponding area based on dynamic point cloud data and spatial location information, and guides the lidar data processing module to perform key detection of the corresponding three-dimensional space based on visual detection results and semantic information, and comprehensively judges and outputs the final motion event detection results. S5. Output the motion event detection results to the outside world through the interface, and at the same time feed the motion event detection results back to the feature library to complete the iterative optimization of the detection rules. All kinds of data in the above steps are stored in layers synchronously.

10. The method for motion event detection by fusion of lidar and vision according to claim 9, characterized in that, In step S2, the 3D point cloud data is first mapped to the angle space to construct the angular domain pixel-level depth memory model and the local static environment model. Then, through the occlusion consistency detection theory, the relationship between the current observed depth value and the historical depth front is compared to identify dynamic points and complete the output of dynamic point cloud data.

Images