Lidar-based multi-chain pooling neural network target detection system and method

Abstract

The present application relates to the field of computer vision and three-dimensional perception technology, in particular to a multi-chain pooling neural network target detection system based on laser radar. The system comprises data acquisition, preprocessing, multi-chain pooling neural network detection and visualization modules. The neural network adopts a three-dimensional sparse backbone and a two-dimensional dense backbone containing an inverted bottle neck structure; the key modules, multi-chain sparse pooling module and multi-chain dense pooling module, are connected through parallel branches and short circuit connection design, and cooperatively extract global and local features. The present application simultaneously supports indoor and outdoor scene detection through fine-tuning technology, guarantees input order using special data processing logic, and realizes edge deployment with the help of ONNXRuntime and custom operators. The system solves the problems of single scene, insufficient detection accuracy of long-distance and small targets, and the split between algorithm and visualization software in the prior art, and realizes high-precision, real-time and easy-to-use end-to-end three-dimensional target detection.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D perception technology, and more specifically, to a target detection system and method based on a multi-chain pooling neural network using LiDAR. Background Technology

[0002] LiDAR (Light Detection and Ranging) is a remote sensing technology that acquires target distance, orientation, and three-dimensional spatial information by emitting and measuring reflected laser beams. The point cloud data it provides is highly accurate and is widely used in fields requiring three-dimensional environmental perception, such as autonomous driving and terrain mapping. Three-dimensional target detection, as an important branch of computer vision, aims to identify and locate objects of interest from the aforementioned point cloud data, and is a key step in achieving intelligent environmental understanding.

[0003] With the advancement of deep learning technology, 3D target detection models based on LiDAR point clouds have continued to develop. For example, models such as LargeKernel3D, PV-RCNN++, FSD, and DVST-Voxel have achieved significant results in this field. In particular, models like HEDNet and SAFDNet represent advanced solutions for outdoor autonomous driving scenarios. These models typically follow a general architecture including a voxel feature encoding module, 3D and 2D backbone networks, a bird's-eye view transformation module, and a detection module. HEDNet utilizes encoder-decoder blocks in its 3D / 2D backbone network to capture long-range dependencies between features, thereby improving the detection capability for large and distant targets. SAFDNet proposes an adaptive feature diffusion strategy to address the problem of missing center features of distant targets in the backbone network feature map, thus improving long-range detection performance.

[0004] The training and evaluation of these models rely on large-scale public datasets. HEDNet is primarily trained using the WaymoOpen and nuScenes datasets, supporting detection distances of 0-75 meters, and reporting a detection score (NDS) of 72.0 on the nuScenes dataset. SAFDNet was further trained on the Argoverse2 dataset, extending the effective detection distance to 0-200 meters and achieving high average accuracy (L1 / L2AP) and heading accuracy. These datasets, such as WaymoOpen containing over 100,000 scenes, nuScenes consisting of 1,000 scenes, and Argoverse2 providing approximately 20,000 LiDAR sequences, provide a data foundation for research on 3D detection for outdoor autonomous driving.

[0005] However, existing advanced technologies still have several limitations. First, models like HEDNet and SAFDNet are primarily optimized for outdoor autonomous driving scenarios, with their training data and model design focusing on targets such as vehicles. Their application potential in indoor scenarios (such as airports and shopping malls) has not been fully explored and validated, as the dense layouts and diverse range of small and medium-sized objects in indoor environments present different detection challenges. Second, with the continuous improvement of LiDAR hardware detection range (currently reaching up to 200 meters), the detection accuracy of existing neural networks at long distances, especially in the 75-200 meter range, still needs further improvement. Third, existing model architectures (such as encoder-decoder based designs) perform well in recognizing large targets, but their ability to recognize small and medium-sized objects such as pedestrians, pets, and indoor furniture is relatively limited, resulting in only average detection performance.

[0006] From the perspective of system integration and application deployment, the existing technology ecosystem suffers from functional fragmentation. Currently used point cloud processing and visualization software, such as GlobalMapper, Pix4Dmapper, and QuickTerrainReader, primarily offer data import, visualization browsing, basic editing, and specific product (such as digital surface model) generation functions, but generally lack integration of advanced intelligent perception algorithms such as 3D object detection and point cloud segmentation. Users find it difficult to complete the entire analysis process from raw data to semantic understanding results within a single software platform. Furthermore, raw point cloud data acquired by LiDAR typically requires specific preprocessing (such as data cleaning and format normalization) before being used as input for neural networks, but existing software lacks dedicated data processing functions to support such algorithms. Simultaneously, advancements in semiconductor technology have driven the development of edge computing capabilities, making real-time neural network inference possible on edge devices. However, existing visualization software lacks the ability to deploy and run neural network models at the edge, which limits the realization of real-time inference and also affects data privacy protection.

[0007] Therefore, there is a need for a 3D target detection system and solution that can effectively take into account complex indoor and outdoor scenarios, improve the detection accuracy of small and medium-sized targets across the entire range (especially at long distances), and achieve full-process integration from data preprocessing and intelligent detection to result visualization. Summary of the Invention

[0008] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a multi-chain pooling neural network target detection system and method based on lidar to solve the problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution: This invention provides a multi-chain pooling neural network target detection system based on lidar, comprising: An ordered data preprocessing and real-time inference subsystem deployed at the edge, and a visualization and interaction subsystem deployed at the user side; The ordered data preprocessing and real-time inference subsystem includes: The data receiving and ordered reassembly module is connected to the LiDAR at its input end to receive the raw point cloud data stream and outputs sequentially continuous point cloud frames based on the sequence index, sliding window mechanism and timeout retransmission judgment rules embedded in the UDP data packet. A multi-chain pooling neural network detection module, whose input is connected to the output of the data receiving and ordered recombination module, is used to perform voxelization, feature encoding, and 3D target detection on the point cloud frame, and output target category and 3D bounding box information; the multi-chain pooling neural network detection module includes a multi-chain pooling neural network, which has a 3D sparse backbone network and a 2D dense backbone network with an inverted bottleneck structure, wherein at least one multi-chain sparse pooling module is provided in the 3D sparse backbone network, and at least one multi-chain dense pooling module is provided in the 2D dense backbone network; An adaptive inference deployment module, integrated with the multi-chain pooled neural network detection module, is used to provide heterogeneous inference engine support and custom acceleration operators for the multi-chain pooled neural network on edge computing devices. The visualization and interaction subsystem has its input end connected to the output end of the multi-chain pooled neural network detection module, and is used to fuse and display the target category, 3D bounding box information, and corresponding original point cloud data.

[0010] A 3D target detection method based on lidar point clouds includes the following steps: S1. Data stream ordering and reassembly: Receive the raw point cloud data stream transmitted by the lidar via the UDP protocol, extract the sequence index from each data packet, cache it according to the sliding window rule, and trigger timeout waiting or data compensation when packet loss or sequence discontinuity is detected to obtain a point cloud frame with a coherent time sequence. S2. Voxel feature generation: The temporally coherent point cloud frames are divided into voxel grids and feature encoded to obtain voxel features; S3. Multi-chain collaborative feature extraction and target detection: The voxel features are input into a multi-chain pooling neural network; the three-dimensional sparse features are extracted through the multi-chain sparse pooling module in the three-dimensional sparse backbone network of the neural network; the three-dimensional sparse features are converted into two-dimensional dense features through the bird's-eye view feature generation layer; the two-dimensional dense features are enhanced and fused through the multi-chain dense pooling module in the two-dimensional dense backbone network; and the three-dimensional bounding box and category information of the target are output through the detection head. S4. Visualization of detection results: The three-dimensional bounding box and category information are overlaid and visualized with the original point cloud data.

[0011] Furthermore, the multi-chain pooling neural network includes a voxel feature encoding layer, a three-dimensional sparse backbone network, a bird's-eye view feature generation layer, a two-dimensional dense backbone network, and a detection head connected in sequence.

[0012] Furthermore, the multi-chain sparse pooling module includes a dual-chain downsampling submodule, a sparse three-dimensional deconvolution submodule, and a first short-circuit connection connecting the input and output of the dual-chain downsampling submodule; The dual-chain downsampling submodule includes a first branch and a second branch configured in parallel; The first branch includes a large kernel sparse 3D convolutional unit and a first normalization unit connected in sequence; The second branch includes a three-dimensional submanifold convolution unit, a second normalization unit, an activation function unit, and a sparse three-dimensional max pooling unit connected in sequence; The outputs of the first branch and the second branch are superimposed with the input features transmitted through the first short-circuit connection, processed by the activation function, and then input to the sparse three-dimensional deconvolution submodule.

[0013] Furthermore, the multi-chain dense pooling module includes a multi-chain convolution sub-module, an average pooling sub-module, a residual convolution sub-module, a dense deconvolution sub-module, and a second short-circuit connection connecting the input of the multi-chain convolution sub-module and the output of the residual convolution sub-module. The multi-chain convolutional submodule includes a third branch and a fourth branch arranged in parallel, and both the third branch and the fourth branch contain convolutional units and normalization units; The outputs of the third and fourth branches are superimposed and processed by the activation function, and then sequentially input to the average pooling submodule, the residual convolution submodule, and the dense deconvolution submodule.

[0014] Furthermore, the data receiving and orderly reassembly module is configured as follows: Parse UDP packets to extract point cloud data and sequence indexes; The data is stored in the sliding window cache matrix according to the sequence index; If the expected sequence of data packets is not received within the preset timeout period, compensation will be made based on the previous frame data or an interpolation method. When the sequence index exceeds the current sliding window range, output the ordered data within the current window as a point cloud frame and initialize a new window.

[0015] Furthermore, the adaptive inference deployment module supports inference engines including at least one of ONNXRuntime, TensorRT, and CoreML; the implementation language of the custom acceleration operator includes C++, CUDA, or Metal.

[0016] Furthermore, the visualization and interaction subsystem is a desktop application based on the PyQt5 framework or a web application based on the Streamlit framework.

[0017] Furthermore, the basic unit of the inverted bottleneck structure includes, in sequence, a 1x1x1 convolutional layer, a 3x3x3 depth convolutional layer, and a 1x1x1 convolutional layer.

[0018] Furthermore, the multi-chain pooled neural network is trained through the following process: pre-training based on an outdoor scene point cloud dataset; and fine-tuning the parameters based on an indoor scene point cloud dataset using a low-rank adaptation technique.

[0019] Compared with the prior art, the technical solution provided by this invention can achieve the following beneficial technical effects: Regarding detection accuracy and robustness: By employing parallel global and local feature extraction branches (multi-chain sparse pooling module and multi-chain dense pooling module) in the multi-chain pooling neural network, the network can collaboratively capture wide-area contextual information and fine geometric structures of the scene. This design enhances the model's ability to represent features of distant targets as well as small to medium-sized targets such as pedestrians and small furniture, thereby improving the overall detection accuracy and robustness in complex scenes across the entire distance range (0-200 meters).

[0020] Regarding scene adaptability and model generalization ability: A phased training strategy combining "outdoor scene pre-training with indoor scene low-rank adaptation (LoRA) fine-tuning" enables a single multi-chain pooling neural network model to effectively adapt to differences in target density, scale distribution, and object categories between indoor and outdoor scenes. This method helps broaden the application boundaries of this technical solution, extending it beyond outdoor autonomous driving to intelligent perception tasks in indoor environments such as airports, shopping malls, and warehouses, thus improving the technology's versatility.

[0021] In terms of system integration and ease of use: By integrating the ordered reassembly of UDP data streams, real-time inference of multi-chain pooled neural networks, and interactive 3D visualization of detection results into a unified software framework, an end-to-end analysis process from receiving raw point cloud data and intelligent processing to intuitive display of results is achieved. This effectively overcomes the current situation where data processing tools, intelligent algorithms, and visualization platforms are fragmented, thus facilitating user operation.

[0022] In terms of deployment flexibility, real-time performance, and data security: Based on a deployment architecture of "heterogeneous inference engine adaptation" and "custom acceleration operator registration," this system can be flexibly deployed on various edge computing devices and fully utilize hardware acceleration capabilities to achieve efficient inference. This not only ensures the real-time requirements of 3D object detection, meeting the low-latency response needs of applications such as autonomous driving and security monitoring, but also helps protect the security and privacy of user data, as the raw point cloud data can be processed locally without being uploaded to the cloud.

[0023] Regarding input data reliability and system stability: A UDP data reassembly mechanism based on sliding window, sequence indexing, timeout determination, and packet loss compensation was designed. This mechanism can proactively address common network transmission issues such as packet loss and out-of-order delivery, ensuring that each frame of point cloud data input to the neural network for inference maintains good temporal continuity and spatial integrity. This provides a data foundation for the stable and reliable operation of the backend detection algorithm, reducing the risk of false positives or false negatives due to data quality issues. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the structure of a multi-chain pooled neural network provided in an embodiment of the present invention.

[0025] Figure 2 This is a schematic diagram of the structure of the multi-chain sparse pooling module provided in an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the structure of a multi-chain dense pooling module provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] This invention provides a multi-chain pooling neural network target detection system based on lidar. For example... Figure 1 As shown, the system logically mainly includes a data acquisition unit, a data processing and algorithm unit, and a human-computer interaction unit connected in sequence.

[0029] Ordered data preprocessing and real-time inference subsystem deployed at the edge: Typically deployed in embedded AI computing boxes or edge servers, it is responsible for receiving and reassembling point cloud data in real time and performing high-performance neural network inference.

[0030] The visualization and interaction subsystem deployed on the user side runs on a desktop computer or web browser, providing users with an intuitive display of 3D detection results and an interactive interface.

[0031] The following combination Figures 1 to 3 The specific implementation of this technical solution will be described in detail.

[0032] 1. Implementation of Multi-Chain Pooled Neural Network Architecture refer to Figure 1 The multi-chain pooling neural network constructed in this embodiment takes ordered recombined LiDAR point cloud frames as input and outputs 3D bounding boxes and category labels. The network specifically includes the following parts: Voxel Feature Encoding Layer: This layer receives temporally continuous point cloud frames. First, it creates a uniform voxel grid in 3D space, dividing the point cloud into regular voxel units. Then, it aggregates features (such as coordinates and reflection intensity) of points within each non-empty voxel to generate the aggregated features of that voxel. Finally, it outputs a sparse 3D voxel feature tensor, which serves as the input to subsequent networks.

[0033] The 3D sparse backbone network is the core of the network for processing 3D point clouds, consisting of multiple stacked stages. The core of each stage is a multi-chain sparse pooling module. This module, through parallel global branches (large kernel sparse convolutions) and local branches (sub-manifold convolutions and pooling), collaboratively extracts global contextual information and fine local geometric features of the scene, enabling the network to simultaneously focus on large-scale scene structure and close-range object details. Throughout the processing, the network maintains the sparsity of features, processing large-scale scenes with extremely high efficiency. This backbone network adopts an inverted bottleneck structure, performing a sequence of operations within each basic unit: "expansion (1x1x1 convolution) - depthwise convolution (3x3x3 spatial convolution) - compression (1x1x1 convolution)," enabling the extraction of richer features at a lower computational cost.

[0034] Bird's-eye view feature generation layer: The 3D sparse feature tensor output by the 3D sparse backbone network is compressed or max-pooled along the height direction (Z-axis) to generate a 2D, dense bird's-eye view feature map. This transformation converts the 3D detection problem into an object detection problem on a 2D feature map, while preserving the key position, size, and semantic information of the object on the horizontal plane.

[0035] Two-dimensional dense backbone network: This network receives a bird's-eye view feature map, and its core is composed of multiple multi-chain dense pooling modules connected in series. This module performs deeper feature fusion and enhancement on the two-dimensional dense feature map through a combination of multi-branch convolution, average pooling, residual connections and deconvolution, further improving the semantic richness and spatial consistency of the features.

[0036] Detection head: Receives the fused feature map output by the two-dimensional dense backbone network, predicts the class probability, center position of the bounding box, size (length, width and height) and orientation angle of the target at each preset anchor point or feature map position through a series of convolutional layers, and finally outputs the three-dimensional detection results of all detected targets.

[0037] 2. Specific implementation of key modules 2.1 Multi-chain sparse pooling module like Figure 2 As shown, this module is the core of the three-dimensional sparse backbone network. Its implementation process is as follows: The input features are first copied to three paths: a first short-circuit connection, and two parallel processing branches (i.e., dual-chain downsampling submodules).

[0038] In the dual-chain downsampling submodule, the first branch (global branch) passes sequentially through a large kernel sparse 3D convolutional unit (e.g., kernel size 7x7x7) and a first normalization unit (e.g., batch normalization), aiming to capture global contextual information and long-range dependencies under a large receptive field.

[0039] The second branch (local branch) sequentially passes through a 3D submanifold convolution unit, a second normalization unit, a ReLU activation function unit, and a sparse 3D max pooling unit, focusing on extracting and enhancing detailed geometric features within the local neighborhood.

[0040] The output features of the two branches are added element-wise to the input features directly passed through the first short-circuit connection, and then passed through a ReLU activation function to form features that integrate multi-level information.

[0041] The fused features are fed into a sparse 3D deconvolution submodule for upsampling to restore or improve the spatial resolution of the feature map, making it easier to fuse with features from subsequent network layers.

[0042] 2.2 Multi-chain dense pooling module like Figure 3 As shown, this module is the core of the two-dimensional dense backbone network. Its implementation process is as follows: The input features are fed into a multi-chain convolutional submodule. This submodule contains a main branch (third branch) by default and can optionally enable a downsampling branch (fourth branch) to construct multi-scale features.

[0043] Both branches contain a convolutional unit (such as a standard 3x3 convolution) and a normalization unit. If the fourth branch is enabled, its convolution stride can be set to 2 to achieve downsampling of the feature map.

[0044] The outputs of the two branches are added together and then activated by the ReLU function. The result is then fed into the average pooling submodule for feature compression and smoothing, which enhances the translation invariance of the features.

[0045] The compressed features undergo nonlinear transformation and feature activation through a residual convolution submodule (usually consisting of 1x1 convolution, normalization, and activation functions).

[0046] The output of the residual convolutional submodule is added to the features passed directly from the input of the multi-chain convolutional submodule through the second short-circuit connection to form a residual learning structure, which effectively alleviates the gradient vanishing problem in deep network training.

[0047] Finally, the features are upsampled by the dense deconvolution submodule and output to provide the detection head with the required multi-scale, high semantic features.

[0048] 3. Implementation of Network Training and Fine-tuning To obtain a general detection model capable of handling both indoor and outdoor scenes, the following two-stage training process is implemented: First, large-scale outdoor autonomous driving point cloud datasets (such as nuScenes and Argoverse2) are collected and preprocessed. These datasets are then used to perform end-to-end pre-training of the multi-chain pooling neural network. The loss function typically includes a focal loss for classification and a smoothed L1 loss for bounding box regression.

[0049] Then, indoor scene point cloud datasets (such as S3DIS or ScanNet) are collected. Low-rank adaptation (LoRA) and other efficient parameter fine-tuning techniques are employed to fine-tune some parameters of the pre-trained model (typically the attention mechanism or parameters of the core convolutional layers). This method enables the model to quickly adapt to the characteristics of indoor scenes—higher target density, smaller scale, and different class distributions—with minimal additional parameters, while preserving the general feature representation capabilities learned in outdoor scenes to the greatest extent possible, thus achieving effective adaptation of a single model to dual scenes.

[0050] 4. System Integration and Deployment Implementation 4.1 Data Preprocessing and Robust Ordered Reassembly In the software of the edge subsystem, socket programming or efficient network libraries are used to listen on a specified port and receive the data stream sent by the LiDAR via the UDP protocol.

[0051] Parse each UDP packet and extract the point cloud data field and the built-in frame sequence number.

[0052] The system maintains a fixed-size sliding window buffer (e.g., corresponding to 10 consecutive frames of data). Based on the parsed sequence number, the data packet is stored in the corresponding position of the sliding window.

[0053] Set a timeout threshold (e.g., 50ms). If a data packet with the expected sequence number is not received within the threshold time, the data for that frame is considered lost. At this time, a packet loss compensation mechanism is triggered, which uses a copy of the valid data from the previous frame or performs linear interpolation using data from the previous and next frames to generate a compensation frame to fill the window, in order to ensure the continuity of the sequence.

[0054] When the sequence number of a new data packet indicates that it has exceeded the range of the current sliding window, the system outputs the ordered and completely reconstructed point cloud data within the current window as a frame and sends it to the subsequent processing pipeline. At the same time, a new sliding window is initialized to continue receiving subsequent data.

[0055] The aforementioned collaborative mechanism based on sliding windows, timeout determination, and intelligent compensation constitutes a robust data order reassembly system. Its purpose is to proactively mitigate the inherent unreliability issues in UDP transmission, ensuring that each frame of point cloud data input to the multi-chain pooled neural network is continuous, complete, and reliable in both time and space, thus providing a stable data foundation for high-precision detection.

[0056] 4.2 Adaptive Edge Model Deployment To meet real-time requirements and adapt to the heterogeneous computing environment of edge devices, this invention adopts a deployment architecture of "inference engine adaptation + custom operator registration".

[0057] First, export the trained and fine-tuned model (such as in PyTorch or TensorFlow format) to a common intermediate representation format (such as ONNX).

[0058] The system features an adaptive inference deployment module that can adapt to various mainstream heterogeneous inference engines such as ONNX Runtime, TensorRT, or CoreML. Users can select the optimal engine based on the specific hardware of the edge device (NVIDIA GPU, Intel CPU, Apple Silicon, etc.).

[0059] For key operations in multi-chain pooled neural networks, such as voxel feature encoding and sparse 3D convolution, these operations may not be natively supported by some inference engines. Therefore, custom acceleration operators need to be implemented. For example, on NVIDIA GPU platforms, high-performance operators can be implemented using CUDA programming; on x86 CPU platforms, they can be implemented using C++ with SIMD instruction optimizations; and on Apple devices, they can be implemented using the Metal graphics API. The implemented operators are then registered with the corresponding inference engine.

[0060] During deployment and runtime, the system loads the model files, the corresponding inference engine, and the custom operator library, enabling efficient and low-latency inference of the entire network on edge devices. This solves the problem of flexibly and efficiently deploying advanced neural networks to diverse edge hardware.

[0061] 4.3 Visual Interface Implementation To provide users with intuitive results display and interaction, the visualization and interaction subsystem offers two optional implementation methods: One type is a desktop application developed based on the PyQt5 framework. This program can receive detection results from the edge subsystem in real time (via network serial port or ROS topic), and use OpenGL or similar graphics libraries to render 3D point clouds. It accurately overlays the 3D bounding boxes output by the neural network with the category labels on the point cloud, while providing interactive controls such as viewpoint control, target filtering, and result export.

[0062] Another approach is a web application developed using the Streamlit framework. This solution encapsulates the data processing and model inference backend as a service, accessible through a browser. Users can upload point cloud files or connect to real-time data streams to trigger online inference and view the 3D visualization results in real time on a webpage. This method facilitates remote access and collaboration.

[0063] In a specific indoor security monitoring embodiment, a LiDAR system is deployed in the airport lobby, and its data is sent via a local area network to an edge server where the system is deployed. The system continuously runs data preprocessing and reassembly procedures to ensure that the input point cloud frames are ordered and reliable. Each point cloud frame is fed into a multi-chain pooled neural network (loaded via ONNXRuntime and calling a custom sparse convolution operator implemented in C++) that has been fine-tuned and adapted to the indoor scene using LoRA technology for real-time inference. The inference results (such as "pedestrians," "suitcases," and their precise 3D positions and bounding boxes) are sent to a PC in the monitoring center via the network. This PC runs a visualization client developed based on PyQt5, allowing security personnel to intuitively observe all detected targets and their dynamic trajectories in a 3D scene, thereby achieving intelligent real-time area monitoring and early warning.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0065] Finally, the following points should be noted: First, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installation", "connection", and "linkage" should be interpreted broadly, and can be mechanical or electrical connections, or internal connections between two components, or direct connections. "Up", "down", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may change. Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other. In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-chain pooling neural network target detection system based on lidar, characterized in that, include: An ordered data preprocessing and real-time inference subsystem deployed at the edge, and a visualization and interaction subsystem deployed at the user side; The ordered data preprocessing and real-time inference subsystem includes: The data receiving and ordered reassembly module is connected to the LiDAR at its input end to receive the raw point cloud data stream and outputs sequentially continuous point cloud frames based on the sequence index, sliding window mechanism and timeout retransmission judgment rules embedded in the UDP data packet. A multi-chain pooling neural network detection module, whose input is connected to the output of the data receiving and ordered recombination module, is used to perform voxelization, feature encoding, and 3D target detection on the point cloud frame, and output target category and 3D bounding box information; the multi-chain pooling neural network detection module includes a multi-chain pooling neural network, which has a 3D sparse backbone network and a 2D dense backbone network with an inverted bottleneck structure, wherein at least one multi-chain sparse pooling module is provided in the 3D sparse backbone network, and at least one multi-chain dense pooling module is provided in the 2D dense backbone network; An adaptive inference deployment module, integrated with the multi-chain pooled neural network detection module, is used to provide heterogeneous inference engine support and custom acceleration operators for the multi-chain pooled neural network on edge computing devices. The visualization and interaction subsystem has its input end connected to the output end of the multi-chain pooled neural network detection module, and is used to fuse and display the target category, 3D bounding box information, and corresponding original point cloud data.

2. A three-dimensional target detection method based on lidar point clouds, characterized in that, Includes the following steps: S1. Data stream ordering and reassembly: Receive the raw point cloud data stream transmitted by the lidar via the UDP protocol, extract the sequence index from each data packet, cache it according to the sliding window rule, and trigger timeout waiting or data compensation when packet loss or sequence discontinuity is detected to obtain a point cloud frame with a coherent time sequence. S2. Voxel feature generation: The temporally coherent point cloud frames are divided into voxel grids and feature encoded to obtain voxel features; S3. Multi-chain collaborative feature extraction and target detection: The voxel features are input into a multi-chain pooling neural network; the three-dimensional sparse features are extracted through the multi-chain sparse pooling module in the three-dimensional sparse backbone network of the neural network; the three-dimensional sparse features are converted into two-dimensional dense features through the bird's-eye view feature generation layer; the two-dimensional dense features are enhanced and fused through the multi-chain dense pooling module in the two-dimensional dense backbone network; and the three-dimensional bounding box and category information of the target are output through the detection head. S4. Visualization of detection results: The three-dimensional bounding box and category information are overlaid and visualized with the original point cloud data.

3. The multi-chain pooling neural network target detection system based on lidar according to claim 1, characterized in that, The multi-chain pooling neural network includes a voxel feature encoding layer, a three-dimensional sparse backbone network, a bird's-eye view feature generation layer, a two-dimensional dense backbone network, and a detection head, which are connected in sequence.

4. The multi-chain pooling neural network target detection system based on lidar according to claim 3, characterized in that, The multi-chain sparse pooling module includes a dual-chain downsampling submodule, a sparse three-dimensional deconvolution submodule, and a first short-circuit connection connecting the input and output of the dual-chain downsampling submodule. The dual-chain downsampling submodule includes a first branch and a second branch configured in parallel; The first branch includes a large kernel sparse 3D convolutional unit and a first normalization unit connected in sequence; The second branch includes a three-dimensional submanifold convolution unit, a second normalization unit, an activation function unit, and a sparse three-dimensional max pooling unit connected in sequence; The outputs of the first branch and the second branch are superimposed with the input features transmitted through the first short-circuit connection, processed by the activation function, and then input to the sparse three-dimensional deconvolution submodule.

5. The multi-chain pooling neural network target detection system based on lidar according to claim 3, characterized in that, The multi-chain dense pooling module includes a multi-chain convolution sub-module, an average pooling sub-module, a residual convolution sub-module, a dense deconvolution sub-module, and a second short-circuit connection connecting the input of the multi-chain convolution sub-module and the output of the residual convolution sub-module. The multi-chain convolutional submodule includes a third branch and a fourth branch arranged in parallel, and both the third branch and the fourth branch contain convolutional units and normalization units; The outputs of the third and fourth branches are superimposed and processed by the activation function, and then sequentially input to the average pooling submodule, the residual convolution submodule, and the dense deconvolution submodule.

6. The multi-chain pooling neural network target detection system based on lidar according to claim 1, characterized in that, The data receiving and ordered reassembly module is configured as follows: Parse UDP packets to extract point cloud data and sequence indexes; The data is stored in the sliding window cache matrix according to the sequence index; If the expected sequence of data packets is not received within the preset timeout period, compensation will be made based on the previous frame data or an interpolation method. When the sequence index exceeds the current sliding window range, output the ordered data within the current window as a point cloud frame and initialize a new window.

7. The multi-chain pooling neural network target detection system based on lidar according to claim 1, characterized in that, The adaptive inference deployment module supports at least one of the following inference engines: ONNXRuntime, TensorRT, and CoreML; the implementation language of the custom acceleration operator includes C++, CUDA, or Metal.

8. The multi-chain pooling neural network target detection system based on lidar according to claim 1, characterized in that, The visualization and interaction subsystem is a desktop application based on the PyQt5 framework or a web application based on the Streamlit framework.

9. The multi-chain pooling neural network target detection system based on lidar according to claim 3, characterized in that, The basic unit of the inverted bottleneck structure includes, in sequence, a 1x1x1 convolutional layer, a 3x3x3 depth convolutional layer, and a 1x1x1 convolutional layer.

10. The multi-chain pooling neural network target detection system based on lidar according to claim 1, characterized in that, The multi-chain pooled neural network is trained through the following process: pre-training based on an outdoor scene point cloud dataset; and fine-tuning the parameters based on an indoor scene point cloud dataset using a low-rank adaptation technique.

Images