Point Cloud Moving Target Detection Method and System

Abstract

This invention discloses a point cloud moving target detection method and system. The method includes: S1, acquiring multiple frames of point clouds and dividing them into voxel grids; S2, calculating residual feature maps based on all voxels; S3, calculating full feature maps; S4, inputting the residual feature maps into a main branch network to obtain main branch features; S5, inputting the full feature maps into an auxiliary branch network to obtain auxiliary branch features; S6, fusing the main branch features and auxiliary branch features to obtain fused features; S7, inputting the fused features into the encoder of the main branch network for encoding, repeating S4 to S6 until all levels of encoding are completed to obtain encoding results, upsampling and decoding the encoding results to obtain predicted labels; S8, clustering all points in the current frame according to the predicted labels to obtain the detected targets. This method improves the detection accuracy of slow-moving and small-sized targets while ensuring real-time processing.

Description

Technical Field

[0001] This invention relates to the field of computer vision, and in particular to a method and system for detecting moving targets in point clouds. Background Technology

[0002] With the development of LiDAR hardware, point cloud-based target detection, due to its independence from lighting conditions and accurate distance information, has been widely applied in fields such as intelligent driving, park security, and safe production. Moving target detection, compared to traditional category segmentation, no longer focuses on the specific category of the object, but rather on moving targets that are more likely to pose security risks, making it more versatile. In intrusion detection, by segmenting moving target point clouds from the collected data, the shape and speed of intruding objects can be better perceived, thereby enabling tiered reporting and alerts.

[0003] Traditional moving target detection schemes typically use a pre-made background model to differiate with the current frame to obtain moving targets. However, this method is often affected by point drift of the non-repeating scanning laser hardware itself, artifacts of the lidar at the edges of objects, and noise caused by environmental factors. Furthermore, it is difficult to adapt in real time to natural changes in the production environment, such as vegetation growth and equipment adjustments. False alarms are easily generated, resulting in wasted manpower.

[0004] With the development of deep learning technology, 3D deep learning technology has begun to be introduced into the field of point cloud processing. The earliest 3D deep learning technologies, such as PointNet++, were limited by computational load and their receptive fields could not adapt to the data scale of large-scale monitoring, which made it difficult to apply them to security intrusion detection. In recent years, with the development of attention architectures represented by Transformer, especially the improvement of Mamba's State Space Model (SSM) through selective scanning mechanism, the computational complexity of algorithm overhead and sequence length has been compressed into a linear relationship, making it possible to implement related applications.

[0005] Existing target segmentation methods based on Mamba networks process input data through multiple sequencer methods, decoupling the original causal relationships in the SSM process. While this process provides a better global understanding, it disrupts the model's focus on local features, leading to missed detections of slow-moving and small targets. Furthermore, to reduce computational load, some methods use projection methods such as top-down views and distance maps to migrate 3D point clouds to 2D computation. However, this process compresses spatial information, making it difficult to adapt to complex indoor environments such as production workshops, and it is not compatible with current mainstream non-repeating scanning lasers.

[0006] There is currently no effective solution to the above problems in existing technologies. Summary of the Invention

[0007] To address the aforementioned issues, this invention provides a point cloud moving target detection method and system. By constructing a main branch to extract residual features to preserve local details and an auxiliary branch to extract full features to capture global motion information, and by fusing features with the auxiliary branch features after each downsampling of the main branch, the network can maintain the linear computational complexity of Mamba while taking into account both local details and global semantics. This solves the problems of missed detection of slow and small targets and poor environmental adaptability in existing technologies.

[0008] To achieve the above objectives, the present invention provides a point cloud moving target detection method, comprising: S1, acquiring multiple frames of point clouds at the current time and historical time, dividing the multiple frames of point clouds into voxels to obtain a voxel grid; S2, for each voxel in the voxel grid, extracting the first point information of the strongest point in the target frame closest to the current time, the strongest point intensity value of the most recent preset number of frames, and the point presence flag of the remaining frames, calculating the first time difference between the target frame and the current frame corresponding to the current time, concatenating the first point information, the strongest intensity value, the point presence flag, and the first time difference to obtain residual features, and concatenating the residual features of all voxels to form a residual feature map; S3, extracting the second point information of the strongest point in each frame within each voxel, calculating the second time difference between each frame and the current frame, and obtaining a full feature map based on the second point information and the second time difference corresponding to all voxels; S4. Input the residual feature map into the main branch network, and sequentially perform serialization and downsampling processing on the residual features to obtain the main branch features; S5. Input the full feature map into the auxiliary branch network, and sequentially perform downsampling, serialization, and encoding processing on the full feature map according to the downsampling result of the main branch network to obtain the auxiliary branch features; S6. After each downsampling process of the main branch network, fuse the main branch features and the auxiliary branch features to obtain the fused features; S7. Input the fused features into the encoder of the main branch network for encoding, and repeat S4 to S6 until the encoding of all levels is completed to obtain the encoding result. Sequentially upsample and decode the encoding result, and obtain the predicted label for each point in the current frame according to the decoding result; S8. Perform target clustering on all points in the current frame according to the predicted labels to obtain the detected target.

[0009] Optionally, the full feature map is downsampled sequentially based on the downsampling results of the main branch network, including: S501, obtaining the downsampling results of the main branch network after each downsampling process, wherein the downsampling results include multiple target voxels; S502, for each target voxel, extracting the full features of the eight neighboring voxels surrounding the target voxel; S503, using the full features of all target voxels and the full features of their corresponding neighboring voxels as the downsampling results of the auxiliary branch network.

[0010] Further optionally, the serialization process of the auxiliary branch network includes: S504, for each target voxel, taking the full features of the target voxel and its corresponding neighboring voxels as a set of local spatiotemporal sequences, and calculating the spatial distance from each neighboring voxel to the corresponding target voxel based on the full features; S505, in the local spatiotemporal sequence, sorting all voxels in order of spatial distance from farthest to closest, and for each frame in the voxel, sorting the full features of all frames in order of time from the current frame from oldest to newest, to obtain a feature sequence.

[0011] Further optionally, the step of acquiring the point cloud of the current time and the historical time in multiple frames includes: S101, acquiring the point cloud of the current frame corresponding to the current time; S102, selecting multiple frames within a preset time window based on the current time, and acquiring the point cloud of each frame; wherein, the sampling interval of each frame increases as the time difference between the frame and the current frame increases.

[0012] Further optionally, the step of fusing the main branch feature and the auxiliary branch feature to obtain a fused feature includes: S601, obtaining a time weight based on the auxiliary branch feature and its corresponding second time difference, and adjusting the auxiliary branch feature according to the time weight to obtain a first adjusted feature; S602, obtaining a gating signal based on the auxiliary branch feature, and adjusting the main branch feature according to the gating signal to obtain a second adjusted feature; S603, concatenating the first adjusted feature and the second adjusted feature to obtain a fused feature.

[0013] Further optionally, obtaining the predicted label for each point in the current frame based on the decoding result includes: S701, obtaining the voxel-level features corresponding to each voxel in the decoding result; S702, for each point in the current frame, calculating the offset of the point from the center coordinates of its corresponding voxel and the intensity difference with the strongest point; S703, concatenating the voxel-level features of the voxel to which the point belongs with the corresponding coordinate offset and intensity difference to obtain the point-by-point features; S704, mapping the point-by-point features of the point through a multilayer perceptron to obtain the predicted label; wherein, the predicted label includes static, movable, or dynamic features.

[0014] Further optionally, the step of performing target clustering on all points in the current frame based on the predicted labels includes: S801, taking points with predicted labels as movable and dynamic as clustering points, performing Euclidean clustering on the clustering points to obtain at least one cluster; S802. Count the total number of points in each cluster and filter out clusters with a total number of points less than a first preset threshold; S803. Count the number of moving points in each cluster, calculate the first proportion of the number of moving points relative to the total number of points in the cluster, and filter out clusters with the first proportion less than a second preset threshold; S804. Calculate the moving point density within a preset range for each moving point in each cluster, filter out moving points with a moving point density less than a third preset threshold, and then calculate the second proportion of the number of remaining moving points in the cluster relative to the current total number of points, and filter out clusters with the second proportion less than a fourth preset threshold; wherein, points whose predicted label is dynamic are moving points; S805. For the remaining clusters after filtering, mark the predicted labels of all points in the remaining clusters as dynamic, and add points within a preset distance around each remaining cluster to the corresponding remaining cluster.

[0015] Further optionally, after obtaining the detection target, the method further includes: S9, obtaining each detection target in the current frame and each detection target in the previous frame; S10, for each detection target in the current frame, selecting the detection targets in the previous frame whose cluster center distance to the detection target is less than a preset distance threshold as candidate targets; S11, performing principal component analysis on the detection targets in the current frame and the candidate targets respectively to obtain their respective principal component direction vectors, and eliminating candidate targets whose principal component direction vector similarity is less than a preset similarity threshold; S12, projecting the detection targets in the current frame and the remaining candidate targets to their respective principal component directions, calculating the mean distance from each point in the detection targets in the current frame to the nearest point in the candidate targets, and selecting the candidate target with the smallest mean distance as the matching result of the detection targets in the current frame.

[0016] On the other hand, the present invention also provides a point cloud moving target detection system, comprising: a voxel partitioning module, used to acquire multi-frame point clouds at the current time and historical time, and to partition the multi-frame point clouds into voxels to obtain a voxel grid; a residual feature map generation module, used to extract, for each voxel in the voxel grid, the first point information of the strongest point in the target frame closest to the current time, the strongest point intensity value of the most recent preset number of frames, and the point existence flag of the remaining frames, calculate the first time difference between the target frame and the current frame corresponding to the current time, and concatenate the first point information, the strongest intensity value, the point existence flag, and the first time difference to obtain a residual feature, and concatenate the residual features of all voxels to form a residual feature map; a full feature map generation module, used to extract the second point information of the strongest point in each frame within each voxel, and calculate the second time difference between each frame and the current frame, and obtain a full feature map based on the second point information and the second time difference corresponding to all voxels; and a main branch feature generation module, used to generate the residual feature map... The main branch network is input to the residual features, which are then sequentially serialized and downsampled to obtain the main branch features. An auxiliary branch feature generation module inputs the full feature map into the auxiliary branch network and, based on the downsampling results of the main branch network, sequentially downsampling, serializing, and encoding the full feature map to obtain the auxiliary branch features. A feature fusion module fuses the main branch features with the auxiliary branch features after each downsampling process in the main branch network to obtain fused features. A prediction module inputs the fused features into the encoder of the main branch network for encoding, repeating the operations of the main branch feature generation module, the auxiliary branch feature generation module, and the feature fusion module until all levels of encoding are completed to obtain the encoding result. The encoding result is then sequentially upsampled and decoded, and the predicted label for each point in the current frame is obtained based on the decoding result. A target detection module performs target clustering on all points in the current frame based on the predicted labels to obtain the detected targets.

[0017] Further optionally, the auxiliary branch feature generation module includes: a result acquisition submodule, used to acquire the downsampling result of the main branch network after each downsampling process, wherein the downsampling result includes multiple target voxels; a peripheral voxel extraction submodule, used to extract the full features of the eight neighboring voxels surrounding each target voxel; and a downsampling submodule, used to use the full features of all target voxels and the full features of their corresponding neighboring voxels as the downsampling result of the auxiliary branch network.

[0018] The above technical solution has the following beneficial effects: By constructing a main branch to extract residual features to preserve local details and an auxiliary branch to extract full features to capture global motion information, and fusing the main branch features with the auxiliary branch features after each downsampling, the network can maintain the linear computational complexity of Mamba while taking into account both local details and global semantics, thus improving the recognition accuracy of slow and small moving targets; by using progressive sampling, the sampling interval increases with the time difference from the current frame, preserving rich details near the current moment and retaining the complete motion trajectory far from the current moment, thus improving the comprehensiveness of motion feature extraction; by attenuating historical frames with the time weight of the auxiliary branch and using the auxiliary branch features as a gating signal to correct the main branch features, local fine information can effectively suppress misjudgments caused by global serialization, thus improving the accuracy of fused features; by performing multi-level screening of the total number of points, the proportion of moving points, and the density of moving point clusters, and combining principal component analysis and the mean of nearest neighbor distance for multi-frame matching and tracking, point cloud fragments caused by local object movement or occlusion are effectively eliminated, ensuring the integrity and temporal consistency of the detected targets. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the point cloud moving target detection method provided in the embodiments of the present invention; Figure 2 This is a flowchart of the auxiliary branch downsampling processing method provided in the embodiments of the present invention; Figure 3 This is a flowchart of the auxiliary branch serialization processing method provided in the embodiments of the present invention; Figure 4 This is a flowchart of the adjacent voxel frame sorting method provided in the embodiments of the present invention; Figure 5 This is a flowchart of the multi-frame point cloud acquisition method provided in the embodiments of the present invention; Figure 6 This is a flowchart of the feature fusion method provided in the embodiments of the present invention; Figure 7 This is a flowchart of the label prediction method provided in the embodiments of the present invention; Figure 8 This is a flowchart of the target clustering method provided in the embodiments of the present invention; Figure 9 This is a flowchart of the target tracking method provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the point cloud moving target detection system provided in an embodiment of the present invention; Figure 11 This is a schematic diagram of the structure of the downsampling submodule in the auxiliary branch feature generation module provided in this embodiment of the invention; Figure 12 This is a schematic diagram of the structure of the serialization submodule in the auxiliary branch feature generation module provided in this embodiment of the invention; Figure 13 This is a schematic diagram of the structure of the voxel division module provided in an embodiment of the present invention; Figure 14 This is a schematic diagram of the feature fusion module provided in an embodiment of the present invention; Figure 15 This is a schematic diagram of the prediction module provided in an embodiment of the present invention; Figure 16 This is a schematic diagram of the target detection module provided in an embodiment of the present invention; Figure 17 This is a schematic diagram of the structure of a target tracking module provided in an embodiment of the present invention.

[0021] Figure labeling: 100-Voxel segmentation module; 1001-Current frame acquisition submodule; 1002-Progressive frame acquisition submodule; 200-Residual feature map generation module; 300-Full feature map generation module; 400-Main branch feature generation module; 500-Auxiliary branch feature generation module; 5001-Result acquisition submodule; 5002-Surrounding voxel extraction submodule; 5003-Downsampling submodule; 5004-Spatial distance calculation submodule; 5005-Sorting submodule; 600-Feature fusion module; 6001-First adjustment feature generation submodule; 6002-Second adjustment feature generation submodule ; 6003 - Feature Fusion Submodule; 700 - Prediction Module; 7001 - Voxel-Level Feature Acquisition Submodule; 7002 - Difference Calculation Submodule; 7003 - Point-by-Point Feature Calculation Submodule; 7004 - Label Prediction Submodule; 800 - Target Detection Module; 8001 - Clustering Submodule; 8002 - First Filtering Submodule; 8003 - Second Filtering Submodule; 8004 - Third Filtering Submodule; 8005 - Clustering Processing Submodule; 900 - Adjacent Frame Detection Target Acquisition Module; 1000 - Candidate Target Selection Module; 1100 - Candidate Target Elimination Module; 1200 - Target Matching Module. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] To address the technical problem of missed detection of slow-moving and small targets due to the destruction of local features during serialization in existing Mamba-based point cloud moving target detection methods, this invention provides a point cloud moving target detection method. Figure 1 This is a flowchart of the point cloud moving target detection method provided in the embodiments of the present invention, such as... Figure 1 As shown, it includes: S1. Obtain multiple frames of point clouds at the current and historical times, divide the multiple frames of point clouds into voxels, and obtain a voxel mesh.

[0024] Point cloud data is acquired by lidar, which scans the external environment at a fixed frequency (e.g., 10Hz), that is, outputs a frame of point cloud data containing a large number of three-dimensional spatial points every 100ms.

[0025] The acquired multi-frame point cloud includes the current frame point cloud acquired at the current moment and historical frame point clouds acquired at several previous moments. These multi-frame point clouds together constitute the time-series data used to analyze the target's motion state. In specific implementations, an appropriate number of frames can be selected from the continuously acquired point cloud stream as input for subsequent processing, depending on the needs of the actual application scenario.

[0026] Because lidar moves or changes attitude with the carrier (such as a vehicle or gimbal), point cloud data collected at different times are located in different sensor coordinate systems. Therefore, after acquiring multiple frames of point clouds, it is necessary to transform each frame of point cloud to the same world coordinate system to ensure that the point clouds at different times are spatially comparable.

[0027] Coordinate system transformation can utilize the extrinsic calibration parameters of the lidar and the carrier's motion attitude information (such as pose data provided by the inertial measurement unit, IMU) to transform the point clouds of each frame from the sensor coordinate system at the time of acquisition to a unified world coordinate system. After spatial alignment, the point clouds of the same static object in different frames will overlap at the same spatial position, while the point clouds of dynamic objects will be located at different spatial positions in different frames, thus providing a spatial basis for subsequently distinguishing between static backgrounds and dynamic targets.

[0028] After coordinate system transformation, the superimposed point cloud space is divided into voxels. A voxel is the smallest computational unit in three-dimensional space, equivalent to the extension of pixels in a two-dimensional image into three-dimensional space.

[0029] Specifically, the three-dimensional space is uniformly divided into several small cubic grids, i.e., voxel grids, with a preset voxel size as the side length. Each voxel has a fixed spatial coordinate range to accommodate laser points falling within that range.

[0030] As an optional implementation, the preset voxel size can be set to 0.2m, which can achieve a good balance between preserving the shape features of the target such as the human body and controlling the amount of computation.

[0031] After voxel division is completed, for each laser point in each frame of the point cloud, its corresponding voxel is determined based on its three-dimensional coordinates. When multiple frames of point clouds are superimposed, each frame of point clouds shares the same set of voxel grid division, that is, point clouds acquired at different times fall into the corresponding voxels according to their respective coordinates.

[0032] S2. For each voxel in the voxel grid, extract the first point information of the strongest point in the target frame closest to the current time, the strongest point intensity value of the most recent preset number of frames, and the point existence flag of the remaining frames. Calculate the first time difference between the target frame and the current frame corresponding to the current time. Concatenate the first point information, the strongest intensity value, the point existence flag, and the first time difference to obtain the residual feature. Concatenate the residual features of all voxels to form a residual feature map.

[0033] For each voxel, its target frame needs to be determined. This refers to the frame with the laser point that is closest to the current moment in all the historical frames of that voxel.

[0034] Determine the target frame Then, from the multiple points falling on this voxel in the frame, the point with the highest reflection intensity is selected as the strongest point, and multidimensional information of this point is obtained, which may include coordinate information and intensity information, etc. As an optional implementation method, the three-dimensional coordinates of the strongest point are extracted. and intensity value As the first piece of information .

[0035] To further capture recent dynamic changes at this location, the strongest point intensity value is extracted from a predetermined number of frames closest to the current time. For example, initially, 8 frames of point cloud data are acquired, i.e. One frame was extracted at each time point ( (assuming the current time), then when the preset quantity is 3, then obtain respectively , , The strongest point intensity value at any given moment .

[0036] For the remaining, more distant frames, specific coordinates or intensities are no longer recorded; only a point presence flag indicates whether any point falls within that voxel in each frame. Continuing with the previous example, initially 8 frames of point cloud data were acquired, and the preset quantity is 3. Therefore, the last 5 frames are the remaining frames, and the point presence flags for these 5 frames are extracted. ,in .

[0037] Calculate the target frame With the current frame The first time difference between them enables the model to perceive the temporal distance of the feature information in the voxel.

[0038] As an optional implementation, the time difference is normalized, i.e., the first time difference. This eliminates scale differences between different frame intervals, facilitating neural network processing. Furthermore, for this first time difference... It can also be extended to 4D features using cosine coding and a multilayer perceptron (MLP). .

[0039] The extracted information—namely, the first point information of the strongest point in the target frame, the intensity value of the strongest point in the most recent preset number of frames, the point presence flags in the remaining frames, and the first time difference—is concatenated to form the residual feature of the voxel. As an example, the concatenated result is: The splicing result is the residual feature of each voxel.

[0040] Traverse all voxels in the voxel grid, and concatenate the residual features of each voxel to obtain a residual feature map, which preserves the spatial structure of the original voxel grid.

[0041] S3. Extract the second point information of the strongest point in each frame within each voxel, and calculate the second time difference between each frame and the current frame. Obtain the full feature map based on the second point information and the second time difference corresponding to all voxels.

[0042] For each voxel in the voxel grid, the point cloud information falling within that voxel in each frame is extracted. Since the same voxel may contain multiple points within the same frame, to reduce data redundancy and retain the most representative observation information, the point with the highest reflection intensity among all points of that voxel within that frame is selected as the strongest point of that frame, and its multidimensional information is obtained. This information can include coordinate information and intensity information, etc. As an optional implementation, the three-dimensional coordinates of the strongest point are extracted. and intensity value As the second point of information in this frame .

[0043] To provide the model with a temporal reference for each frame of data, a second temporal difference is calculated between that frame and the current frame for each frame within each voxel. Unlike panoramic features, which only calculate a single temporal difference between the target frame and the current frame, this method requires calculating the temporal difference for each frame of each voxel, forming a temporal difference sequence that corresponds one-to-one with each frame. This temporal difference sequence allows the model to distinguish which frames are closer to the current time and which are farther away, enabling differentiated feature extraction based on the freshness of the information in subsequent processing.

[0044] The extracted second-point information from each frame and its corresponding second-temporal difference are organized into a full feature map. Specifically, for each voxel, its second-point information in each frame is concatenated with its corresponding second-temporal difference to form the corresponding full feature map. The features of this voxel across all frames form a matrix with dimension [number of frames, 5]. Stacking the matrices of all voxels along the voxel index dimension yields a full feature map with dimension [number of voxels, number of frames, 5]. If a voxel in a given frame has no points, all feature vectors at that location are set to zero, and a mask matrix is ​​simultaneously recorded to mark that location as filler data.

[0045] It should be noted that the extraction of panoramic feature maps and residual feature maps can be performed in parallel in actual implementation.

[0046] S4. Input the residual feature map into the main branch network, and perform serialization and downsampling processing on the residual features in sequence to obtain the main branch features.

[0047] The generated residual feature map is used as input and fed into the main branch network, which is a standard Mamba network under U-Net.

[0048] In the main branch network, the input residual feature maps are first serialized. As an optional implementation, space-filling curves, such as Hilbert curves or Z-order curves, can be used for serialization. Space-filling curves expand all voxels in three-dimensional space into a one-dimensional sequence in a specific order. To enhance the robustness of the model, multiple serialization methods can be used randomly or alternately in different submodules, such as [Hilbert, Hilbert-trans, Z, Z-trans], where trans represents a variant curve with interchanged x and y axis priorities. It is sufficient to ensure that the input sequence is spatially locally ordered and has multiple sequences.

[0049] The purpose of serialization is to map voxels in three-dimensional space to a one-dimensional sequence while preserving spatial proximity as much as possible, that is, adjacent voxels in space are also as close as possible in the sequence.

[0050] After serialization, downsampling is performed on the sorted one-dimensional sequence. Since the sequence has already maintained spatial local order, the downsampling operation does not need to rely on computationally intensive neighborhood search methods such as K-nearest neighbors (KNN) or ball lookup, but can be performed directly on the serialization result.

[0051] The serialization result is downsampled using a CODE shifting and merging method. Specifically, the serialization process itself generates a corresponding CODE code for each voxel, and then arranges the voxels into a one-dimensional ordered sequence according to the CODE size. For example, three-dimensional Z-order encoding generates a one-dimensional integer that preserves spatial proximity by interleaving the binary bits of the coordinates (x, y, z). Spatially adjacent voxels also have similar CODE values.

[0052] During downsampling, the CODE of each voxel is right-shifted by a predetermined number of bits. Voxels with the same CODE after the right shift are grouped together. Within each group, voxel features are merged using an aggregation function (such as taking the maximum value), and then mapped to new features through a linear layer, resulting in the downsampled voxel features. For example, in downsampling 2, the downsampled features are... Simultaneously, the CODE for each group is updated. After downsampling, the number of voxels is reduced accordingly.

[0053] After serialization and downsampling, the residual feature map is converted into a main branch feature. This feature retains the key information in the original residual feature while reducing the spatial resolution through downsampling.

[0054] S5. Input the full feature map into the auxiliary branch network. Based on the downsampling results of the main branch network, perform downsampling, serialization and encoding on the full feature map in sequence to obtain the auxiliary branch features.

[0055] The generated full feature map is used as input and fed into the auxiliary branch network.

[0056] The downsampling process of the auxiliary branch network needs to be based on the downsampling results of the main branch network to ensure that the spatial resolution of the two branches remains consistent at each level, thereby ensuring a one-to-one correspondence of spatial positions during subsequent feature fusion.

[0057] After each downsampling process, the main branch network retains a set of target voxels (i.e., voxels that are still retained after downsampling). After the auxiliary branch network obtains the index of the target voxel, it extracts the full features of the target voxel and its surrounding neighboring voxels in each frame from the full feature map for each target voxel, forming a local spatiotemporal block.

[0058] After downsampling, the downsampling results of the auxiliary branches are serialized. Similar to the purpose of serialization in the main branch network, the serialization of the auxiliary branches also converts the disordered local spatiotemporal blocks into a one-dimensional sequence for subsequent encoder processing.

[0059] Since the downsampling of the auxiliary branch yields local neighborhood information centered on each target voxel, the serialization processing of the auxiliary branch needs to be performed separately for each local spatiotemporal block of the target voxel. Each local spatiotemporal block contains the complete temporal features of the target voxel itself and its surrounding neighboring voxels in each frame, forming a small local spatiotemporal cube.

[0060] After serialization, the generated one-dimensional feature sequence is encoded. The encoder uses a Mamba module based on a state-space model (SSM) to model the dependencies between spatiotemporal samples in the sequence and extract features rich in local motion information.

[0061] It should be noted that in the auxiliary branch, due to the large number of voxels initially, direct processing using SSM is computationally expensive. Therefore, a self-attention module is used for computation, employing a mask during the calculation. It can handle frames without points, or even the simplest 1D convolution.

[0062] After downsampling, serialization, and encoding, the full feature map is converted into auxiliary branch features.

[0063] S6. After each downsampling process of the main branch network, the main branch features and auxiliary branch features are fused to obtain the fused features.

[0064] In the processing flow of the main branch network, feature fusion is performed once after each downsampling process. The main branch network performs downsampling multiple times, and correspondingly, feature fusion occurs after each downsampling.

[0065] After each downsampling, the main branch obtains a set of target voxels and their corresponding main branch features. Simultaneously, the auxiliary branch extracts corresponding auxiliary branch features for the same set of target voxels based on the downsampling results of the main branch. Therefore, the fusion operation occurs after the two branches have completed feature extraction for the same set of target voxels. At this point, the features of the two branches are aligned in spatial dimension—both process the same set of voxels in the same spatial location. The difference lies in that the main branch features emphasize global contextual information, while the auxiliary branch features emphasize local motion cues.

[0066] The basic idea behind fusing main branch features with auxiliary branch features is to integrate the information from the two branches along the channel dimension. After fusion processing, the fused features are output.

[0067] S7. Input the fused features into the encoder of the main branch network for encoding. Repeat S4 to S6 until the encoding of all levels is completed to obtain the encoding result. Upsample and decode the encoding result in sequence, and obtain the predicted label of each point in the current frame based on the decoding result.

[0068] The obtained fused features are used as input to the encoder of the main branch network at the current level. The encoder encodes these features based on the state space model to further extract semantic information at this scale.

[0069] Because the main branch network adopts the U-Net architecture, it uses Convolutional Positional Encoding (CPE) to process the data, followed by Mamba processing. This network contains multiple encoding levels. After each layer of encoding is completed, the encoding result is used as a new residual feature, and serialization, downsampling, and feature fusion are repeatedly performed until all preset levels of encoding are completed. That is, the features output by each encoder layer are serialized and downsampled, then fused with the features of the corresponding level of the auxiliary branch, and the fused result is then fed into the next encoder layer.

[0070] After encoding at all levels, the encoded result is obtained. The encoded result is then sequentially upsampled and decoded to gradually restore the spatial resolution. Upsampling refers to progressively enlarging the spatial size of the feature map to restore it to the resolution of the original voxel grid. After each level of upsampling, the decoder performs feature decoding and uses a skip-connection structure to directly concatenate features from the corresponding layer of the encoder (with the same resolution) to the current layer of the decoder to compensate for the details lost during upsampling. The decoder can also use a Mamba module based on a state-space model for feature processing. After multiple levels of upsampling and decoding, the feature map is restored to the same spatial resolution as the original voxel grid.

[0071] After decoding, a voxel-level decoding result is obtained, where each voxel corresponds to a feature vector. This voxel-level feature is then projected back onto the original point cloud of the current frame, so that each original laser point obtains a corresponding predicted label, which includes three categories: static, moving, and dynamic.

[0072] S8. Based on the predicted labels, cluster all points in the current frame to obtain the detected targets.

[0073] For the predicted label of each output point, points with motion attributes (i.e., moving and dynamic points) are extracted from all points in the current frame, and these points are clustered. The purpose of clustering is to group spatially adjacent points that belong to the same object into a single point set, forming several candidate target clusters.

[0074] Specifically, Euclidean distance-based clustering can be used to cluster the extracted points, grouping points with a spatial distance less than a preset threshold into the same cluster. Through clustering, the original point set is divided into multiple independent point cloud clusters, with each cluster corresponding to a detection target.

[0075] After the clustering process described above, one or more detection targets are extracted from the point cloud of the current frame. Each detection target contains a set of point clouds belonging to the same object. Information recognition is performed on the detection target to obtain information such as its position and size. In addition, the detection targets in this frame can be matched with the detection targets in adjacent frames to achieve target tracking of the same object. Furthermore, information such as the speed and direction of movement of the detection targets can be analyzed to achieve the purpose of early warning.

[0076] As an optional implementation method, Figure 2 This is a flowchart of the auxiliary branch downsampling processing method provided in the embodiments of the present invention, as follows: Figure 2 As shown, the full feature map is downsampled sequentially based on the downsampling results of the main branch network, including: S501. Obtain the downsampling result of the main branch network after each downsampling process, wherein the downsampling result includes multiple target voxels.

[0077] The downsampling results of the main branch network after each downsampling process are obtained. Downsampling of the main branch is performed directly on the serialized result after serialization. Since the sequence already maintains spatial local order, downsampling does not rely on computationally intensive neighborhood search methods such as K-nearest neighbors or ball queries; instead, it uses a code shifting and merging method. After downsampling, the number of voxels is reduced to a fraction of its original value; these retained voxels are the target voxels.

[0078] After the auxiliary branch obtains the target voxel index, it can extract the local spatiotemporal information of the corresponding voxel and its adjacent voxels from the full feature map, thereby achieving alignment of the two branches in terms of spatial resolution.

[0079] S502. For each target voxel, extract the full features of the eight neighboring voxels surrounding the target voxel.

[0080] For each target voxel, with its voxel coordinates as the center, retrieve its 8 neighboring voxels (i.e., voxels in the three-dimensional space in the directions of up, down, left, right, front, back and diagonal) in the full feature map, and extract the full features of these 8 voxels in each frame. Together with the full features of the target voxel itself, they form a set of local spatiotemporal blocks.

[0081] S503. Use the full features of all target voxels and the full features of their corresponding adjacent voxels as the downsampling results of the auxiliary branch network.

[0082] By summing up the local spatiotemporal blocks corresponding to all target voxels, the complete downsampling result of the auxiliary branch network at the current level is constituted.

[0083] For example, the dimension of the full feature map was 800×8×T (800 voxels, 8 frames per voxel, and feature dimension T per frame). After downsampling by the auxiliary branch, the 100 target voxels retained by the downsampling of the main branch, each target voxel and its 8 neighboring voxels constitute a local neighborhood of 9 voxels. Each voxel retains 8 frames of full features, so the dimension after downsampling is 100×9×8×T.

[0084] As an optional implementation method, Figure 3 This is a flowchart of the auxiliary branch serialization processing method provided in the embodiments of the present invention, such as... Figure 3 As shown, the serialization process for the auxiliary branch network includes: S504. For each target voxel, the full features of the target voxel and its corresponding neighboring voxels are taken as a set of local spatiotemporal sequences, and the spatial distance from each neighboring voxel to the corresponding target voxel is calculated based on the full features.

[0085] For each target voxel, its own full features are merged with the extracted full features of its eight neighboring voxels to form a local spatiotemporal sequence. This sequence, centered on the target voxel, contains complete temporal information of the nine voxels across all frames.

[0086] Simultaneously, the spatial distance between the two is calculated based on the coordinates of each adjacent voxel and the coordinates of the target voxel.

[0087] S505. In the local spatiotemporal sequence, all voxels are sorted in order of spatial distance from farthest to nearest. For each voxel, the full features of all frames are sorted in order of time from the current frame to the oldest, to obtain the feature sequence.

[0088] Spatiotemporal samples in the local spatiotemporal sequence are uniformly sorted to generate a one-dimensional feature sequence. The sorting employs a composite sorting rule: First, all voxels are sorted in order of spatial distance from farthest to closest, with voxels farther from the target voxel appearing earlier in the sequence and those closer appearing later. The target voxel itself (with a spatial distance of 0) is placed last in the voxel dimension. For multiple frames within each voxel, they are arranged in order of time from the current moment, with older frames appearing earlier and newer frames appearing later. The point cloud frame within the target voxel that is closest to the current moment is placed last. If a point cloud frame exists, it is placed last. This sorting rule places the target voxel's features at the end of the entire one-dimensional sequence, with the point cloud frame closest to the current time step positioned closer to the end. Since subsequent encoders (such as Mamba) can integrate information from all preceding positions when processing the sequence, placing the target voxel's closest point cloud frame at the end of the sequence allows it to fully absorb the motion features of its eight neighbors and all historical time steps, thus enriching the target voxel with neighborhood information.

[0089] Figure 4 This is a flowchart of the adjacent voxel frame sorting method provided in an embodiment of the present invention. A schematic diagram when all voxels exist in the point cloud frame at any given time, as shown below. Figure 4 As shown, For the current moment, The time furthest from the current time. For the target voxel, distance The farthest voxel represents the point feature of the target voxel. Current moment Based on the features of the corresponding frame, when sorting the features, ( , ) is the first digit, representing the point characteristics ( , () is the last digit.

[0090] It should be noted that, Figure 4 for A schematic diagram when all voxels exist in the point cloud frame at any given time. If a point cloud frame does not exist in a certain voxel at a given time, then Figure 4 The corresponding voxel in the middle A frame at a given time is a time elapsed since the current time. In the most recent frame, when there is no point cloud frame in the target voxel, the representative point feature is the target voxel. Mid-distance from current time Features of the most recent frame.

[0091] The sequences of all target voxels are spliced ​​together to obtain the serialization result.

[0092] As an optional implementation, to explicitly emphasize the positional relationships between points, an additional pair of adjacent voxels and their relative positions to the corresponding target voxel are concatenated to each full feature, based on the serialization result. This represents the distance difference from each point to the target voxel j, while retaining the second time difference of voxel j. Then, the full set of features, after splicing the relative position features, is input into the encoder for encoding.

[0093] As an optional implementation method, Figure 5 This is a flowchart of the multi-frame point cloud acquisition method provided in the embodiments of the present invention, such as... Figure 5 As shown, multiple frames of point clouds at the current and historical times are obtained, including: S101. Obtain the point cloud corresponding to the current frame at the current moment.

[0094] During the continuous acquisition process of the lidar, each frame of point cloud data has a corresponding timestamp. The current frame refers to the point cloud frame most recent to the current moment, denoted as . time.

[0095] S102. Based on the current time, select multiple frames within a preset time window and acquire the point cloud of each frame; wherein, the sampling interval of each frame increases as the time difference between the frame and the current frame increases.

[0096] Because the lidar continuously acquires point cloud data, there are a large number of historical frames preceding the current frame. Instead of simply including all historical frames, a predetermined number of frames are selected as input from these historical frames. This selection method is based on a progressive sampling strategy, the core principle of which is: the closer the time period is to the current frame, the denser the sampling and the smaller the inter-frame interval; the farther the time period is from the current frame, the sparser the sampling and the larger the inter-frame interval.

[0097] As an optional implementation, to adapt to running intrusion detection in narrow passages, a cumulative input of approximately 2 seconds is used as the input, i.e., the preset time window is 2 seconds, and target detection calculation is performed every 400ms. Assuming the lidar scans at a frequency of 10Hz (i.e., outputting one frame every 100ms), and the preset frame selection is 8 frames, the sampling rule is as follows: the current frame is denoted as... In the recent time period (approximately 0–400 ms) close to the current frame, the sampling interval is relatively small (e.g., 100 ms) to retain a sufficiently dense number of observation points on the recent trajectory of the target's motion, enriching the local details of the object's contour. In the distant time period (approximately 400 ms–2000 ms) far from the current frame, the sampling interval gradually increases (e.g., 300 ms, 400 ms) to cover a longer motion time span, enabling the model to observe the complete motion process of the target from far to near. For example, eight frames with timestamps of 2000 ms, 1900 ms, 1800 ms, 1600 ms, 1300 ms, 900 ms, 400 ms, and 0 ms can be selected, with adjacent frame intervals of 100 ms, 100 ms, 200 ms, 300 ms, 400 ms, 500 ms, and 400 ms, respectively.

[0098] As an optional implementation method, Figure 6 This is a flowchart of the feature fusion method provided in the embodiments of the present invention, such as... Figure 6As shown, the main branch features and auxiliary branch features are fused to obtain the fused features, including: S601. Obtain the time weight based on the auxiliary branch features and their corresponding second time difference values, and adjust the auxiliary branch features according to the time weights to obtain the first adjustment feature.

[0099] Auxiliary branch characteristics of auxiliary branch networks , Using the second time difference Features of auxiliary branches Adjustments are made to obtain the first regulatory characteristic.

[0100] Specifically, the first regulatory feature Among them, time weight , To calculate time weights The linear layer bias term.

[0101] S602. Obtain the gating signal based on the auxiliary branch characteristics, and adjust the main branch characteristics based on the gating signal to obtain the second adjustment characteristic.

[0102] For the main branch characteristics output by the main branch network Using auxiliary branch features By performing gating, a second regulatory feature is obtained.

[0103] Specifically, the second regulatory feature Among them, gating signals , To calculate the gating signal The linear layer bias term.

[0104] S603. The first adjustment feature and the second adjustment feature are concatenated to obtain the fused feature.

[0105] For the first regulatory feature Second regulatory features The features are then projected onto the main branch feature dimension via a linear layer to obtain the fused features.

[0106] Specifically, fusion features , It is a linear layer bias term for feature fusion dimensionality reduction.

[0107] It should be noted that using time weighting for weighting and gating is a preferred embodiment, and can be selectively used in actual implementation. If the second time difference is not used for time weighting adjustment, the auxiliary branch feature will no longer undergo weighted attenuation in the time dimension, meaning the first adjusted feature equals the auxiliary branch feature. Similarly, if gating is not performed, the second adjusted feature equals the main branch feature.

[0108] It's also worth noting that downsampling of the serialization results is used on the main branch, rather than voxels or KNN. Therefore, the target voxel j retained after each round of downsampling in the main branch is always fixed, and the results of the auxiliary branches do not depend on the main branch. In this case, the auxiliary and main branches can operate in parallel, with the main branch waiting to acquire the corresponding stage features from the auxiliary branches during fusion. Although this consumes some space to store the results of the auxiliary branches, it helps to accelerate the computation.

[0109]

[0110] Table 1 Table 1 shows the feature channel configuration for the main branch and auxiliary branches at each network layer. (Encoder section, main branch feature channel count...) The number of auxiliary branch feature channels doubles with each level (32→64→128→256) to extract higher-level semantic information; The number of features doubles synchronously with the main branch (16→32→64). After feature fusion... The number of channels remains consistent with the main branch (32→64→128) to ensure that the fused features can be directly input into the next encoder layer. In encoder 3 (the deepest layer), the auxiliary branches no longer participate, and the main branch independently processes the global semantics. The decoder part only performs upsampling by the main branch, without involving auxiliary branches, and receives features from the corresponding layer of the encoder through a skip connection structure to recover spatial details.

[0111] As an optional implementation method, Figure 7 This is a flowchart of the label prediction method provided in the embodiments of the present invention, such as... Figure 7 As shown, the predicted label for each point in the current frame is obtained based on the decoding result, including: S701. Obtain the voxel-level features corresponding to each voxel in the decoding result.

[0112] The decoder restores the feature map to the same spatial resolution as the original voxel grid by upsampling. At this point, each voxel corresponds to a feature vector, i.e., voxel-level features.

[0113] S702. For each point in the current frame, calculate the offset of the point from the center coordinates of its corresponding voxel and the intensity difference from the strongest point.

[0114] Each voxel in the voxel mesh has a fixed spatial extent, and the coordinates of the center of this extent are the voxel center coordinates. For each originating point belonging to a voxel, there is a three-dimensional spatial offset between its coordinates and the voxel center coordinates. This offset reflects the precise location of the point within the voxel.

[0115] Simultaneously, the intensity difference between the laser reflection intensity value at this point and the intensity value of the representative point of the corresponding voxel (i.e., the point with the strongest intensity in that voxel) is calculated. This difference reflects the difference in reflectance characteristics of this point relative to the voxel representative point.

[0116] S703. The voxel-level features of the voxel to which the point belongs are concatenated with the corresponding coordinate offset and intensity difference to obtain the point-by-point features.

[0117] For each point in the current frame, calculate its point-by-point features. Point-by-point features ,in, This represents the voxel-level feature corresponding to this point. For point The corresponding voxel index, The offset of the point coordinates relative to the voxel center (divided by the voxel resolution) (normalization) The difference in strength between the point and the strongest point of the voxel.

[0118] S704. The point-by-point features of the point are mapped through a multilayer perceptron to obtain the predicted label; wherein the predicted label includes static, movable or dynamic.

[0119] Point-by-point features After mapping by a multilayer perceptron, the predicted label corresponding to the point is output. The multilayer perceptron consists of multiple fully connected layers and nonlinear activation functions, which transform and reduce the dimensionality of the point-by-point features layer by layer, and finally map the features to three output nodes, corresponding to the prediction scores of the static, movable, and dynamic categories, respectively. By selecting the category with the highest output score as the predicted label of the point, each original laser point obtains the determination result of its category.

[0120] Specifically, the MLP (Multilayer Perceptron) projection output design can be represented as follows.

[0121]

[0122]

[0123]

[0124]

[0125] in, , , These are the weight matrices for the first fully connected layer, the second fully connected layer, and the output layer, respectively. , , These are the bias vectors for the first fully connected layer, the second fully connected layer, and the output layer, respectively. The third type of predicted score (static / dynamic / interactive).

[0126] It should be noted that the above is a preferred implementation method, which can be selectively applied in actual implementation. If it is not used, there is no need to calculate the coordinate offset and intensity difference, and the voxel-level features are directly mapped and output as predicted labels through a multilayer perceptron.

[0127] As an optional implementation method, Figure 8 This is a flowchart of the target clustering method provided in the embodiments of the present invention, such as... Figure 8 As shown, target clustering is performed on all points in the current frame based on the predicted labels, including: S801. Using points predicted as movable and dynamic as cluster points, perform Euclidean clustering on the cluster points to obtain at least one cluster.

[0128] For each predicted label of the output point, points labeled as movable and dynamic are extracted as cluster points. Euclidean clustering is then performed on these cluster points to obtain one or more clusters.

[0129] S802. Count the total number of points in each cluster and filter out clusters with a total number of points less than the first preset threshold.

[0130] Count the total number of points in each cluster and filter out clusters with a total number of points less than a first preset threshold.

[0131] In lidar point cloud data, real moving targets (such as people or vehicles) are typically composed of a certain number of lidar points, and their point cloud density is related to the target's physical size, distance from the lidar, and lidar resolution. False targets, formed by noise or edge misjudgments, usually contain only a very small number of points, insufficient to constitute a valid physical entity. By setting a first preset threshold (e.g., 10 points), clusters with fewer than this threshold are considered noise and discarded, effectively filtering out these invalid clusters and reducing interference in subsequent processing.

[0132] S803. Count the number of moving points in each cluster, calculate the first proportion of the number of moving points relative to the total number of points in the cluster, and filter out clusters whose first proportion is less than the second preset threshold.

[0133] For each cluster remaining after filtering, a second level of filtering is performed: count the number of moving points (i.e. points with dynamic predicted labels) in each cluster, calculate the first proportion of the number of moving points relative to the total number of points in the cluster, and filter out clusters whose first proportion is less than a second preset threshold.

[0134] The proportion of moving points reflects the percentage of points in a cluster that truly possess motion attributes. If a cluster contains a large number of points, but most of them are movable and only a small number are moving, then the cluster is insufficient to constitute a valid moving target. By setting a second preset threshold (e.g., 50%), clusters with a moving point proportion below this threshold can be removed, effectively eliminating false clusters caused by misjudgment of static object edges or noise.

[0135] S804. Calculate the density of moving points within a preset range for each moving point in each cluster, filter out moving points whose density is less than a third preset threshold, and then calculate the second proportion of the number of remaining moving points in the cluster relative to the current total number of points. Filter out clusters whose second proportion is less than a fourth preset threshold. Among them, points whose predicted label is dynamic are moving points.

[0136] Within each remaining cluster after filtering, density screening is performed on the moving points. First, the number of moving points within a preset spatial radius of each moving point (i.e., the number of other moving points within a specified radius around that point) is calculated. If this number is less than a third preset threshold, the point is determined to be an isolated noise point and is filtered out. Then, the second proportion of the number of remaining moving points in the cluster after density screening is calculated relative to the current total number of points in the cluster (i.e., the total number of points after filtering out isolated points). If the second proportion is less than a fourth preset threshold, the entire cluster is filtered out.

[0137] By first removing isolated noise points within the clusters, and then making a secondary judgment based on the proportion of remaining moving points, unstable clusters consisting of only a small number of sparse moving points are further filtered out. The remaining clusters have higher moving point density and spatial aggregation, and are more likely to be real moving targets.

[0138] S805. For the filtered remaining clusters, mark the predicted labels of all points in the remaining clusters as dynamic, and add the points within a preset distance around each remaining cluster to the corresponding remaining cluster.

[0139] Post-processing is performed to complete each remaining cluster after filtering. First, the predicted labels of all points within the remaining clusters are uniformly and forcibly marked as dynamic, so that all points within the clusters have consistent motion attribute labels. Then, unclassified points within a preset distance range around each remaining cluster that are not yet included in any cluster are found, and these neighboring points are added to the corresponding remaining clusters to complete the missing target contours caused by sparse laser scanning or uncertainty in edge point classification.

[0140] As an optional implementation method, Figure 9 This is a flowchart of the target tracking method provided in an embodiment of the present invention, such as... Figure 9 As shown, after obtaining the target for detection, the following steps are also included: S9. Obtain each detection target in the current frame and each detection target in the previous frame.

[0141] The system acquires the detected targets obtained after target clustering in the current frame, as well as the detected targets from the previous frame's historical moments. In the point cloud moving target detection process, to achieve continuous tracking of moving targets, it is necessary to establish a correspondence between the same target in different frames over time.

[0142] S10. For each detected target in the current frame, the detected targets in the previous frame whose cluster center distance to the detected target is less than a preset distance threshold are selected as candidate targets.

[0143] In target tracking scenarios, the displacement of the same target between two adjacent frames is constrained by the continuity of physical motion, and its inter-frame displacement usually does not exceed a certain range. By setting a preset distance threshold, detected targets that are far away from the current target in the previous frame can be directly excluded, and only spatially adjacent detected targets can be retained as candidates. This can effectively narrow the matching search range, reduce the amount of subsequent matching calculations, and avoid false matching between distant targets.

[0144] S11. Perform principal component analysis on the detected target and candidate targets in the current frame to obtain their respective principal component direction vectors, and eliminate candidate targets whose principal component direction vector similarity is less than a preset similarity threshold.

[0145] Principal Component Analysis (PCA) is performed on the detected target and each candidate target retained after screening in the current frame to obtain the principal component direction vectors of their respective point cloud distributions. Candidate targets whose principal component direction vector similarity is less than a preset similarity threshold are eliminated.

[0146] Principal component analysis (PCA) is a data dimensionality reduction method that calculates the eigenvectors of the point cloud covariance matrix to obtain the three principal directions of the point cloud distribution. The first principal component direction vector represents the direction of maximum variance in the point cloud distribution, i.e., the principal orientation of the target point cloud. For the same target, its orientation change between adjacent frames is constrained by the continuity of physical motion and usually does not change drastically. If the principal component direction vectors of the target detected in the current frame differ significantly from those of a candidate target, it indicates a significant difference in their spatial poses, and they cannot be the same target. By setting a preset similarity threshold, candidate targets with principal component direction vector similarities below this threshold can be excluded, effectively avoiding target identity confusion caused by mismatches.

[0147] S12. Project the detected target and the remaining candidate targets of the current frame onto their respective principal component directions, calculate the mean distance from each point in the detected target of the current frame to the nearest point in the candidate targets, and select the candidate target with the smallest mean distance as the matching result of the detected target of the current frame.

[0148] For the current frame's detected target and each candidate target, their point clouds are projected onto their respective first principal component directions, reducing the 3D point clouds to one-dimensional space. After dimensionality reduction, for each point in the current frame's detected target, the nearest point is found among the projected points of the candidate target. The distance between the two is calculated, and the sum and average of the distances of all corresponding points are obtained to obtain the average nearest neighbor distance between the current frame's detected target and the candidate target. The smaller this distance, the higher the shape similarity between the two targets along the principal component direction, and the more likely they are to be the same target. By comparing the average nearest neighbor distances between the current frame's detected target and each candidate target, the candidate target with the smallest distance is selected as the matching result.

[0149] Based on the point cloud features of the detected target across multiple frames, the target's motion direction and speed can be determined, and different alarm levels can be set for targets of different sizes and speeds. For example, a target with a height between 1.5m and 2m may be a person, but if its speed exceeds 1.5m / s, it may be a running intrusion, triggering a level one alarm.

[0150] This invention also provides a point cloud moving target detection system. Figure 10 This is a schematic diagram of the point cloud moving target detection system provided in an embodiment of the present invention, as shown below. Figure 10 As shown, the system includes: The voxel division module 100 is used to acquire multiple frames of point clouds at the current time and historical time, and to divide the multiple frames of point clouds into voxels to obtain a voxel mesh.

[0151] Point cloud data is acquired by lidar, which scans the external environment at a fixed frequency (e.g., 10Hz), that is, outputs a frame of point cloud data containing a large number of three-dimensional spatial points every 100ms.

[0152] The acquired multi-frame point cloud includes the current frame point cloud acquired at the current moment and historical frame point clouds acquired at several previous moments. These multi-frame point clouds together constitute the time-series data used to analyze the target's motion state. In specific implementations, an appropriate number of frames can be selected from the continuously acquired point cloud stream as input for subsequent processing, depending on the needs of the actual application scenario.

[0153] Because lidar moves or changes attitude with the carrier (such as a vehicle or gimbal), point cloud data collected at different times are located in different sensor coordinate systems. Therefore, after acquiring multiple frames of point clouds, it is necessary to transform each frame of point cloud to the same world coordinate system to ensure that the point clouds at different times are spatially comparable.

[0154] Coordinate system transformation can utilize the extrinsic calibration parameters of the lidar and the carrier's motion attitude information (such as pose data provided by the inertial measurement unit, IMU) to transform the point clouds of each frame from the sensor coordinate system at the time of acquisition to a unified world coordinate system. After spatial alignment, the point clouds of the same static object in different frames will overlap at the same spatial position, while the point clouds of dynamic objects will be located at different spatial positions in different frames, thus providing a spatial basis for subsequently distinguishing between static backgrounds and dynamic targets.

[0155] After coordinate system transformation, the superimposed point cloud space is divided into voxels. A voxel is the smallest computational unit in three-dimensional space, equivalent to the extension of pixels in a two-dimensional image into three-dimensional space.

[0156] Specifically, the three-dimensional space is uniformly divided into several small cubic grids, i.e., voxel grids, with a preset voxel size as the side length. Each voxel has a fixed spatial coordinate range to accommodate laser points falling within that range.

[0157] As an optional implementation, the preset voxel size can be set to 0.2m, which can achieve a good balance between preserving the shape features of the target such as the human body and controlling the amount of computation.

[0158] After voxel division is completed, for each laser point in each frame of the point cloud, its corresponding voxel is determined based on its three-dimensional coordinates. When multiple frames of point clouds are superimposed, each frame of point clouds shares the same set of voxel grid division, that is, point clouds acquired at different times fall into the corresponding voxels according to their respective coordinates.

[0159] The residual feature map generation module 200 is used to extract the first point information of the strongest point in the target frame closest to the current time, the strongest point intensity value of the most recent preset number of frames, and the point existence flag of the remaining frames for each voxel in the voxel grid, calculate the first time difference between the target frame and the current frame corresponding to the current time, and concatenate the first point information, the strongest intensity value, the point existence flag, and the first time difference to obtain the residual feature, and concatenate the residual features of all voxels to form a residual feature map.

[0160] For each voxel, its target frame needs to be determined. This refers to the frame with the laser point that is closest to the current moment in all the historical frames of that voxel.

[0161] Determine the target frame Then, from the multiple points falling on this voxel in the frame, the point with the highest reflection intensity is selected as the strongest point, and multidimensional information of this point is obtained, which may include coordinate information and intensity information, etc. As an optional implementation method, the three-dimensional coordinates of the strongest point are extracted. and intensity value As the first piece of information .

[0162] To further capture recent dynamic changes at this location, the strongest point intensity value is extracted from a predetermined number of frames closest to the current time. For example, initially, 8 frames of point cloud data are acquired, i.e. One frame was extracted at each time point ( (assuming the current time), then when the preset quantity is 3, then obtain respectively , , The strongest point intensity value at any given moment .

[0163] For the remaining, more distant frames, specific coordinates or intensities are no longer recorded; only a point presence flag indicates whether any point falls within that voxel in each frame. Continuing with the previous example, initially 8 frames of point cloud data were acquired, and the preset quantity is 3. Therefore, the last 5 frames are the remaining frames, and the point presence flags for these 5 frames are extracted. ,in .

[0164] Calculate the target frame With the current frame The first time difference between them enables the model to perceive the temporal distance of the feature information in the voxel.

[0165] As an optional implementation, the time difference is normalized, i.e., the first time difference. This eliminates scale differences between different frame intervals, facilitating neural network processing. Furthermore, for this first time difference... It can also be extended to 4D features using cosine coding and a multilayer perceptron (MLP). .

[0166] The extracted information—namely, the first point information of the strongest point in the target frame, the intensity value of the strongest point in the most recent preset number of frames, the point presence flags in the remaining frames, and the first time difference—is concatenated to form the residual feature of the voxel. As an example, the concatenated result is: The splicing result is the residual feature of each voxel.

[0167] Traverse all voxels in the voxel grid, and concatenate the residual features of each voxel to obtain a residual feature map, which preserves the spatial structure of the original voxel grid.

[0168] The full feature map generation module 300 is used to extract the second point information of the strongest point in each frame within each voxel, and calculate the second time difference between each frame and the current frame. Based on the second point information and the second time difference corresponding to all voxels, the full feature map is obtained.

[0169] For each voxel in the voxel grid, the point cloud information falling within that voxel in each frame is extracted. Since the same voxel may contain multiple points within the same frame, to reduce data redundancy and retain the most representative observation information, the point with the highest reflection intensity among all points of that voxel within that frame is selected as the strongest point of that frame, and its multidimensional information is obtained. This information can include coordinate information and intensity information, etc. As an optional implementation, the three-dimensional coordinates of the strongest point are extracted. and intensity value As the second point of information in this frame .

[0170] To provide the model with a temporal reference for each frame of data, a second temporal difference is calculated between that frame and the current frame for each frame within each voxel. Unlike panoramic features, which only calculate a single temporal difference between the target frame and the current frame, this method requires calculating the temporal difference for each frame of each voxel, forming a temporal difference sequence that corresponds one-to-one with each frame. This temporal difference sequence allows the model to distinguish which frames are closer to the current time and which are farther away, enabling differentiated feature extraction based on the freshness of the information in subsequent processing.

[0171] The extracted second-point information from each frame and its corresponding second-temporal difference are organized into a full feature map. Specifically, for each voxel, its second-point information in each frame is concatenated with its corresponding second-temporal difference to form the corresponding full feature map. The features of this voxel across all frames form a matrix with dimension [number of frames, 5]. Stacking the matrices of all voxels along the voxel index dimension yields a full feature map with dimension [number of voxels, number of frames, 5]. If a voxel in a given frame has no points, all feature vectors at that location are set to zero, and a mask matrix is ​​simultaneously recorded to mark that location as filler data.

[0172] It should be noted that the extraction of panoramic feature maps and residual feature maps can be performed in parallel in actual implementation.

[0173] The main branch feature generation module 400 is used to input the residual feature map into the main branch network, and sequentially perform serialization and downsampling processing on the residual features to obtain the main branch features.

[0174] The generated residual feature map is used as input and fed into the main branch network, which is a standard Mamba network under U-Net.

[0175] In the main branch network, the input residual feature maps are first serialized. As an optional implementation, space-filling curves, such as Hilbert curves or Z-order curves, can be used for serialization. Space-filling curves expand all voxels in three-dimensional space into a one-dimensional sequence in a specific order. To enhance the robustness of the model, multiple serialization methods can be used randomly or alternately in different submodules, such as [Hilbert, Hilbert-trans, Z, Z-trans], where trans represents a variant curve with interchanged x and y axis priorities. It is sufficient to ensure that the input sequence is spatially locally ordered and has multiple sequences.

[0176] The purpose of serialization is to map voxels in three-dimensional space to a one-dimensional sequence while preserving spatial proximity as much as possible, that is, adjacent voxels in space are also as close as possible in the sequence.

[0177] After serialization, downsampling is performed on the sorted one-dimensional sequence. Since the sequence has already maintained spatial local order, the downsampling operation does not need to rely on computationally intensive neighborhood search methods such as K-nearest neighbors (KNN) or ball lookup, but can be performed directly on the serialization result.

[0178] The serialization result is downsampled using a CODE shifting and merging method. Specifically, the serialization process itself generates a corresponding CODE code for each voxel, and then arranges the voxels into a one-dimensional ordered sequence according to the CODE size. For example, three-dimensional Z-order encoding generates a one-dimensional integer that preserves spatial proximity by interleaving the binary bits of the coordinates (x, y, z). Spatially adjacent voxels also have similar CODE values.

[0179] During downsampling, the CODE of each voxel is right-shifted by a predetermined number of bits. Voxels with the same CODE after the right shift are grouped together. Within each group, voxel features are merged using an aggregation function (such as taking the maximum value), and then mapped to new features through a linear layer, resulting in the downsampled voxel features. For example, in downsampling 2, the downsampled features are... Simultaneously, the CODE for each group is updated. After downsampling, the number of voxels is reduced accordingly.

[0180] After serialization and downsampling, the residual feature map is converted into a main branch feature. This feature retains the key information in the original residual feature while reducing the spatial resolution through downsampling.

[0181] The auxiliary branch feature generation module 500 is used to input the full feature map into the auxiliary branch network, and perform downsampling, serialization and encoding on the full feature map according to the downsampling result of the main branch network to obtain the auxiliary branch features.

[0182] The generated full feature map is used as input and fed into the auxiliary branch network.

[0183] The downsampling process of the auxiliary branch network needs to be based on the downsampling results of the main branch network to ensure that the spatial resolution of the two branches remains consistent at each level, thereby ensuring a one-to-one correspondence of spatial positions during subsequent feature fusion.

[0184] After each downsampling process, the main branch network retains a set of target voxels (i.e., voxels that are still retained after downsampling). After the auxiliary branch network obtains the index of the target voxel, it extracts the full features of the target voxel and its surrounding neighboring voxels in each frame from the full feature map for each target voxel, forming a local spatiotemporal block.

[0185] After downsampling, the downsampling results of the auxiliary branches are serialized. Similar to the purpose of serialization in the main branch network, the serialization of the auxiliary branches also converts the disordered local spatiotemporal blocks into a one-dimensional sequence for subsequent encoder processing.

[0186] Since the downsampling of the auxiliary branch yields local neighborhood information centered on each target voxel, the serialization processing of the auxiliary branch needs to be performed separately for each local spatiotemporal block of the target voxel. Each local spatiotemporal block contains the complete temporal features of the target voxel itself and its surrounding neighboring voxels in each frame, forming a small local spatiotemporal cube.

[0187] After serialization, the generated one-dimensional feature sequence is encoded. The encoder uses a Mamba module based on a state-space model (SSM) to model the dependencies between spatiotemporal samples in the sequence and extract features rich in local motion information.

[0188] It should be noted that in the auxiliary branch, due to the large number of voxels initially, direct processing using SSM is computationally expensive. Therefore, a self-attention module is used for computation, employing a mask during the calculation. It can handle frames without points, or even the simplest 1D convolution.

[0189] After downsampling, serialization, and encoding, the full feature map is converted into auxiliary branch features.

[0190] The feature fusion module 600 is used to fuse the main branch features and auxiliary branch features after each downsampling process of the main branch network to obtain fused features.

[0191] In the processing flow of the main branch network, feature fusion is performed once after each downsampling process. The main branch network performs downsampling multiple times, and correspondingly, feature fusion occurs after each downsampling.

[0192] After each downsampling, the main branch obtains a set of target voxels and their corresponding main branch features. Simultaneously, the auxiliary branch extracts corresponding auxiliary branch features for the same set of target voxels based on the downsampling results of the main branch. Therefore, the fusion operation occurs after the two branches have completed feature extraction for the same set of target voxels. At this point, the features of the two branches are aligned in spatial dimension—both process the same set of voxels in the same spatial location. The difference lies in that the main branch features emphasize global contextual information, while the auxiliary branch features emphasize local motion cues.

[0193] The basic idea behind fusing main branch features with auxiliary branch features is to integrate the information from the two branches along the channel dimension. After fusion processing, the fused features are output.

[0194] The prediction module 700 is used to input the fused features into the encoder of the main branch network for encoding. The operation of the main branch feature generation module, the auxiliary branch feature generation module and the feature fusion module is repeated until the encoding of all levels is completed and the encoding result is obtained. The encoding result is then upsampled and decoded in sequence, and the predicted label of each point in the current frame is obtained based on the decoding result.

[0195] The obtained fused features are used as input to the encoder of the main branch network at the current level. The encoder encodes these features based on the state space model to further extract semantic information at this scale.

[0196] Because the main branch network adopts the U-Net architecture, it uses Convolutional Positional Encoding (CPE) to process the data, followed by Mamba processing. This network contains multiple encoding levels. After each layer of encoding is completed, the encoding result is used as a new residual feature, and serialization, downsampling, and feature fusion are repeatedly performed until all preset levels of encoding are completed. That is, the features output by each encoder layer are serialized and downsampled, then fused with the features of the corresponding level of the auxiliary branch, and the fused result is then fed into the next encoder layer.

[0197] After encoding at all levels, the encoded result is obtained. The encoded result is then sequentially upsampled and decoded to gradually restore the spatial resolution. Upsampling refers to progressively enlarging the spatial size of the feature map to restore it to the resolution of the original voxel grid. After each level of upsampling, the decoder performs feature decoding and uses a skip-connection structure to directly concatenate features from the corresponding layer of the encoder (with the same resolution) to the current layer of the decoder to compensate for the details lost during upsampling. The decoder can also use a Mamba module based on a state-space model for feature processing. After multiple levels of upsampling and decoding, the feature map is restored to the same spatial resolution as the original voxel grid.

[0198] After decoding, a voxel-level decoding result is obtained, where each voxel corresponds to a feature vector. This voxel-level feature is then projected back onto the original point cloud of the current frame, so that each original laser point obtains a corresponding predicted label, which includes three categories: static, moving, and dynamic.

[0199] The target detection module 800 is used to cluster all points in the current frame according to the predicted labels to obtain the detected targets.

[0200] For the predicted label of each output point, points with motion attributes (i.e., moving and dynamic points) are extracted from all points in the current frame, and these points are clustered. The purpose of clustering is to group spatially adjacent points that belong to the same object into a single point set, forming several candidate target clusters.

[0201] Specifically, Euclidean distance-based clustering can be used to cluster the extracted points, grouping points with a spatial distance less than a preset threshold into the same cluster. Through clustering, the original point set is divided into multiple independent point cloud clusters, with each cluster corresponding to a detection target.

[0202] After the clustering process described above, one or more detection targets are extracted from the point cloud of the current frame. Each detection target contains a set of point clouds belonging to the same object. Information recognition is performed on the detection target to obtain information such as its position and size. In addition, the detection targets in this frame can be matched with the detection targets in adjacent frames to achieve target tracking of the same object. Furthermore, information such as the speed and direction of movement of the detection targets can be analyzed to achieve the purpose of early warning.

[0203] As an optional implementation method, Figure 11 This is a schematic diagram of the structure of the downsampling submodule in the auxiliary branch feature generation module provided in this embodiment of the invention, as shown below. Figure 11 As shown, the auxiliary branch feature generation module 500 includes: The result acquisition submodule 5001 is used to acquire the downsampling results of the main branch network after each downsampling process, wherein the downsampling results include multiple target voxels.

[0204] The downsampling results of the main branch network after each downsampling process are obtained. Downsampling of the main branch is performed directly on the serialized result after serialization. Since the sequence already maintains spatial local order, downsampling does not rely on computationally intensive neighborhood search methods such as K-nearest neighbors or ball queries; instead, a shift-merge method is used to achieve downsampling. After downsampling, the number of voxels is reduced to a fraction of its original value; these retained voxels are the target voxels.

[0205] After the auxiliary branch obtains the target voxel index, it can extract the local spatiotemporal information of the corresponding voxel and its adjacent voxels from the full feature map, thereby achieving alignment of the two branches in terms of spatial resolution.

[0206] The peripheral voxel extraction submodule 5002 is used to extract the full features of the eight neighboring voxels surrounding each target voxel.

[0207] For each target voxel, with its voxel coordinates as the center, retrieve its 8 neighboring voxels (i.e., voxels in the three-dimensional space in the directions of up, down, left, right, front, back and diagonal) in the full feature map, and extract the full features of these 8 voxels in each frame. Together with the full features of the target voxel itself, they form a set of local spatiotemporal blocks.

[0208] The downsampling submodule 5003 is used to take the full features of all target voxels and the full features of their corresponding adjacent voxels as the downsampling results of the auxiliary branch network.

[0209] By summing up the local spatiotemporal blocks corresponding to all target voxels, the complete downsampling result of the auxiliary branch network at the current level is constituted.

[0210] For example, the dimension of the full feature map was 800×8×T (800 voxels, 8 frames per voxel, and feature dimension T per frame). After downsampling by the auxiliary branch, the 100 target voxels retained by the downsampling of the main branch, each target voxel and its 8 neighboring voxels constitute a local neighborhood of 9 voxels. Each voxel retains 8 frames of full features, so the dimension after downsampling is 100×9×8×T.

[0211] As an optional implementation method, Figure 12 This is a schematic diagram of the structure of the serialization submodule in the auxiliary branch feature generation module provided in this embodiment of the invention, as shown below. Figure 12 As shown, the auxiliary branch feature generation module 500 includes: The spatial distance calculation submodule 5004 is used to, for each target voxel, take the full features of the target voxel and its corresponding neighboring voxels as a set of local spatiotemporal sequences, and calculate the spatial distance from each neighboring voxel to the corresponding target voxel based on the full features.

[0212] For each target voxel, its own full features are merged with the extracted full features of its eight neighboring voxels to form a local spatiotemporal sequence. This sequence, centered on the target voxel, contains complete temporal information of the nine voxels across all frames.

[0213] Simultaneously, the spatial distance between the two is calculated based on the coordinates of each adjacent voxel and the coordinates of the target voxel.

[0214] The sorting submodule 5005 is used to sort all voxels in a local spatiotemporal sequence according to the spatial distance from farthest to nearth. For each voxel, the full features of all frames are sorted according to the time from the current frame to the oldest to the newest, to obtain the feature sequence.

[0215] Spatiotemporal samples in the local spatiotemporal sequence are uniformly sorted to generate a one-dimensional feature sequence. The sorting employs a composite sorting rule: First, all voxels are sorted in order of spatial distance from farthest to closest, with voxels farther from the target voxel appearing earlier in the sequence and those closer appearing later. The target voxel itself (with a spatial distance of 0) is placed last in the voxel dimension. For multiple frames within each voxel, they are arranged in order of time from the current moment, with older frames appearing earlier and newer frames appearing later. The point cloud frame within the target voxel that is closest to the current moment is placed last. If a point cloud frame exists, it is placed last. This sorting rule places the target voxel's features at the end of the entire one-dimensional sequence, with the point cloud frame closest to the current time step positioned closer to the end. Since subsequent encoders (such as Mamba) can integrate information from all preceding positions when processing the sequence, placing the target voxel's closest point cloud frame at the end of the sequence allows it to fully absorb the motion features of its eight neighbors and all historical time steps, thus enriching the target voxel with neighborhood information.

[0216] Figure 4 This is a flowchart of the adjacent voxel frame sorting method provided in an embodiment of the present invention. A schematic diagram when all voxels exist in the point cloud frame at any given time, as shown below. Figure 4 As shown, For the current moment, The time furthest from the current time. For the target voxel, distance The farthest voxel represents the point feature of the target voxel. Current moment Based on the characteristics of the corresponding frames, when sorting them, ( , ) is the first digit, representing the point characteristics ( , () is the last digit.

[0217] It should be noted that, Figure 4 for A schematic diagram when all voxels exist in the point cloud frame at any given time. If a point cloud frame does not exist in a certain voxel at a given time, then Figure 4 The corresponding voxel in the middle A frame at a given time is a time elapsed since the current time. In the most recent frame, when there is no point cloud frame in the target voxel, the representative point feature is the target voxel. Mid-distance from current time Features of the most recent frame.

[0218] The sequences of all target voxels are spliced ​​together to obtain the serialization result.

[0219] As an optional implementation, to explicitly emphasize the positional relationships between points, an additional pair of adjacent voxels and their relative positions to the corresponding target voxel are concatenated to each full feature, based on the serialization result. This represents the distance difference from each point to the target voxel j, while retaining the second time difference of voxel j. Then, the full set of features, after splicing the relative position features, is input into the encoder for encoding.

[0220] As an optional implementation method, Figure 13 This is a schematic diagram of the voxel division module provided in an embodiment of the present invention, as shown below. Figure 13 As shown, the voxel division module 100 includes: The current frame acquisition submodule 1001 is used to acquire the point cloud corresponding to the current frame at the current moment.

[0221] During the continuous acquisition process of the lidar, each frame of point cloud data has a corresponding timestamp. The current frame refers to the point cloud frame most recent to the current moment, denoted as . time.

[0222] The progressive frame acquisition submodule 1002 is used to select multiple frames within a preset time window based on the current time and acquire the point cloud of each frame; wherein, the sampling interval of each frame increases as the time difference between the frame and the current frame increases.

[0223] Because the lidar continuously acquires point cloud data, there are a large number of historical frames preceding the current frame. Instead of simply including all historical frames, a predetermined number of frames are selected as input from these historical frames. This selection method is based on a progressive sampling strategy, the core principle of which is: the closer the time period is to the current frame, the denser the sampling and the smaller the inter-frame interval; the farther the time period is from the current frame, the sparser the sampling and the larger the inter-frame interval.

[0224] As an optional implementation, to adapt to running intrusion detection in narrow passages, a cumulative input of approximately 2 seconds is used as the input, i.e., the preset time window is 2 seconds, and target detection calculation is performed every 400ms. Assuming the lidar scans at a frequency of 10Hz (i.e., outputting one frame every 100ms), and the preset frame selection is 8 frames, the sampling rule is as follows: the current frame is denoted as... In the recent time period (approximately 0–400 ms) close to the current frame, the sampling interval is relatively small (e.g., 100 ms) to retain a sufficiently dense number of observation points on the recent trajectory of the target's motion, enriching the local details of the object's contour. In the distant time period (approximately 400 ms–2000 ms) far from the current frame, the sampling interval gradually increases (e.g., 300 ms, 400 ms) to cover a longer motion time span, enabling the model to observe the complete motion process of the target from far to near. For example, eight frames with timestamps of 2000 ms, 1900 ms, 1800 ms, 1600 ms, 1300 ms, 900 ms, 400 ms, and 0 ms can be selected, with adjacent frame intervals of 100 ms, 100 ms, 200 ms, 300 ms, 400 ms, 500 ms, and 400 ms, respectively.

[0225] As an optional implementation method, Figure 14 This is a schematic diagram of the feature fusion module provided in an embodiment of the present invention, as shown below. Figure 14 As shown, the feature fusion module 600 includes: The first adjustment feature generation submodule 6001 is used to obtain the time weight based on the auxiliary branch feature and its corresponding second time difference, and adjust the auxiliary branch feature according to the time weight to obtain the first adjustment feature.

[0226] Auxiliary branch characteristics of auxiliary branch networks , Using the second time difference Features of auxiliary branches Adjustments are made to obtain the first regulatory characteristic.

[0227] Specifically, the first regulatory feature Among them, time weight , To calculate time weights The linear layer bias term.

[0228] The second adjustment feature generation submodule 6002 is used to obtain a gating signal based on the auxiliary branch feature, and adjust the main branch feature based on the gating signal to obtain the second adjustment feature.

[0229] For the main branch characteristics output by the main branch network Using auxiliary branch features By performing gating, a second regulatory feature is obtained.

[0230] Specifically, the second regulatory feature Among them, gating signals , To calculate the gating signal The linear layer bias term.

[0231] The feature fusion submodule 6003 is used to concatenate the first adjustment feature and the second adjustment feature to obtain the fused feature.

[0232] For the first regulatory feature Second regulatory features The features are then projected onto the main branch feature dimension via a linear layer to obtain the fused features.

[0233] Specifically, fusion features , It is a linear layer bias term for feature fusion dimensionality reduction.

[0234] It should be noted that using time weighting for weighting and gating is a preferred embodiment, and can be selectively used in actual implementation. If the second time difference is not used for time weighting adjustment, the auxiliary branch feature will no longer undergo weighted attenuation in the time dimension, meaning the first adjusted feature equals the auxiliary branch feature. Similarly, if gating is not performed, the second adjusted feature equals the main branch feature.

[0235] It's also worth noting that downsampling of the serialization results is used on the main branch, rather than voxels or KNN. Therefore, the target voxel j retained after each round of downsampling in the main branch is always fixed, and the results of the auxiliary branches do not depend on the main branch. In this case, the auxiliary and main branches can operate in parallel, with the main branch waiting to acquire the corresponding stage features from the auxiliary branches during fusion. Although this consumes some space to store the results of the auxiliary branches, it helps to accelerate the computation.

[0236] Table 1 shows the feature channel configuration for the main branch and auxiliary branches at each network layer. (Encoder section, main branch feature channel count...) The number of auxiliary branch feature channels doubles with each level (32→64→128→256) to extract higher-level semantic information; The number of features doubles synchronously with the main branch (16→32→64). After feature fusion... The number of channels remains consistent with the main branch (32→64→128) to ensure that the fused features can be directly input into the next encoder layer. In encoder 3 (the deepest layer), the auxiliary branches no longer participate, and the main branch independently processes the global semantics. The decoder part only performs upsampling by the main branch, without involving auxiliary branches, and receives features from the corresponding layer of the encoder through a skip connection structure to recover spatial details.

[0237] As an optional implementation method, Figure 15 This is a schematic diagram of the prediction module provided in an embodiment of the present invention, as shown below. Figure 15 As shown, the prediction module 700 includes: The voxel-level feature acquisition submodule 7001 is used to acquire the voxel-level features corresponding to each voxel in the decoding result.

[0238] The decoder restores the feature map to the same spatial resolution as the original voxel grid by upsampling. At this point, each voxel corresponds to a feature vector, i.e., voxel-level features.

[0239] The difference calculation submodule 7002 is used to calculate the offset of each point from the center coordinates of its corresponding voxel and the intensity difference from the strongest point for each point in the current frame.

[0240] Each voxel in the voxel mesh has a fixed spatial extent, and the coordinates of the center of this extent are the voxel center coordinates. For each originating point belonging to a voxel, there is a three-dimensional spatial offset between its coordinates and the voxel center coordinates. This offset reflects the precise location of the point within the voxel.

[0241] Simultaneously, the intensity difference between the laser reflection intensity value at this point and the intensity value of the representative point of the corresponding voxel (i.e., the point with the strongest intensity in that voxel) is calculated. This difference reflects the difference in reflectance characteristics of this point relative to the voxel representative point.

[0242] The point-by-point feature calculation submodule 7003 is used to concatenate the voxel-level features of the voxel to which the point belongs with the corresponding coordinate offset and intensity difference to obtain the point-by-point features.

[0243] For each point in the current frame, calculate its point-by-point features. Point-by-point features ,in, This represents the voxel-level feature corresponding to this point. For point The corresponding voxel index, The offset of the point coordinates relative to the voxel center (divided by the voxel resolution) (normalization) The difference in strength between the point and the strongest point of the voxel.

[0244] The label prediction submodule 7004 is used to map the point-by-point features of the point through a multilayer perceptron to obtain the predicted label; wherein the predicted label includes static, dynamic or dynamic.

[0245] Point-by-point features After mapping by a multilayer perceptron, the predicted label corresponding to the point is output. The multilayer perceptron consists of multiple fully connected layers and nonlinear activation functions, which transform and reduce the dimensionality of the point-by-point features layer by layer, and finally map the features to three output nodes, corresponding to the prediction scores of the static, movable, and dynamic categories, respectively. By selecting the category with the highest output score as the predicted label of the point, each original laser point obtains the determination result of its category.

[0246] Specifically, the MLP (Multilayer Perceptron) projection output design can be represented as follows.

[0247]

[0248]

[0249]

[0250]

[0251] in, , , These are the weight matrices for the first fully connected layer, the second fully connected layer, and the output layer, respectively. , , These are the bias vectors for the first fully connected layer, the second fully connected layer, and the output layer, respectively. The third type of predicted score (static / dynamic / interactive).

[0252] It should be noted that the above is a preferred implementation method, which can be selectively applied in actual implementation. If it is not used, there is no need to calculate the coordinate offset and intensity difference, and the voxel-level features are directly mapped and output as predicted labels through a multilayer perceptron.

[0253] As an optional implementation method, Figure 16 This is a schematic diagram of the target detection module provided in an embodiment of the present invention, as shown below. Figure 16 As shown, the target detection module 800 includes: The clustering submodule 8001 is used to take points with predicted labels of movable and dynamic as cluster points, perform Euclidean clustering on the cluster points, and obtain at least one cluster.

[0254] For each predicted label of the output point, points labeled as movable and dynamic are extracted as cluster points. Euclidean clustering is then performed on these cluster points to obtain one or more clusters.

[0255] The first filtering submodule 8002 is used to count the total number of points in each cluster and filter out clusters with a total number of points less than a first preset threshold.

[0256] Count the total number of points in each cluster and filter out clusters with a total number of points less than a first preset threshold.

[0257] In lidar point cloud data, real moving targets (such as people or vehicles) are typically composed of a certain number of lidar points, and their point cloud density is related to the target's physical size, distance from the lidar, and lidar resolution. False targets, formed by noise or edge misjudgments, usually contain only a very small number of points, insufficient to constitute a valid physical entity. By setting a first preset threshold (e.g., 10 points), clusters with fewer than this threshold are considered noise and discarded, effectively filtering out these invalid clusters and reducing interference in subsequent processing.

[0258] The second filtering submodule 8003 is used to count the number of moving points in each cluster, calculate the first proportion of the number of moving points relative to the total number of points in the cluster, and filter out clusters whose first proportion is less than a second preset threshold.

[0259] For each cluster remaining after filtering, a second level of filtering is performed: count the number of moving points (i.e. points with dynamic predicted labels) in each cluster, calculate the first proportion of the number of moving points relative to the total number of points in the cluster, and filter out clusters whose first proportion is less than a second preset threshold.

[0260] The proportion of moving points reflects the percentage of points in a cluster that truly possess motion attributes. If a cluster contains a large number of points, but most of them are movable and only a small number are moving, then the cluster is insufficient to constitute a valid moving target. By setting a second preset threshold (e.g., 50%), clusters with a moving point proportion below this threshold can be removed, effectively eliminating false clusters caused by misjudgment of static object edges or noise.

[0261] The third filtering submodule 8004 is used to calculate the density of moving points within a preset range in each cluster, filter out moving points whose density is less than a third preset threshold, and then calculate the second proportion of the number of remaining moving points in the cluster relative to the current total number of points, and filter out clusters whose second proportion is less than a fourth preset threshold; wherein, points whose predicted label is dynamic are moving points.

[0262] Within each remaining cluster after filtering, density screening is performed on the moving points. First, the number of moving points within a preset spatial radius of each moving point (i.e., the number of other moving points within a specified radius around that point) is calculated. If this number is less than a third preset threshold, the point is determined to be an isolated noise point and is filtered out. Then, the second proportion of the number of remaining moving points in the cluster after density screening is calculated relative to the current total number of points in the cluster (i.e., the total number of points after filtering out isolated points). If the second proportion is less than a fourth preset threshold, the entire cluster is filtered out.

[0263] By first removing isolated noise points within the clusters, and then making a secondary judgment based on the proportion of remaining moving points, unstable clusters consisting of only a small number of sparse moving points are further filtered out. The remaining clusters have higher moving point density and spatial aggregation, and are more likely to be real moving targets.

[0264] The clustering processing submodule 8005 is used to mark the predicted labels of all points in the remaining clusters as dynamic and add the points within a preset distance of each remaining cluster to the corresponding remaining cluster.

[0265] Post-processing is performed to complete each remaining cluster after filtering. First, the predicted labels of all points within the remaining clusters are uniformly and forcibly marked as dynamic, so that all points within the clusters have consistent motion attribute labels. Then, unclassified points within a preset distance range around each remaining cluster that are not yet included in any cluster are found, and these neighboring points are added to the corresponding remaining clusters to complete the missing target contours caused by sparse laser scanning or uncertainty in edge point classification.

[0266] As an optional implementation method, Figure 17 This is a schematic diagram of the structure of a target tracking module provided in an embodiment of the present invention, as shown below. Figure 17 As shown, the system also includes: The adjacent frame detection target acquisition module 900 is used to acquire each detection target in the current frame and each detection target in the previous frame.

[0267] The system acquires the detected targets obtained after target clustering in the current frame, as well as the detected targets from the previous frame's historical moments. In the point cloud moving target detection process, to achieve continuous tracking of moving targets, it is necessary to establish a correspondence between the same target in different frames over time.

[0268] The candidate target selection module 1000 is used to select, for each detected target in the current frame, a detected target whose cluster center distance from the detected target in the previous frame is less than a preset distance threshold as a candidate target.

[0269] In target tracking scenarios, the displacement of the same target between two adjacent frames is constrained by the continuity of physical motion, and its inter-frame displacement usually does not exceed a certain range. By setting a preset distance threshold, detected targets that are far away from the current target in the previous frame can be directly excluded, and only spatially adjacent detected targets can be retained as candidates. This can effectively narrow the matching search range, reduce the amount of subsequent matching calculations, and avoid false matching between distant targets.

[0270] The candidate target elimination module 1100 is used to perform principal component analysis on the detected target and candidate targets in the current frame to obtain their respective principal component direction vectors, and eliminate candidate targets whose principal component direction vector similarity is less than a preset similarity threshold.

[0271] Principal Component Analysis (PCA) is performed on the detected target and each candidate target retained after screening in the current frame to obtain the principal component direction vectors of their respective point cloud distributions. Candidate targets whose principal component direction vector similarity is less than a preset similarity threshold are eliminated.

[0272] Principal component analysis (PCA) is a data dimensionality reduction method that calculates the eigenvectors of the point cloud covariance matrix to obtain the three principal directions of the point cloud distribution. The first principal component direction vector represents the direction of maximum variance in the point cloud distribution, i.e., the principal orientation of the target point cloud. For the same target, its orientation change between adjacent frames is constrained by the continuity of physical motion and usually does not change drastically. If the principal component direction vectors of the target detected in the current frame differ significantly from those of a candidate target, it indicates a significant difference in their spatial poses, and they cannot be the same target. By setting a preset similarity threshold, candidate targets with principal component direction vector similarities below this threshold can be excluded, effectively avoiding target identity confusion caused by mismatches.

[0273] The target matching module 1200 is used to project the detected target and the remaining candidate targets of the current frame onto their respective principal component directions, calculate the mean distance from each point in the detected target of the current frame to the nearest point in the candidate targets, and select the candidate target with the smallest mean distance as the matching result of the detected target of the current frame.

[0274] For the current frame's detected target and each candidate target, their point clouds are projected onto their respective first principal component directions, reducing the 3D point clouds to one-dimensional space. After dimensionality reduction, for each point in the current frame's detected target, the nearest point is found among the projected points of the candidate target. The distance between the two is calculated, and the sum and average of the distances of all corresponding points are obtained to obtain the average nearest neighbor distance between the current frame's detected target and the candidate target. The smaller this distance, the higher the shape similarity between the two targets along the principal component direction, and the more likely they are to be the same target. By comparing the average nearest neighbor distances between the current frame's detected target and each candidate target, the candidate target with the smallest distance is selected as the matching result.

[0275] Based on the point cloud features of the detected target across multiple frames, the target's motion direction and speed can be determined, and different alarm levels can be set for targets of different sizes and speeds. For example, a target with a height between 1.5m and 2m may be a person, but if its speed exceeds 1.5m / s, it may be a running intrusion, triggering a level one alarm.

[0276] The above technical solution has the following beneficial effects: By constructing a main branch to extract residual features to preserve local details and an auxiliary branch to extract full features to capture global motion information, and fusing the main branch features with the auxiliary branch features after each downsampling, the network can maintain the linear computational complexity of Mamba while taking into account both local details and global semantics, thus improving the recognition accuracy of slow and small moving targets; by using progressive sampling, the sampling interval increases with the time difference from the current frame, preserving rich details near the current moment and retaining the complete motion trajectory far from the current moment, thus improving the comprehensiveness of motion feature extraction; by attenuating historical frames with the time weight of the auxiliary branch and using the auxiliary branch features as a gating signal to correct the main branch features, local fine information can effectively suppress misjudgments caused by global serialization, thus improving the accuracy of fused features; by performing multi-level screening of the total number of points, the proportion of moving points, and the density of moving point clusters, and combining principal component analysis and the mean of nearest neighbor distance for multi-frame matching and tracking, point cloud fragments caused by local object movement or occlusion are effectively eliminated, ensuring the integrity and temporal consistency of the detected targets.

[0277] The above-described specific embodiments of the invention further illustrate the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above content is only for specific embodiments of the invention and is not intended to limit the scope of protection of the invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for detecting moving targets in point clouds, characterized in that, include: S1. Obtain multiple frames of point clouds at the current time and historical time, and divide the multiple frames of point clouds into voxels to obtain a voxel mesh. S2. For each voxel in the voxel grid, extract the first point information of the strongest point in the target frame closest to the current time, the intensity value of the strongest point in the most recent preset number of frames, and the point existence flag of the remaining frames. Calculate the first time difference between the target frame and the current frame corresponding to the current time. Concatenate the first point information, the strongest intensity value, the point existence flag, and the first time difference to obtain the residual feature. Concatenate the residual features of all voxels to form a residual feature map. S3. Extract the second point information of the strongest point in each frame within each voxel, and calculate the second time difference between each frame and the current frame. Obtain the full feature map based on the second point information and the second time difference corresponding to all voxels. S4. Input the residual feature map into the main branch network, and sequentially perform serialization and downsampling processing on the residual features to obtain the main branch features; S5. Input the full feature map into the auxiliary branch network, and perform downsampling, serialization and encoding on the full feature map in sequence according to the downsampling result of the main branch network to obtain the auxiliary branch features; S6. After each downsampling process of the main branch network, the main branch features and auxiliary branch features are fused to obtain fused features; S7. Input the fused features into the encoder of the main branch network for encoding, repeat S4 to S6 until all levels of encoding are completed to obtain the encoding result, and then upsample and decode the encoding result in sequence to obtain the predicted label of each point in the current frame based on the decoding result. S8. Based on the predicted labels, cluster all points in the current frame to obtain the detected targets.

2. The point cloud moving target detection method according to claim 1, characterized in that, The full feature map is downsampled sequentially based on the downsampling results of the main branch network, including: S501. Obtain the downsampling result of the main branch network after each downsampling process, wherein the downsampling result includes multiple target voxels; S502. For each target voxel, extract the full features of the eight neighboring voxels surrounding the target voxel. S503. Use the full features of all target voxels and the full features of their corresponding adjacent voxels as the downsampling results of the auxiliary branch network.

3. The point cloud moving target detection method according to claim 2, characterized in that, The serialization process of the auxiliary branch network includes: S504. For each target voxel, the full features of the target voxel and its corresponding neighboring voxels are taken as a set of local spatiotemporal sequences, and the spatial distance from each neighboring voxel to the corresponding target voxel is calculated based on the full features. S505. In the local spatiotemporal sequence, all voxels are sorted in order of spatial distance from farthest to closest. For each voxel, the full features of all frames are sorted in order of time from the current frame to the oldest, to obtain a feature sequence.

4. The point cloud moving target detection method according to claim 1, characterized in that, The acquisition of multi-frame point clouds at the current and historical times includes: S101. Obtain the point cloud corresponding to the current frame at the current moment; S102. Based on the current time, select multiple frames within a preset time window and acquire the point cloud of each frame; wherein, the sampling interval of each frame increases as the time difference between the frame and the current frame increases.

5. The point cloud moving target detection method according to claim 1, characterized in that, The process of fusing the main branch features with the auxiliary branch features to obtain the fused features includes: S601. Obtain a time weight based on the auxiliary branch feature and its corresponding second time difference, and adjust the auxiliary branch feature according to the time weight to obtain a first adjustment feature; S602. Obtain a gating signal based on the auxiliary branch characteristics, and adjust the main branch characteristics based on the gating signal to obtain a second adjustment characteristic; S603. The first adjustment feature and the second adjustment feature are concatenated to obtain the fused feature.

6. The point cloud moving target detection method according to claim 1, characterized in that, The step of obtaining the predicted label for each point in the current frame based on the decoding result includes: S701. Obtain the voxel-level features corresponding to each voxel in the decoding result; S702. For each point in the current frame, calculate the offset of the point from the center coordinates of its corresponding voxel and the intensity difference from the strongest point. S703. Concatenate the voxel-level features of the voxel to which the point belongs with the corresponding coordinate offset and intensity difference to obtain the point-by-point features. S704. The point-by-point features of the point are mapped through a multilayer perceptron to obtain the predicted label; wherein the predicted label includes static, movable, or dynamic features.

7. The point cloud moving target detection method according to claim 6, characterized in that, The step of clustering all points in the current frame based on the predicted labels includes: S801. Points with predicted labels of movable and dynamic are used as cluster points, and Euclidean clustering is performed on the cluster points to obtain at least one cluster. S802. Count the total number of points in each cluster and filter out clusters with a total number of points less than a first preset threshold. S803. Count the number of moving points in each cluster, calculate the first proportion of the number of moving points relative to the total number of points in the cluster, and filter out clusters whose first proportion is less than a second preset threshold. S804. Calculate the density of moving points within a preset range for each moving point in each cluster, filter out moving points whose density is less than a third preset threshold, and then calculate the second proportion of the number of remaining moving points in the cluster relative to the current total number of points. Filter out clusters whose second proportion is less than a fourth preset threshold. Wherein, points whose predicted label is dynamic are moving points. S805. For the filtered remaining clusters, mark the predicted labels of all points in the remaining clusters as dynamic, and add the points within a preset distance around each remaining cluster to the corresponding remaining cluster.

8. The point cloud moving target detection method according to claim 7, characterized in that, After obtaining the detection target, the following is also included: S9. Obtain each detected target in the current frame and each detected target in the previous frame; S10. For each detected target in the current frame, the detected targets in the previous frame whose cluster center distance to the detected target is less than a preset distance threshold are selected as candidate targets. S11. Perform principal component analysis on the detected target and the candidate target in the current frame respectively to obtain their respective principal component direction vectors, and eliminate candidate targets whose principal component direction vector similarity is less than a preset similarity threshold. S12. Project the detected target and the remaining candidate targets of the current frame onto their respective principal component directions, calculate the mean distance from each point in the detected target of the current frame to the nearest point in the candidate targets, and select the candidate target with the smallest mean distance as the matching result of the detected target of the current frame.

9. A point cloud moving target detection system, characterized in that, include: The voxel partitioning module is used to acquire multiple frames of point clouds at the current time and historical time, and to perform voxel partitioning on the multiple frames of point clouds to obtain a voxel mesh. The residual feature map generation module is used to extract the first point information of the strongest point in the target frame closest to the current time, the intensity value of the strongest point in the most recent preset number of frames, and the point existence flag of the remaining frames for each voxel in the voxel grid; calculate the first time difference between the target frame and the current frame corresponding to the current time; concatenate the first point information, the strongest intensity value, the point existence flag, and the first time difference to obtain the residual feature; and concatenate the residual features of all voxels to form a residual feature map. The full feature map generation module is used to extract the second point information of the strongest point in each frame within each voxel, and calculate the second time difference between each frame and the current frame. Based on the second point information and the second time difference corresponding to all voxels, the full feature map is obtained. The main branch feature generation module is used to input the residual feature map into the main branch network, and sequentially perform serialization and downsampling processing on the residual features to obtain the main branch features. The auxiliary branch feature generation module is used to input the full feature map into the auxiliary branch network, and perform downsampling, serialization and encoding on the full feature map in sequence according to the downsampling result of the main branch network to obtain the auxiliary branch features; The feature fusion module is used to fuse the main branch features and auxiliary branch features after each downsampling process of the main branch network to obtain fused features; The prediction module is used to input the fused features into the encoder of the main branch network for encoding, repeat the operations of the main branch feature generation module, the auxiliary branch feature generation module and the feature fusion module until the encoding of all levels is completed, and obtain the encoding result. The encoding result is upsampled and decoded in sequence, and the predicted label of each point in the current frame is obtained based on the decoding result. The target detection module is used to cluster all points in the current frame based on the predicted labels to obtain the detected targets.

10. The point cloud moving target detection system according to claim 9, characterized in that, The auxiliary branch feature generation module includes: The result acquisition submodule is used to acquire the downsampling result of the main branch network after each downsampling process, wherein the downsampling result includes multiple target voxels; The peripheral voxel extraction submodule is used to extract the full features of the eight neighboring voxels surrounding each target voxel. The downsampling submodule is used to take the full features of all target voxels and the full features of their corresponding adjacent voxels as the downsampling results of the auxiliary branch network.

Images