Dynamic scene target perception method based on three-dimensional point cloud and state space model

Abstract

The application discloses a kind of dynamic scene target perception methods based on three-dimensional point cloud and state space model, comprising: extracting key point set in the three-dimensional point cloud data of dynamic scene;Point-level feature extraction is carried out to key point set, and the point-level feature of each key point and initial semantic probability distribution are obtained;Two-dimensional BEV feature map is obtained by semantic enhancement;In semantic prior weight vector, the weight coefficient corresponding to dynamic target class is higher than the weight coefficient corresponding to static background class;Obtain space-time enhanced BEV feature map;Obtain dynamic occupancy grid prediction result and two-dimensional motion information prediction result.The above scheme introduces the differential weighting aggregation mechanism based on semantic prior weight vector in BEV projection stage, so that the weight coefficient of dynamic target class is higher than that of static background class in the aggregation process, effectively inhibits the masking effect of static background features on dynamic small target features.

Description

Technical Field

[0001] This invention relates to the field of 3D point cloud data processing technology, and in particular to a dynamic scene target perception method based on 3D point cloud and state space model. Background Technology

[0002] In autonomous driving, autonomous mobile robots, and intelligent transportation systems, 3D point cloud perception is a key technological link in achieving environmental understanding and safety decision-making. In recent years, this technological field has evolved from early single-frame point cloud target detection to a comprehensive approach integrating multi-view point cloud feature extraction, unified Bird's Eye View (BEV) representation, cross-frame spatiotemporal modeling, and joint prediction of dynamic occupancy and motion information. Among these, BEV projection, by compressing and mapping 3D point cloud features onto a 2D bird's eye view plane, establishes a spatial representation compatible with autonomous driving planning and control modules, and has become an important bridge connecting 3D perception and downstream decision-making tasks. To further improve the continuity of perception in dynamic scenes, researchers have successively introduced architectures such as recurrent neural networks, temporal convolutional networks, and state-space models to perform temporal evolution modeling of BEV feature sequences, capturing the motion patterns and occupancy state changes of dynamic targets. These technological approaches have made progress in perception accuracy and temporal modeling capabilities, forming a relatively complete 3D dynamic scene perception framework.

[0003] However, in the process of projecting 3D point cloud features to 2D BEV, existing methods typically perform fixed pooling or max pooling aggregation operations on multiple key points falling within the same BEV grid space. This aggregation method treats all key points within the grid as having equal contributions. However, since the number of key points in static background categories (such as roads, buildings, and vegetation) usually far exceeds the number of key points in dynamic small targets (such as distant pedestrians and cyclists), under the equal aggregation mechanism, the point cloud feature contribution of dynamic small targets is submerged by static background features. This results in a severe weakening or even complete loss of information representation of dynamic small targets in the 2D BEV feature map, leading to a decrease in the perception sensitivity of downstream dynamic occupancy prediction and motion estimation tasks for small targets, thus restricting the safety perception capability of autonomous driving systems in complex dynamic scenes. Summary of the Invention

[0004] This invention provides a dynamic scene target perception method based on a 3D point cloud and state space model to solve the above-mentioned problems.

[0005] In a first aspect, embodiments of the present invention provide a dynamic scene target perception method based on a 3D point cloud and a state space model. The method includes: extracting a set of key points from the 3D point cloud data of a dynamic scene; extracting point-level features from the set of key points to obtain point-level features and an initial semantic probability distribution for each key point; weighting and aggregating the point-level features of the key points based on the initial semantic probability distribution and semantic prior weight vector of each key point within the same BEV grid to obtain a semantically enhanced 2D BEV feature map; wherein the semantic prior weight vector is differentiated based on semantic categories, and the weight coefficient corresponding to the dynamic target category is higher than the weight coefficient corresponding to the static background category in the semantic prior weight vector; performing cross-frame temporal feature enhancement on the 2D BEV feature map using a state space model to obtain a spatiotemporally enhanced BEV feature map; and decoding the spatiotemporally enhanced BEV feature map to obtain a dynamic grid occupancy prediction result and a 2D motion information prediction result.

[0006] The above scheme introduces a differentiated weighted aggregation mechanism based on semantic prior weight vectors in the BEV projection stage, which enables dynamic target categories to obtain higher weight coefficients than static background categories during the aggregation process. This effectively suppresses the masking effect of static background features on dynamic small target features, improves the feature preservation ability of dynamic targets in two-dimensional bird's-eye view representation, and thus enhances the integrity and reliability of dynamic scene target perception results.

[0007] Further, the step of weighted aggregation of the point-level features of key points based on the initial semantic probability distribution and semantic prior weight vector of each key point within the same BEV grid to obtain a semantically enhanced two-dimensional BEV feature map includes: for each key point within the same BEV grid, performing an inner product operation between the initial semantic probability distribution of the key point and the semantic prior weight vector to obtain a semantic weight scalar of the key point; normalizing the semantic weight scalar of each key point within the same BEV grid to obtain the aggregated weight of each key point within the BEV grid; and using the aggregated weight to weighted aggregate the point-level features of each key point within the same BEV grid to obtain a semantically enhanced two-dimensional BEV feature map.

[0008] The above scheme generates a semantic weight scalar by performing an inner product operation between the initial semantic probability distribution of key points and the semantic prior weight vector, and then performs normalization processing within the same BEV grid to obtain aggregate weights. This ensures that the aggregate contribution of each key point can reflect the category differentiation guidance of the semantic prior while maintaining a relatively balanced proportion within the grid. As a result, when performing weighted aggregation, it can more accurately suppress the masking of dynamic target features by static background features, and improve the accuracy and stability of retaining dynamic small target information in the semantically enhanced 2D BEV feature map.

[0009] Furthermore, the step of normalizing the semantic weight scalars of each key point within the same BEV grid to obtain the aggregate weight of each key point within the BEV grid includes: normalizing the semantic weight scalars of each key point in the set of points to be aggregated to obtain the aggregate weight of each key point in the set of points to be aggregated within the corresponding BEV grid; wherein, the set of points to be aggregated is a set of key points that fall into the same BEV grid but are located at different height layers. The step of using the aggregation weights to perform weighted aggregation of the point-level features of each key point within the same BEV grid to obtain a semantically enhanced two-dimensional BEV feature map includes: using the aggregation weights to perform weighted compression and feature aggregation on the point-level features of each key point in the set of points to be aggregated in the height direction to obtain a two-dimensional BEV feature representation for the corresponding BEV grid position; and obtaining a semantically enhanced two-dimensional BEV feature map based on the two-dimensional BEV feature representations for each BEV grid position.

[0010] The above scheme forms a set of key points that fall within the same BEV grid but are located at different height layers into a set of points to be aggregated. Then, it performs weighted compression and feature aggregation based on semantically guided aggregation weights in the height direction. This allows the multi-height-layer point cloud information in 3D space to be compressed in the height dimension based on the differentiated contribution of the semantic importance of each key point when projected into a 2D bird's-eye view. This effectively preserves the dynamic target features distributed at different height layers and suppresses static background interference during the 3D to 2D dimensionality reduction aggregation process, thereby improving the spatial restoration capability and semantic fidelity of 2D BEV feature representation for 3D dynamic scenes.

[0011] Further, the step of extracting point-level features from the keypoint set to obtain the point-level features of each keypoint and the initial semantic probability distribution includes: for each keypoint in the keypoint set, extracting local geometric detail features based on the geometric structure of the local neighborhood of each keypoint; for each keypoint in the keypoint set, extracting global spatial context features based on the spatial dependencies between each keypoint; fusing the local geometric detail features and the global spatial context features to obtain the point-level features of each keypoint; and inputting the point-level features into a classification head to obtain the initial semantic probability distribution of each keypoint.

[0012] The above scheme extracts local geometric detail features and global spatial context features for the same set of key points and then fuses them. This allows the point-level features of each key point to simultaneously carry the geometric structure information of the local neighborhood and the spatial dependencies of the long distance, thereby providing a more complete point-level representation basis for subsequent semantic probability prediction and improving the accuracy and reliability of the initial semantic probability distribution estimation.

[0013] Furthermore, the step of fusing the local geometric detail features and the global spatial context features to obtain point-level features for each key point includes: concatenating the local geometric detail features and the global spatial context features along the channel dimension to obtain concatenated features; inputting the concatenated features into a gating network to obtain a fusion weight matrix; and dynamically weighting and fusing the local geometric detail features and the global spatial context features based on the fusion weight matrix to obtain point-level features for each key point.

[0014] The above scheme generates a fusion weight matrix by concatenating local geometric detail features and global spatial context features along the channel dimension and inputting them into a gating network. Based on this fusion weight matrix, dynamic weighted fusion of the two types of features is performed, enabling each key point to adaptively adjust the contribution ratio of local and global information according to the complexity of its local neighborhood geometry and the requirements of global spatial dependence. This enhances the ability to represent details in areas with drastic local geometric changes and improves the ability to perceive context in areas where global structural associations dominate, thereby obtaining point-level features that combine local detail representation and global scene understanding.

[0015] Further, the step of using a state-space model to perform cross-frame temporal feature enhancement on the two-dimensional BEV feature map to obtain a spatiotemporally enhanced BEV feature map includes: sequentially unfolding the two-dimensional BEV feature map along multiple preset directions to obtain multiple one-dimensional sequences; wherein the one-dimensional sequences have an index mapping relationship with the original spatial position of the two-dimensional BEV feature map; inputting each of the one-dimensional sequences into the state-space model and recursively updating it in combination with the historical hidden state of the corresponding direction to obtain the temporal enhancement sequence corresponding to each direction; rearranging each of the temporal enhancement sequences into a two-dimensional feature matrix according to the inverse index mapping relationship corresponding to each of the sequential unfolding rules; and fusing the two-dimensional feature matrices obtained by rearranging in multiple directions to obtain the spatiotemporally enhanced BEV feature map.

[0016] The above scheme serializes and expands the two-dimensional BEV feature map along multiple preset directions, and then inputs it into the state space model to recursively update it in combination with the historical hidden states. After reverse rearrangement and fusion, the spatiotemporally enhanced BEV feature map is obtained. This allows the two-dimensional spatial structure information to be fully preserved during the multi-directional serialization process. At the same time, the linear recursiveness of the state space model is used to achieve effective fusion of cross-frame historical information and long-range dependency modeling with low computational overhead. Thus, the computational efficiency of continuous temporal feature enhancement and dynamic information capture capability are improved while maintaining the integrity of spatial relationship expression.

[0017] Furthermore, the step of fusing the two-dimensional feature matrix obtained by multi-directional rearrangement to obtain a spatiotemporally enhanced BEV feature map includes: summing the two-dimensional feature matrix obtained by multi-directional rearrangement element by element according to the corresponding spatial index position to obtain the spatiotemporally enhanced BEV feature map.

[0018] The above scheme sums the two-dimensional feature matrix obtained by multi-directional rearrangement element by element at the corresponding spatial index position, so that the temporal information enhanced by temporal sequence in each direction can be directly superimposed and fused at the same spatial position. Thus, a simple fusion mechanism is used to integrate the temporal evolution information captured by multi-directional serialization unfolding under different spatial relationships, and obtain a unified BEV feature expression with multi-directional spatiotemporal perception capability.

[0019] Furthermore, the step of serializing and unfolding the two-dimensional BEV feature map along multiple preset directions to obtain multiple one-dimensional sequences includes: serializing and unfolding the two-dimensional BEV feature map along four preset directions to obtain four one-dimensional sequences; wherein, the preset directions include a row traversal direction from top left to bottom right, a row traversal direction from bottom right to top left, a column traversal direction from top right to bottom left, and a column traversal direction from bottom left to top right.

[0020] The above scheme serializes the two-dimensional BEV feature map along four directions: top left to bottom right row, bottom right to top left row, top right to bottom left column, and bottom left to top right column. This allows the positional relationships in the two-dimensional space to be fully encoded in multiple one-dimensional sequences with different scanning orders, thereby alleviating the problem of insufficient spatial structure expression by single-direction serialization and improving the ability to preserve the topological relationships in the two-dimensional space during subsequent temporal modeling.

[0021] Furthermore, the extraction of key point sets from the 3D point cloud data of the dynamic scene includes: acquiring raw point cloud data collected by multi-view LiDAR around the vehicle; uniformly transforming the point clouds from each viewpoint to the vehicle coordinate system using a pre-calibrated extrinsic transformation matrix and performing spatial stitching to obtain a panoramic 3D point cloud set; downsampling the panoramic 3D point cloud set according to regular 3D grid units; wherein the unit size of the regular 3D grid unit in the horizontal plane is consistent with the planar grid size used in BEV projection; for multiple points falling within the same 3D grid unit, extracting representative points within the 3D grid unit to obtain a key point set composed of the representative points.

[0022] The above solution transforms the raw point cloud collected by multi-view LiDAR around the vehicle into the vehicle coordinate system for spatial stitching. Then, it performs downsampling on the panoramic 3D point cloud set according to the 3D grid units with the same grid size as the BEV projection plane to extract representative points. This ensures that the downsampled key point set is aligned with the subsequent BEV projection in terms of horizontal spatial division. This reduces the scale of point cloud data and the computational cost of subsequent feature extraction, while avoiding key point mapping misalignment or information loss caused by inconsistent spatial division. It also improves the accuracy of spatial correspondence and data processing efficiency in the process of converting from 3D point cloud extraction to 2D BEV projection.

[0023] Furthermore, the method further includes: the following steps performed by a 3D spatiotemporal perception network: extracting a set of key points from the 3D point cloud data of a dynamic scene; extracting point-level features from the set of key points to obtain the point-level features and initial semantic probability distribution of each key point; weighting and aggregating the point-level features of the key points based on the initial semantic probability distribution and semantic prior weight vector of each key point within the same BEV grid to obtain a semantically enhanced 2D BEV feature map; wherein the semantic prior weight vector is differentially set based on semantic categories, and the weight coefficient corresponding to the dynamic target category is higher than the weight coefficient corresponding to the static background category in the semantic prior weight vector; performing cross-frame temporal feature enhancement on the 2D BEV feature map using a state space model to obtain a spatiotemporally enhanced BEV feature map; and decoding the spatiotemporally enhanced BEV feature map to obtain dynamic grid occupancy prediction results and 2D motion information prediction results. The training method for the three-dimensional spatiotemporal perception network includes: The 3D spatiotemporal perception network is jointly trained end-to-end using a multi-task joint loss function; wherein the multi-task joint loss function includes: Semantic classification error is used to quantify the classification deviation between the initial semantic probability distribution and the corresponding true semantic label; Dynamic occupancy prediction error is used to quantify the prediction deviation between the dynamic occupancy grid prediction result and the corresponding true binary occupancy label. Motion information regression error is used to quantify the regression deviation between the two-dimensional motion information prediction result and the corresponding real two-dimensional motion vector at the grid position where the actual occupation label is positive.

[0024] Other features and advantages of the present invention will be set forth in the following description. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the dynamic scene target perception method based on a 3D point cloud and state space model provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of downsampling provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the serialization and unfolding of a certain BEV feature map provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the architecture of a three-dimensional spatiotemporal sensing network provided in an embodiment of the present invention. Detailed Implementation

[0026] The present application will now be described in further detail with reference to the accompanying drawings.

[0027] Currently, in related technologies, methods such as BEVFormer (a pure vision perception method for autonomous driving, whose core function is to convert 2D images captured by multiple cameras into a 3D bird's-eye view of the vehicle's surroundings) embody the research trend of using a unified BEV representation to carry multi-task perception and spatiotemporal information modeling; PointNeXt (a point cloud deep learning model based on PointNet++, which improves performance in 3D classification and segmentation tasks by optimizing training strategies and introducing modern network architectures, while maintaining fast inference speed) represents the continued development of local neighborhood and hierarchical aggregation point cloud feature extraction routes; PointCloudMamba (PCM, the first efficient architecture to successfully apply Mamba (state space model, SSM) to 3D point cloud analysis, achieving global modeling with linear computational complexity (O(N))) and OccMamba (the first semantic occupancy prediction network based on the Mamba state space model, aiming to efficiently model the geometric and semantic occupancy state of each voxel in the 3D environment) further demonstrate that using point cloud or raster serialization combined with state space models for low-complexity long-range dependency modeling has become a new research direction.

[0028] Furthermore, STCOcc (Sparse Spatial-Temporal Cascade Renovation for 3DOccupancy and Scene Flow Prediction, a joint prediction method for 3D occupancy and scene flow in autonomous driving perception) combines 3D occupancy and scene flow prediction and models them within a spatiotemporal cascade framework; FSF-Net, on the other hand, performs 4D occupancy prediction based on coarse-grained BEV scene flow and introduces VQ-Mamba to enhance spatiotemporal structural feature representation. These works demonstrate that the approaches surrounding "BEV representation and dynamic occupancy / motion prediction" and "state-space model and spatiotemporal feature enhancement" are continuously advancing.

[0029] Although the above-mentioned technical approaches have made progress in terms of perception accuracy and temporal modeling capabilities, the following problems still exist: In the process of compressing and projecting 3D features to 2D BEV features, the existing fixed pooling or max pooling methods are difficult to differentiate and aggregate key points in the same cylindrical space based on the semantic importance of the points, which makes it easy for dynamic small target features such as distant pedestrians and cyclists to be masked by static backgrounds.

[0030] In view of this, embodiments of the present invention provide a dynamic scene target perception method based on a 3D point cloud and state space model. This method introduces a differentiated weighted aggregation mechanism based on semantic prior weight vectors in the BEV projection stage, so that the dynamic target category obtains a higher weight coefficient than the static background category during the aggregation process. This effectively suppresses the masking effect of static background features on dynamic small target features, improves the feature preservation ability of dynamic targets in the 2D bird's-eye view representation, and thus enhances the integrity and reliability of the dynamic scene target perception results.

[0031] It is understood that the dynamic scene target perception method based on three-dimensional point cloud and state space model provided in the embodiments of the present invention is mainly used for environmental perception and understanding tasks in scenarios such as autonomous driving, autonomous mobile robots and intelligent transportation systems. It can collect three-dimensional point cloud data by deploying multi-view LiDAR around the vehicle, and perform stable perception and continuous representation of targets in dynamic scenes in the vehicle coordinate system, thereby making safety decisions and path planning based on the perception results.

[0032] like Figure 1 As shown, this embodiment of the invention provides a dynamic scene target perception method based on a 3D point cloud and state space model, including: Step S110: Extract the set of key points from the 3D point cloud data of the dynamic scene.

[0033] It is understandable that the raw point cloud data is collected by multi-view LiDAR deployed around the vehicle. The data volume is large and the spatial distribution is dense. Directly performing subsequent processing on all raw points would result in excessive computational complexity and low processing efficiency. Therefore, step S110 above extracts a set of key points from the 3D point cloud data of the dynamic scene. One possible extraction scheme is as follows: Optionally, step S110 includes: acquiring raw point cloud data collected by multi-view LiDAR around the vehicle; uniformly transforming the point clouds from each viewpoint to the vehicle coordinate system using a pre-calibrated extrinsic transformation matrix and performing spatial stitching to obtain a panoramic 3D point cloud set; downsampling the panoramic 3D point cloud set according to regular 3D grid units; wherein the unit size of the regular 3D grid unit in the horizontal plane is consistent with the planar grid size used in BEV projection; for multiple points falling into the same 3D grid unit, extracting representative points in the 3D grid unit to obtain a key point set composed of representative points.

[0034] Using the pre-calibrated extrinsic transformation matrices of each lidar system to the vehicle coordinate system, the point clouds from each viewpoint are uniformly transformed to the vehicle coordinate system. The aforementioned extrinsic transformation matrices describe the rotation and translation relationships of each lidar coordinate system relative to the vehicle coordinate system. Coordinate alignment is achieved through rigid transformation, and then the multi-view point clouds are spatially stitched together under a unified spatial reference frame to form a panoramic 3D point cloud set covering the vehicle's surrounding environment.

[0035] To reduce the computational complexity of subsequent 3D feature extraction and ensure consistency in spatial division between the downsampling process and the subsequent bird's-eye view projection, a 3D raster downsampling method can be used to preprocess the stitched panoramic point cloud. Specifically, the space around the vehicle can be divided into regular 3D raster units. During the division of these regular 3D raster units, the unit size in the horizontal plane (i.e., the XY plane) is consistent with the planar raster size used in the subsequent BEV projection, while the Z-direction is divided into several layers according to a preset height range. Maintaining consistency between the unit size in the XY plane and the planar raster size used in the subsequent BEV projection ensures a one-to-one spatial mapping relationship between the 3D downsampling raster and the 2D BEV raster in the horizontal plane. This allows key points falling within the same 3D raster unit to be accurately mapped to their corresponding 2D raster positions during subsequent BEV projection, avoiding key point mapping misalignment, cross-raster information aliasing, or spatial resolution loss due to size mismatch between the 3D and BEV rasteresses. This ensures the accuracy of the spatial correspondence during the conversion from 3D point cloud to 2D bird's-eye view projection.

[0036] Representative points can be represented by the geometric center of all points within the grid, or a representative point can be selected from the grid using preset rules, while retaining the distribution information of the points within the grid as auxiliary input for subsequent feature extraction. Through the above processing, the original dense point cloud is compressed into a set of key points that are fewer in number and more regularly distributed.

[0037] like Figure 2 As shown, after downsampling, the set of key points is obtained. Where N is the number of keypoints after downsampling. Each keypoint retains its corresponding 3D spatial position and establishes an index relationship with its corresponding 3D raster cell.

[0038] Step S120: Extract point-level features from the keypoint set to obtain the point-level features of each keypoint and the initial semantic probability distribution.

[0039] It is understandable that in technologies related to dynamic scene perception based on 3D point clouds, the point-level feature extraction stage typically uses a single network structure to process the set of key points. This type of single-path feature extraction method focuses on modeling the fine geometric structure around key points through local neighborhood aggregation mechanisms to capture details such as edges, contours, and local shape changes, or on capturing the overall scene layout and spatial context through global attention mechanisms or long-distance dependency modeling. However, due to structural differences between local geometric detail modeling and global spatial context modeling in terms of receptive field range, feature abstraction level, and information interaction methods, a single network structure struggles to effectively coordinate local fine representation and global relational reasoning simultaneously in the same feature space. This results in the extracted point-level features lacking sufficient detail resolution in areas of significant local geometric changes, thus hindering the accurate representation of targets at different scales in complex scenes during subsequent semantic probability estimation and BEV projection stages. Based on this, the embodiments of the present invention provide the following solution: Optionally, step S120 includes: for each key point in the key point set, extracting local geometric detail features of each key point based on the geometric structure of the local neighborhood of the key point; for each key point in the key point set, extracting global spatial context features of each key point based on the spatial dependency relationship between each key point; fusing local geometric detail features and global spatial context features to obtain point-level features of each key point; and inputting the point-level features into the classification head to obtain the initial semantic probability distribution of each key point.

[0040] To simultaneously characterize both local geometric details and global spatial context information in point cloud data, the keypoint set is input into two parallel feature extraction branches for local and global feature extraction, respectively. Local geometric details reflect target edges, local shape changes, and fine-grained geometric structures, while global spatial context features reflect long-distance spatial dependencies and the overall scene layout. The two branches can process the same keypoint set in parallel.

[0041] In the local feature extraction branch, the PointNeXt point cloud encoding structure can be used to model the features of the local neighborhood of each keypoint. The PointNeXt encoding structure is mainly based on neighborhood grouping, local aggregation, and hierarchical feature extraction to extract the local geometric details around the keypoint. In addition to the PointNeXt point cloud encoding structure mentioned above, the local feature extraction branch can also use point cloud encoding structures with local neighborhood modeling capabilities, such as PointNet++, DGCNN, KPConv, and PointMLP, to extract the local geometric detail features of each keypoint based on the geometric structure of its local neighborhood.

[0042] Taking the PointNeXt point cloud encoding structure as an example, the main architecture and working principle of the above local feature extraction branch are introduced: PointNeXt's point cloud encoding structure employs a hierarchical network architecture, achieving multi-scale local geometric modeling through layer-by-layer downsampling and feature abstraction. Its backbone network consists of multiple cascaded encoding layers. Each layer first performs neighborhood grouping on the input keypoint set, selecting a local neighborhood point set for each keypoint based on the spatial distance between points. Then, local aggregation is performed within each neighborhood, encoding the coordinate differences and feature responses of neighboring points through an aggregation function to generate neighborhood features characterizing the local geometric structure. Next, a hierarchical feature extraction module transforms and maps these neighborhood features to obtain the output feature representation for that layer. As the layers deepen, the network repeatedly performs neighborhood grouping, local aggregation, and feature extraction on the downsampled sparse keypoint set, gradually expanding the effective receptive field and extracting higher-order geometric abstract features, ultimately outputting hierarchical local geometric detail features for each keypoint.

[0043] The working principle of the PointNeXt point cloud encoding structure includes: characterizing geometric details by utilizing the spatial distribution and feature association of points within a local neighborhood. For each keypoint in the input keypoint set, the PointNeXt point cloud encoding structure searches for neighboring points to form a local support region. By calculating the spatial offset and feature differences of neighboring points relative to the center point, it captures fine-grained geometric attributes such as edge orientation, surface curvature, and local shape changes. During hierarchical propagation, downsampling operations retain representative keypoints, allowing subsequent levels to continue neighborhood grouping and aggregation based on abstract features within a larger spatial range. This achieves hierarchical representation of multi-scale geometric structures without sacrificing local details. The PointNeXt point cloud encoding structure maintains the invariance of point cloud arrangement through differentiable operations within the local neighborhood, ensuring that the extracted local geometric details accurately reflect the target edges, contours, and local shape changes.

[0044] In the global feature extraction branch, the PointCloudMamba point cloud encoding structure can be used to model global features for the same set of keypoints. The PointCloudMamba structure primarily extracts global spatial context and overall scene information by spatially serializing keypoints in the point cloud and using a state-space model to model the dependencies between distant points. Besides the PointCloudMamba structure, the global feature extraction branch can also use PointTransformer, PCT, Point-BERT, or other state-space model structures to extract global spatial context features for each keypoint based on the spatial dependencies between them.

[0045] Taking the PointCloudMamba point cloud encoding structure as an example, this paper introduces the main structure and working principle of the global feature extraction branch: The PointCloudMamba point cloud encoding structure adopts a sequence modeling architecture based on a state-space model, mainly including input sequence mapping, state-space recursive update, and output mapping. The PointCloudMamba point cloud encoding structure first performs spatial serialization on the input keypoint set, transforming the discretely distributed point cloud data in three-dimensional space into an ordered one-dimensional sequence representation. Then, the input sequence mapping module converts the serialized point cloud features into the state input, state transition, and output projection parameters required by the state-space model. In the state-space recursive update stage, the model performs recursive calculations on the current input features based on historical hidden states, establishing state associations between distant keypoints along the sequence direction. Finally, the output mapping generates a point-level feature representation containing global spatial context information.

[0046] The PointCloudMamba point cloud encoding structure works by using spatial serialization to break the inherent disorder and irregularity of point cloud data, transforming 3D point clouds into a one-dimensional sequence suitable for state-space model processing. This allows the model to pass state information along the sequence direction and establish dependencies between distant points. During the recursive update process, historical hidden states carry global scene information from previous key points and are fused with the current input features. The linear recursive mechanism of the state-space model captures long-range dependencies across spatial distances, thus encoding the overall scene layout and long-distance spatial associations into the feature representations of each key point. The PointCloudMamba point cloud encoding structure achieves global context modeling with linear computational complexity, avoiding the high quadratic complexity of global attention calculations, and ultimately outputs global spatial context features that reflect the overall scene structure.

[0047] Optionally, the above-mentioned fusion of local geometric detail features and global spatial context features to obtain point-level features for each key point includes: concatenating local geometric detail features and global spatial context features in the channel dimension to obtain concatenated features; inputting the concatenated features into a gating network to obtain a fusion weight matrix; and dynamically weighting and fusing local geometric detail features and global spatial context features based on the fusion weight matrix to obtain point-level features for each key point.

[0048] After obtaining local and global features, the two types of features are concatenated along the channel dimension. Then, adaptive gating fusion generates point-by-point and channel-by-channel fusion weights, dynamically weighting the local and global features for fusion. This fusion mechanism can adaptively adjust the contribution ratio of local and global information based on the response differences of keypoints at different spatial locations and in different feature channels. The overall calculation process can be uniformly represented by the following formula: ; In the formula, For a moment Local geometric features corresponding to the set of key points; Number of key points; For feature dimensions; For a moment Global context features corresponding to the set of key points; This refers to the splicing features in the channel dimension. ; The adaptive gating weight matrix is ​​generated by the gating network based on the concatenated features and normalized by the Sigmoid function. interval, i.e. , For the Sigmoid function; This represents the Hadamard product operator, which implements point-by-point and channel-by-channel weighted fusion; These are the point-level features obtained after fusion.

[0049] The fused point-level deep features are input into the classification head, which outputs the initial semantic probability distribution for each keypoint. The classification head consists of a learnable weight matrix and a bias vector. The unnormalized classification score is converted into a probability distribution using the Softmax function, calculated as follows: ; In the formula, This is the weight matrix for the classification heads; This is the bias vector for the classification head. The number of semantic categories; For a moment The initial semantic probability distribution of each key point; Indicates the first The point belongs to the th point The probability of each semantic category.

[0050] Step S130: Based on the initial semantic probability distribution and semantic prior weight vector of each key point within the same BEV grid, the point-level features of the key points are weighted and aggregated to obtain a semantically enhanced two-dimensional BEV feature map; wherein, the semantic prior weight vector is set differently based on the semantic category, and the weight coefficient corresponding to the dynamic target category in the semantic prior weight vector is higher than the weight coefficient corresponding to the static background category.

[0051] The aforementioned semantic prior weight vector is a category-level weight parameter pre-configured based on the differences in the importance of semantic categories in modeling dynamic scenes. Its dimension is consistent with the total number of semantic categories, and each element in the vector corresponds to the prior weight coefficient of different semantic categories. The semantic prior weights are configured according to the principle that the weight coefficient of dynamic target categories is higher than that of static background categories. High-dynamic-risk categories such as pedestrians, vehicles, and cyclists are assigned relatively high values, potential-risk categories such as traffic cones and obstacles are assigned medium values, and static background categories such as roads, buildings, and vegetation are assigned relatively low values. This ensures that dynamic target categories receive a higher aggregation contribution than static background categories during BEV projection aggregation. In the weighted aggregation stage, the semantic prior weight vector and the initial semantic probability distribution of keypoints are multiplied by an inner product, mapping the category-level prior importance differences to semantic weight scalars for each keypoint. This guides keypoints within the same BEV grid to be weighted differently based on their semantic category importance, suppressing the masking of dynamic small target features by static background features.

[0052] Optionally, step S130 includes: for each key point within the same BEV grid, performing an inner product operation between the initial semantic probability distribution of the key point and the semantic prior weight vector to obtain the semantic weight scalar of the key point; normalizing the semantic weight scalar of each key point within the same BEV grid to obtain the aggregated weight of each key point within the BEV grid; and using the aggregated weight, performing weighted aggregation of the point-level features of each key point within the same BEV grid to obtain a semantically enhanced two-dimensional BEV feature map.

[0053] Optionally, the above-mentioned normalization processing of the semantic weight scalars of each key point within the same BEV grid to obtain the aggregate weight of each key point within the BEV grid includes: normalizing the semantic weight scalars of each key point in the set of points to be aggregated to obtain the aggregate weight of each key point in the set of points to be aggregated within the corresponding BEV grid; wherein, the set of points to be aggregated is a set of key points that fall into the same BEV grid but are located at different height layers. The above-mentioned method utilizes aggregation weights to perform weighted aggregation of point-level features of key points within the same BEV grid to obtain a semantically enhanced two-dimensional BEV feature map. This includes: using aggregation weights, performing weighted compression and feature aggregation on the height direction of the point-level features of each key point in the set of points to be aggregated to obtain a two-dimensional BEV feature representation of the corresponding BEV grid position; and obtaining a semantically enhanced two-dimensional BEV feature map based on the two-dimensional BEV feature representation of each BEV grid position.

[0054] The above scheme divides the space surrounding the vehicle into a regular two-dimensional BEV grid matrix in the horizontal plane. The size of the planar grid is consistent with the cell size of the three-dimensional grid in the data preprocessing module in the horizontal plane, so as to ensure the spatial mapping alignment between the three-dimensional key points and the two-dimensional grid positions. For any two-dimensional grid coordinate position, all key points that are indexed and mapped to that grid in the horizontal plane but are distributed in different height layers are formed into a set of points to be aggregated. This set of points to be aggregated is the set of key points that fall within the same BEV grid space and need to be feature compressed and aggregated in the height direction.

[0055] For each keypoint within the same BEV grid, the initial semantic probability distribution of the keypoint is multiplied by the semantic prior weight vector to obtain the semantic weight scalar of the keypoint. This inner product operation multiplies the probability value of the keypoint belonging to each semantic category by the prior weight coefficient of the corresponding category and then sums the results. This ensures that keypoints belonging to the high-weight dynamic target category receive a higher semantic weight scalar, while keypoints belonging to the low-weight static background category receive a lower semantic weight scalar.

[0056] Within the same BEV grid, the semantic weight scalar of each keypoint is normalized to obtain the normalized aggregate weight of each keypoint within the corresponding BEV grid; the formula for calculating the normalized aggregate weight is as follows: ; In the formula, For the first Normalized aggregate weights of key points within the corresponding BEV grid; For a moment Next The initial semantic probability vector of each key point has the dimension of the number of semantic categories, and each element represents the probability that the key point belongs to the corresponding semantic category; The preset semantic prior weight vector has the same dimension as the number of semantic categories, and each element is the prior weight coefficient of the corresponding semantic category. To fall into the first The set of key points within a BEV grid; For set Key point index in the text.

[0057] The fused features of each key point within the same BEV grid are weighted and summed using normalized aggregation weights to generate a two-dimensional BEV feature representation at the corresponding grid location. The formula for weighted aggregation is as follows: ; In the formula, For a moment Next The feature vector of each BEV grid has the same feature dimension as the point-level fused feature. For a moment Next The fused point-level features corresponding to each key point; For the first The normalized aggregate weights of each key point within the corresponding BEV grid are calculated. By traversing all BEV grid positions, a semantically enhanced two-dimensional BEV feature map is constructed from the two-dimensional BEV feature representations of each grid position.

[0058] Step S140: Use the state-space model to perform cross-frame temporal feature enhancement on the two-dimensional BEV feature map to obtain a spatiotemporally enhanced BEV feature map.

[0059] It is understandable that in the related technologies of dynamic scene perception based on 3D point clouds, the BEV temporal modeling stage usually uses Transformer-type methods or a single-direction serialized state space model to process the BEV features of consecutive frames. While Transformer-based methods possess strong global modeling capabilities, the computational complexity of their self-attention mechanism increases quadratically with the number of BEV grids. Under conditions of high-resolution BEV feature maps and long temporal windows, the memory usage and computational overhead are significant, making it difficult to meet the low-latency, real-time perception requirements of autonomous driving systems. Although single-direction serialization state-space models achieve long-range dependency modeling with linear complexity, flattening the two-dimensional BEV feature map into a one-dimensional sequence along a single scanning direction disrupts the horizontal and vertical positional relationships in the two-dimensional space. This results in a significant loss of spatial adjacency relationships and geometric topology between grids during serialization. Furthermore, existing state-space model methods are mostly focused on static point cloud understanding or single-sequence modeling, failing to adequately address the unified connection between multi-directional serialization of BEV features, cross-frame hidden state updates, and inverse reconstruction decoding. This makes it difficult to simultaneously address the continuous evolution of cross-frame dynamic information and the restoration of two-dimensional spatial structures, resulting in a situation where insufficient efficiency in dynamic information extraction and weakened spatial relationship preservation coexist during temporal modeling. Based on this, the embodiments of the present invention provide the following solutions: Optionally, step S140 includes: sequentially unfolding the two-dimensional BEV feature map along multiple preset directions to obtain multiple one-dimensional sequences; wherein the one-dimensional sequences have an index mapping relationship with the original spatial positions of the two-dimensional BEV feature map; inputting each one-dimensional sequence into the state space model and recursively updating it in combination with the historical hidden states of the corresponding directions to obtain the temporal enhancement sequences corresponding to each direction; rearranging each temporal enhancement sequence into a two-dimensional feature matrix according to the inverse index mapping relationship corresponding to each sequential unfolding rule; and fusing the two-dimensional feature matrices obtained by rearranging in multiple directions to obtain a spatiotemporally enhanced BEV feature map.

[0060] For the semantically enhanced 2D BEV feature map generated at the current moment, a multi-directional serialization unrolling strategy can be used to rearrange it into multiple 1D sequences to alleviate the problem of insufficient expression of 2D spatial relationships by a single-directional scan. During the serialization unrolling process, each scanning method corresponds to a preset spatial index mapping relationship, ensuring that each element in the 1D sequence maintains a one-to-one correspondence with the spatial grid position in the original 2D BEV feature map.

[0061] Optionally, the above-mentioned serialization and unfolding of the two-dimensional BEV feature map along multiple preset directions to obtain multiple one-dimensional sequences includes: serializing and unfolding the two-dimensional BEV feature map along four preset directions to obtain four one-dimensional sequences; wherein, the preset directions include the row traversal direction from top left to bottom right, the row traversal direction from bottom right to top left, the column traversal direction from top right to bottom left, and the column traversal direction from bottom left to top right.

[0062] Figure 3 A 2D BEV feature map with a size of 3×3 is shown. The schematic diagram illustrates the serialization and unfolding process along four preset directions, where the grid positions are identified by the numbers 1 to 9 in their arrangement order in two-dimensional space. The first row from left to right consists of grids 1, 2, and 3; the second row from left to right consists of grids 4, 5, and 6; and the third row from left to right consists of grids 7, 8, and 9. After serialization and unfolding, the length of each one-dimensional sequence is equal to the total number of grids, which is 9, and each element in the sequence maintains a one-to-one mapping relationship with the original two-dimensional grid positions.

[0063] When expanding along direction 1 (row-by-row from top left to bottom right), starting from the left side of the first row, traverse grids 1, 2, and 3 sequentially to the right. Then, move to the second row and traverse grids 4, 5, and 6 from left to right. Finally, move to the third row and traverse grids 7, 8, and 9 from left to right, generating the sequence. When expanding along direction 2 (row-by-row from bottom right to top left), starting from the right side of the third row, traverse grids 9, 8, and 7 sequentially to the left. Then, move to the second row and traverse grids 6, 5, and 4 from right to left. Finally, move to the first row and traverse grids 3, 2, and 1 from right to left, generating the sequence. When expanding along direction 3 (traversing columns from top right to bottom left), starting from the right column of the first row, traverse grids 3, 6, and 9 sequentially downwards. Then, move to the adjacent left column and traverse grids 2, 5, and 8 from top to bottom. Finally, move to the leftmost column and traverse grids 1, 4, and 7 from top to bottom, generating the sequence. When expanding along direction 4 (traversing columns from bottom left to top right), starting from the left column of the third row, traverse grids 7, 4, and 1 sequentially upwards. Then, move to the adjacent right column and traverse grids 8, 5, and 2 from bottom to top. Finally, move to the rightmost column and traverse grids 9, 6, and 3 from bottom to top, generating the sequence. The sequence length L of each one-dimensional sequence is equal to the total number of spatial grids in the two-dimensional BEV feature map, i.e., L = H × W, where H is the number of grid rows and W is the number of grid columns. In the example attached, H = 3 and W = 3. During the serialization unrolling process, each scanning direction corresponds to a preset two-dimensional index mapping rule. This rule records the grid coordinates in the original two-dimensional BEV feature map corresponding to each position in the sequence, thereby establishing a reversible one-to-one mapping relationship between the one-dimensional sequence and the original two-dimensional spatial position, providing an index basis for subsequent reverse reconstruction decoding.

[0064] The four-way cross-scanning strategy, through a complementary combination of row-first and column-first, forward and reverse directions, allows the spatial relationships of the same grid neighborhood in a 2D BEV feature map to be expressed in different contextual orders in different directional sequences. Compared to single-directional scanning, which can only maintain one-dimensional adjacency relationships along the scanning path, multi-directional unfolding distributes the spatial relationships of horizontal adjacency, vertical adjacency, and different diagonal directions into multiple sequences for separate encoding.

[0065] It is understandable that, in addition to the four-way cross scanning strategy mentioned above, other scanning strategies such as zigzag scanning, diagonal scanning, Hilbert curve scanning, or Z-order scanning can also be used.

[0066] One-dimensional sequences in each direction are input into a time-series processing module based on a state-space model, and recursive updates are performed by combining the historical hidden states of the corresponding directions from the previous time step. The recursive update, through the linear recursive mechanism of the state-space model, allows the input features at the current time step to fuse historical dynamic information during state transitions, capturing long-range dependencies across frames. The mathematical expression of the time-series enhancement mapping and the state update mapping is as follows: ; In the formula, For a moment Next The one-dimensional input sequence corresponding to each scanning direction has a sequence length equal to the total number of grids in the two-dimensional BEV feature map, and the feature dimension is consistent with the point-level fusion feature dimension. For the first The hidden historical state of each direction at the previous time step, and its state dimension is determined by the internal parameters of the state space model. For a moment Next The one-dimensional temporally enhanced sequence obtained after temporal enhancement in each scanning direction has the same sequence length and feature dimension as the input sequence. For a moment Next Hidden state after updating in each scanning direction; This represents the temporal enhancement mapping, which is responsible for fusing the current input with historical states into temporal enhancement features; This represents the state update mapping, which is responsible for generating new hidden states based on the current input and historical states.

[0067] Optionally, the above-mentioned fusion of the two-dimensional feature matrix obtained by multi-directional rearrangement to obtain the spatiotemporally enhanced BEV feature map includes: summing the two-dimensional feature matrix obtained by multi-directional rearrangement element by element according to the corresponding spatial index position to obtain the spatiotemporally enhanced BEV feature map.

[0068] The temporal enhancement sequences in each direction need to be rearranged according to the inverse index mapping relationship corresponding to the serialization unrolling rule to restore the two-dimensional spatial structure. Since the two-dimensional index mapping is preserved during serialization unrolling, the inverse reconstruction operator uses this mapping to place the elements in the one-dimensional sequence back into the original BEV grid positions, obtaining the two-dimensional feature matrix corresponding to each direction. The two-dimensional feature matrices obtained from the multi-directional rearrangement are summed element-wise at the same spatial index positions to achieve the fusion of multi-directional temporal enhancement information and obtain a unified spatiotemporal enhanced BEV feature map; the mathematical expression of the fusion process is as follows: ; In the formula, For a moment The spatiotemporally enhanced BEV feature map after fusion has the same spatial dimensions as the original two-dimensional BEV feature map. Figure 1 The feature dimensions are consistent with those of the temporal enhancement sequence; For the first The reverse reconstruction operator corresponding to each scanning direction restores the one-dimensional time-enhanced sequence into a two-dimensional feature matrix based on the two-dimensional index mapping relationship saved during serialization. For the first The temporal enhancement sequence corresponds to each scan direction. It can be understood that the number 4 in the above formula corresponds to the number of scan directions. It can be understood that in the above four-way cross-scan strategy, the number of scan directions is 4, but in other scan strategies, the number of scan directions is not necessarily 4.

[0069] It is understood that step S140 above uses the spatiotemporal feature processing module of Mamba-3 to perform cross-frame temporal feature enhancement on the two-dimensional BEV feature map. In addition to using the Mamba-3 spatiotemporal feature processing module, alternative solutions such as ConvLSTM, GRU, LSTM, Transformer temporal attention network or other state-space models can also be used to perform cross-frame temporal feature enhancement.

[0070] Taking the spatiotemporal feature processing module of Mamba-3 as an example, the main architecture and working principle of the spatiotemporal feature processing module are introduced: The Mamba-3 spatiotemporal feature processing module adopts a sequence modeling architecture based on a state-space model. Its core components include input sequence mapping, state-space recursive update, complex-valued state representation, MIMO (Multiple-Input Multiple-Output) projection, and output mapping. Input sequence mapping transforms the one-dimensional input sequence into the state input, state transition, and output projection parameters required by the state-space model. The state-space recursive update module performs recursive calculations on the current input and historical hidden states based on the state transition parameters. Complex-valued state representation enhances the model's ability to express state changes by introducing complex-domain state variables. The MIMO structure processes multi-channel sequence features to enhance the parallel modeling capability and inference efficiency of multi-dimensional features. Finally, output mapping projects the updated state variables into an enhanced sequence output containing temporal context information.

[0071] The working principle of the Mamba-3 spatiotemporal feature processing module includes: the input one-dimensional sequence is first converted into recursive input parameters of the state-space model through input sequence mapping. These parameters, along with the historical hidden states in the corresponding direction of the previous time step, are sent to the state-space recursive update module; the complex-valued state representation maintains and updates the hidden state vector with complex domain expressive power during the recursive process, enabling the model to more finely characterize the state evolution trajectory; the MIMO structure performs parallel projection and interaction on multi-channel features, improving the joint modeling capability of multi-dimensional temporal features; through a linear recursive mechanism, the current sequence features fuse the hidden state information transmitted by historical frames during the state transition process, gradually establishing long-range dependency associations along the sequence direction, and finally generating a temporally enhanced sequence that fuses historical dynamic information through output mapping, realizing the fusion of temporal information and capture of dynamic changes between the current frame and historical frames.

[0072] Step S150: Decode the spatiotemporally enhanced BEV feature map to obtain the dynamic occupancy grid prediction result and the two-dimensional motion information prediction result.

[0073] In step S150 above, the spatiotemporally enhanced BEV feature map can be input into a multi-task convolutional decoding head. This decoding head performs feature transformation and task decoupling on the spatiotemporally enhanced features through convolution operations, mapping them to the dynamic occupancy prediction branch and the two-dimensional motion information prediction branch, respectively. In the dynamic occupancy prediction branch, the decoding head outputs a dynamic occupancy probability map with the same spatial size as the BEV grid matrix, where the predicted value of each grid position represents the probability that the position is occupied by a dynamic target, forming a dynamic occupancy grid prediction result. In the two-dimensional motion information prediction branch, the decoding head outputs a two-dimensional motion vector field corresponding to the spatial size of the BEV grid matrix, where the predicted value of each grid position represents the two-dimensional motion information of the dynamic target at that position in the horizontal plane, forming a two-dimensional motion information prediction result.

[0074] Based on the same inventive concept, this invention also provides another dynamic scene target perception method based on a 3D point cloud and state space model. This method includes the following steps performed by a 3D spatiotemporal perception network: extracting a set of key points from the 3D point cloud data of the dynamic scene; extracting point-level features from the set of key points to obtain the point-level features and initial semantic probability distribution of each key point; weighting and aggregating the point-level features of the key points based on the initial semantic probability distribution and semantic prior weight vector within the same BEV grid to obtain a semantically enhanced 2D BEV feature map; wherein the semantic prior weight vector is differentiated based on semantic categories, with the weight coefficient corresponding to the dynamic target category being higher than the weight coefficient corresponding to the static background category in the semantic prior weight vector; using a state space model to perform cross-frame temporal feature enhancement on the 2D BEV feature map to obtain a spatiotemporally enhanced BEV feature map; and decoding the spatiotemporally enhanced BEV feature map to obtain dynamic grid occupancy prediction results and 2D motion information prediction results.

[0075] The main architecture and working principle of the aforementioned three-dimensional spatiotemporal perception network will be introduced below: like Figure 4 As shown, the feature fusion and semantic probability generation module in the 3D spatiotemporal perception network receives a set of keypoints as input and constructs a local geometric detail feature extraction branch and a global spatial context feature extraction branch in parallel for the same set of keypoints. The local feature extraction branch uses the PointNeXt point cloud encoding structure to model the features of the local neighborhood of each keypoint in the set and outputs a local feature matrix. ,in, Number of key points; For the feature dimension; the global feature extraction branch adopts the PointCloudMamba point cloud encoding structure, performs global feature modeling on the same set of key points, and outputs a global feature matrix. The feature matrices output by both branches correspond to the same spatial dimension. Each of the key points is in terms of feature dimension. This ensures a one-to-one correspondence at each point in the subsequent fusion operations.

[0076] The local feature matrix and the global feature matrix are concatenated along the channel dimension to obtain the concatenated features. The spliced ​​features are input into a gating network composed of multilayer perceptrons. via the Sigmoid function Normalize the output to Intervals are used to generate point-by-point and channel-by-channel fusion weight matrices. Based on the fusion weight matrix, a Hadamard product weighted fusion is performed on the local and global features to obtain point-level features. The fusion formula is: In the formula, This represents the Hadamard product operator; For the range of values The adaptive gating weight matrix; , and All are real-valued eigenma matrices, and their elements are eigenresponse values.

[0077] Point-level features Input a classification head, which consists of a learnable weight matrix. With bias vector Composition, among which, The number of semantic categories; the classification head generates unnormalized classification scores through matrix multiplication and bias addition. The calculation formula is: Then, the Softmax function will... Normalize to a probability distribution, and output the initial semantic probability distribution of each keypoint. ,Right now: ; In the formula, Indicates the first The key point is in the first Unnormalized classification scores for each semantic category; Indicates the first The point belongs to the th point The probability of each semantic category.

[0078] The training method for the aforementioned three-dimensional spatiotemporal perception network is described below: To enable the entire 3D spatiotemporal perception network to learn, achieving end-to-end joint training from raw point cloud input to dynamic scene prediction results, this embodiment of the invention provides a multi-task joint loss function. This loss function simultaneously supervises the point-level initial semantic probability prediction results in the feature fusion and semantic probability generation modules, as well as the dynamic occupancy prediction results and motion information prediction results in the spatiotemporal enhancement module. The total loss function consists of three error-weighted components: ; In the formula, For a moment The total loss value below; For a moment The semantic classification error is as follows; For a moment Dynamic occupancy prediction error; For a moment The regression error of motion information under the following conditions; , and These are the weight coefficients for semantic classification error, dynamic occupancy prediction error, and motion information regression error, respectively. Initially, all three are set to 1. During training, they are dynamically adjusted based on the numerical scale of each loss term, the proportion of dynamic target samples, the prediction effect on the validation set, and the convergence speed of different tasks, in order to balance the gradient contribution of each loss term during multi-task training.

[0079] For the initial semantic probability distribution output by the feature fusion and semantic probability generation modules, a focus loss function is used for supervision to alleviate the class imbalance problem caused by the number of static background samples being much larger than the number of dynamic target samples in autonomous driving scenarios. The formula for calculating the semantic classification error is: ; In the formula, Number of key points; The total number of semantic categories; For a moment Next The key point belongs to the first Predicted probability of semantic category; For the first The key point corresponds to the first The true semantic label of the semantic category, with a value of 0 or 1; for Balanced weights corresponding to semantic categories; This is the adjustment coefficient for focus loss.

[0080] For the dynamic occupancy probability map output by the spatiotemporal augmentation module, a binary cross-entropy loss is used for supervision. The formula for calculating the dynamic occupancy prediction error is: ; In the formula, This represents the number of rows in the BEV grid. This represents the number of grid columns for BEVs. For a moment Next The predicted probability that a BEV grid cell is occupied by a dynamic target; For a moment Next Each BEV grid has a real binary label, with a value of 0 or 1.

[0081] For the two-dimensional motion information tensor output by the spatiotemporal augmentation module, a smoothed L1 loss is used for supervision. Since the motion information only has regression significance for valid BEV graticles containing dynamic targets, the regression error is calculated only at the graticle positions where the actual occupant label is positive. The dynamic graticle mask is defined as: ; In the formula, For a moment Next Dynamic mask for each BEV grid position; The total number of valid dynamic grids is: ; The formula for calculating the regression error of motion information is: ; In the formula, For a moment Next Predicted two-dimensional motion vectors for each BEV grid; For a moment Next The true two-dimensional motion vector of each BEV grid; To smooth the L1 loss function, a piecewise linear penalty is applied to the deviation between the predicted and the true values. When the deviation is small, L2 loss is used to ensure gradient smoothness, and when the deviation is large, L1 loss is used to suppress the influence of outliers.

[0082] Through end-to-end supervision of the multi-task joint loss function, the 3D spatiotemporal perception network simultaneously optimizes three tasks during training: point-level semantic classification, dynamic occupancy prediction, and motion information regression, enabling the parameters of each module to converge collaboratively under a unified objective function constraint.

[0083] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A dynamic scene target perception method based on a 3D point cloud and state space model, characterized in that, The method includes: Extracting key point sets from 3D point cloud data in dynamic scenes; Point-level feature extraction is performed on the set of key points to obtain the point-level features of each key point and the initial semantic probability distribution; Based on the initial semantic probability distribution and semantic prior weight vector of each key point within the same BEV grid, the point-level features of the key points are weighted and aggregated to obtain a semantically enhanced two-dimensional BEV feature map; wherein, the semantic prior weight vector is set differently based on semantic categories, and the weight coefficient corresponding to the dynamic target category is higher than the weight coefficient corresponding to the static background category in the semantic prior weight vector. The state-space model is used to perform cross-frame temporal feature enhancement on the two-dimensional BEV feature map to obtain a spatiotemporally enhanced BEV feature map. The spatiotemporally enhanced BEV feature map is decoded to obtain the dynamic occupancy grid prediction result and the two-dimensional motion information prediction result; The step of weighted aggregation of the point-level features of key points based on the initial semantic probability distribution and semantic prior weight vector of each key point within the same BEV grid to obtain a semantically enhanced two-dimensional BEV feature map includes: for each key point within the same BEV grid, performing an inner product operation between the initial semantic probability distribution of the key point and the semantic prior weight vector to obtain a semantic weight scalar of the key point; normalizing the semantic weight scalar of each key point within the same BEV grid to obtain the aggregated weight of each key point within the BEV grid; and using the aggregated weight to weighted aggregate the point-level features of each key point within the same BEV grid to obtain a semantically enhanced two-dimensional BEV feature map. The step of using a state-space model to perform cross-frame temporal feature enhancement on the two-dimensional BEV feature map to obtain a spatiotemporally enhanced BEV feature map includes: sequentially unfolding the two-dimensional BEV feature map along multiple preset directions to obtain multiple one-dimensional sequences; wherein the one-dimensional sequences have an index mapping relationship with the original spatial position of the two-dimensional BEV feature map; inputting each of the one-dimensional sequences into the state-space model and recursively updating it in combination with the historical hidden state of the corresponding direction to obtain the temporal enhancement sequence corresponding to each direction; rearranging each of the temporal enhancement sequences into a two-dimensional feature matrix according to the inverse index mapping relationship corresponding to each of the sequential unfolding rules; and fusing the two-dimensional feature matrices obtained by rearranging in multiple directions to obtain the spatiotemporally enhanced BEV feature map.

2. The dynamic scene target perception method based on a 3D point cloud and state space model according to claim 1, characterized in that, The normalization process for the semantic weight scalars of each key point within the same BEV grid to obtain the aggregate weight of each key point within the BEV grid includes: The semantic weight scalar of each key point in the set of points to be aggregated is normalized to obtain the aggregation weight of each key point in the set of points to be aggregated within the corresponding BEV grid; wherein, the set of points to be aggregated is a set of key points that fall into the same BEV grid but are located at different height layers. The step of using the aggregation weights to perform weighted aggregation of the point-level features of each key point within the same BEV grid to obtain a semantically enhanced two-dimensional BEV feature map includes: Using the aggregation weight, the point-level features of each key point in the set of points to be aggregated are weighted, compressed and aggregated in the height direction to obtain a two-dimensional BEV feature representation of the corresponding BEV grid position; Based on the two-dimensional BEV feature representation of each BEV grid position, a semantically enhanced two-dimensional BEV feature map is obtained.

3. The dynamic scene target perception method based on a 3D point cloud and state space model according to claim 1, characterized in that, The step of extracting point-level features from the set of key points to obtain the point-level features of each key point and the initial semantic probability distribution includes: For each key point in the set of key points, local geometric detail features of each key point are extracted based on the geometric structure of the local neighborhood of the key point. For each key point in the set of key points, global spatial context features of each key point are extracted based on the spatial dependencies between them. By fusing the local geometric detail features and the global spatial context features, point-level features of each key point are obtained; The point-level features are input into the classification head to obtain the initial semantic probability distribution of each key point.

4. The dynamic scene target perception method based on a three-dimensional point cloud and state space model according to claim 3, characterized in that, The process of fusing the local geometric detail features and the global spatial context features to obtain point-level features for each key point includes: The local geometric detail features and the global spatial context features are concatenated along the channel dimension to obtain the concatenated features; The spliced ​​features are input into a gated network to obtain a fusion weight matrix; Based on the fusion weight matrix, the local geometric detail features and the global spatial context features are dynamically weighted and fused to obtain the point-level features of each key point.

5. The dynamic scene target perception method based on a three-dimensional point cloud and state space model according to claim 1, characterized in that, The process of fusing the two-dimensional feature matrix obtained by multi-directional rearrangement to obtain a spatiotemporally enhanced BEV feature map includes: The two-dimensional feature matrix obtained by multi-directional rearrangement is summed element by element according to the corresponding spatial index position to obtain the spatiotemporally enhanced BEV feature map.

6. The dynamic scene target perception method based on a 3D point cloud and state space model according to claim 1, characterized in that, The step of sequentially unfolding the two-dimensional BEV feature map along multiple preset directions to obtain multiple one-dimensional sequences includes: The two-dimensional BEV feature map is serialized and unfolded along four preset directions to obtain four one-dimensional sequences; wherein, the preset directions include the row traversal direction from top left to bottom right, the row traversal direction from bottom right to top left, the column traversal direction from top right to bottom left, and the column traversal direction from bottom left to top right.

7. The dynamic scene target perception method based on a three-dimensional point cloud and state space model according to any one of claims 1 to 6, characterized in that, The extraction of key point sets from 3D point cloud data in dynamic scenes includes: Acquire raw point cloud data collected by multi-view LiDAR around the vehicle; By using a pre-calibrated extrinsic transformation matrix, the point clouds from each viewpoint are uniformly transformed to the vehicle coordinate system and spatially stitched together to obtain a panoramic 3D point cloud set. The panoramic 3D point cloud set is downsampled according to regular 3D grid units; wherein, the unit size of the regular 3D grid unit in the horizontal plane is consistent with the planar grid size used in BEV projection; For multiple points falling within the same 3D grid cell, extract representative points within the 3D grid cell to obtain a key point set composed of the representative points.

8. The dynamic scene target perception method based on a three-dimensional point cloud and state space model according to any one of claims 1 to 6, characterized in that, The method further includes: The following steps are performed by a 3D spatiotemporal perception network: Extracting a set of key points from the 3D point cloud data of a dynamic scene; extracting point-level features from the set of key points to obtain the point-level features and initial semantic probability distribution of each key point; weighting and aggregating the point-level features of the key points based on the initial semantic probability distribution and semantic prior weight vector of each key point within the same BEV grid to obtain a semantically enhanced 2D BEV feature map; wherein the semantic prior weight vector is differentiated based on semantic categories, with the weight coefficient corresponding to the dynamic target category being higher than the weight coefficient corresponding to the static background category; using a state-space model to perform cross-frame temporal feature enhancement on the 2D BEV feature map to obtain a spatiotemporally enhanced BEV feature map; and decoding the spatiotemporally enhanced BEV feature map to obtain dynamic grid occupancy prediction results and 2D motion information prediction results. The training method for the three-dimensional spatiotemporal perception network includes: The 3D spatiotemporal perception network is jointly trained end-to-end using a multi-task joint loss function; wherein the multi-task joint loss function includes: Semantic classification error is used to quantify the classification deviation between the initial semantic probability distribution and the corresponding true semantic label; Dynamic occupancy prediction error is used to quantify the prediction deviation between the dynamic occupancy grid prediction result and the corresponding true binary occupancy label. Motion information regression error is used to quantify the regression deviation between the two-dimensional motion information prediction result and the corresponding real two-dimensional motion vector at the grid position where the actual occupation label is positive.

Images