Automatic driving scene dynamic three-dimensional reconstruction method and device

Abstract

The application relates to the technical field of intelligent driving, in particular to an automatic driving scene dynamic three-dimensional reconstruction method and device, wherein the method comprises the following steps: extracting features of an automatic driving scene at multiple angles from a key frame image sequence to determine fused features, and determining a dense depth value of each pixel point in the key frame image sequence and a three-dimensional Gaussian point cloud of a corresponding scene according to the fused features and an initial three-dimensional point cloud; assigning a dynamic attribute label or a static attribute label to each point in the three-dimensional Gaussian point cloud to determine a dynamic attribute mask, and reconstructing a three-dimensional scene representation containing a static background and dynamic features according to the dynamic attribute mask. Thus, the problems that the related art relies on adjusting transparency to realize simple superposition of foreground and background, it is difficult to truly reflect the occlusion relationship of multiple dynamic objects, the occlusion boundary is prone to be blurred, the geometric structure is distorted, and the rendering reality is insufficient, thereby affecting the precision and visual effect of automatic driving scene modeling and the like are solved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving technology, and in particular to a method and apparatus for dynamic three-dimensional reconstruction of autonomous driving scenarios. Background Technology

[0002] In related technologies, autonomous driving scene modeling typically employs methods such as 3D point cloud, Gaussian point cloud, or neural rendering to reconstruct the scene. When multiple dynamic objects occlude each other, opacity adjustment is usually used to handle the depth relationship between different objects, in order to alleviate rendering conflicts in occluded areas.

[0003] However, the relevant technologies mainly achieve simple overlay of foreground and background by adjusting transparency, without fully considering the complex spatial geometric relationships and occlusion interaction processes between dynamic objects. This makes it difficult to realistically reflect the occlusion relationships of multiple dynamic objects, which can easily lead to blurred occlusion boundaries, distorted geometric structures, and insufficient rendering realism, thus affecting the accuracy and visual effects of autonomous driving scene modeling. Summary of the Invention

[0004] This application provides a method and apparatus for dynamic 3D reconstruction of autonomous driving scenes to solve the problem that in related technologies, the simple superposition of foreground and background is mainly achieved by adjusting transparency, which is difficult to realistically reflect the occlusion relationship of multiple dynamic objects, and is prone to blurred occlusion boundaries, geometric distortion and insufficient rendering realism, thereby affecting the accuracy and visual effect of autonomous driving scene modeling.

[0005] The first aspect of this application provides a method for dynamic 3D reconstruction of an autonomous driving scene, comprising the following steps: generating a keyframe image sequence that satisfies preset incremental optimization conditions based on images from a vehicle-mounted camera in the autonomous driving scene to construct an initial 3D point cloud of the autonomous driving scene; extracting features of the autonomous driving scene from the keyframe image sequence from multiple perspectives to determine fusion features, and determining the density depth value of each pixel in the keyframe image sequence and the corresponding 3D Gaussian point cloud of the scene based on the fusion features and the initial 3D point cloud; assigning dynamic attribute labels or static attribute labels to each point in the 3D Gaussian point cloud based on the density depth values ​​to determine a dynamic attribute mask, and reconstructing a 3D scene representation containing static background and dynamic features based on the dynamic attribute mask.

[0006] Using the above technical means, a 3D Gaussian point cloud is generated based on the fusion features from multiple perspectives. Then, based on the density depth value and attribute information of each pixel, dynamic targets and static backgrounds in the scene are separated. The scene is then optimized and reconstructed by combining the 3D Gaussian point cloud after dynamic and static separation. This can effectively avoid the interference caused by dynamic targets to the modeling of static environment, while preserving the spatial position and change characteristics of dynamic targets, thereby improving the integrity, accuracy and temporal consistency of the 3D scene representation in complex dynamic scenes of autonomous driving.

[0007] Optionally, in one embodiment of this application, generating a key frame image sequence that satisfies preset incremental optimization conditions based on images from vehicle-mounted cameras in an autonomous driving scenario includes: detecting scale-invariant feature transformation feature points of the images to determine the two-dimensional features of the images; based on the two-dimensional features, selecting image frames that satisfy preset incremental optimization conditions from the images according to at least one of image frame disparity, image overlap rate, and motion baseline; and based on the image frames, generating incremental optimization information of camera pose and sparse three-dimensional point cloud with minimizing reprojection error as the optimization objective to obtain the key frame image sequence.

[0008] Using the above techniques, based on image frames, incremental optimization information for camera pose and sparse 3D point cloud is generated with the goal of minimizing reprojection error. This can iteratively optimize newly added keyframes and their associated 3D points while ensuring local geometric consistency, thereby improving the accuracy of camera pose estimation and the reconstruction quality of sparse 3D point cloud, and providing accurate initial geometric information for subsequent dense depth estimation and 3D scene reconstruction.

[0009] Optionally, in one embodiment of this application, extracting features of an autonomous driving scene from a keyframe image sequence under multiple perspectives to determine fusion features includes: determining the order of image frames in the keyframe image sequence based on the timestamps and poses of the keyframe image sequence, so as to determine the current perspective and multiple neighboring perspectives that meet the preset condition of sufficient disparity information based on the order; extracting features of the current perspective and multiple neighboring perspectives from the image frames; calculating the interaction attention weights of the current perspective and neighboring perspectives based on the features; and determining the fusion features based on the interaction attention weights.

[0010] By using the above techniques, the interaction attention weights between the current viewpoint and neighboring viewpoints are calculated to determine the fusion features. This allows for the adaptive fusion of effective information from different viewpoints, reducing the impact of occluded areas and missing information from a single viewpoint on feature extraction. It also improves the integrity and consistency of multi-view feature representation in complex autonomous driving scenarios, providing a more reliable feature foundation for subsequent dense depth prediction and 3D Gaussian point cloud reconstruction.

[0011] Optionally, in one embodiment of this application, determining the dense depth value of each pixel in the keyframe image sequence and the corresponding 3D Gaussian point cloud of the scene based on fusion features and an initial 3D point cloud includes: predicting the dense depth value of each pixel using fusion features based on a pre-built feedforward neural network, and determining the 3D Gaussian component by combining the initial 3D point cloud with uncertainty, thereby determining the 3D Gaussian point cloud, wherein the loss function of the pre-built feedforward neural network is: , in, For loss function, These are the weight coefficients corresponding to the photometric consistency loss, depth supervision loss, smoothing regularization loss, and confidence correction loss, respectively. This represents the photometric consistency loss used to constrain the consistency of image reprojection results or image feature consistency under different viewpoints. This represents the depth supervision loss used to constrain the difference between predicted depth values ​​and true depth values, lidar depth values, or sparse depth priors. This represents the smooth regularization loss used to constrain the continuity of depth values ​​between adjacent pixels and suppress abrupt changes in depth. This represents the confidence correction loss used to constrain the prediction uncertainty or confidence level to match the actual depth prediction error.

[0012] By employing the above techniques, based on the dense depth value of each pixel and combined with the initial 3D point cloud and uncertainty, a 3D Gaussian point cloud can be determined. This can fully integrate dense geometric information and sparse geometric priors, and adaptively determine the position and spatial distribution range of each 3D Gaussian component according to the reliability of depth prediction, thereby improving the geometric accuracy and spatial continuity of the 3D Gaussian point cloud.

[0013] Optionally, in one embodiment of this application, assigning dynamic attribute labels or static attribute labels to each point in the 3D Gaussian point cloud based on dense depth values ​​to determine a dynamic attribute mask includes: identifying a first pixel that does not meet a preset static co-point constraint condition based on the dense depth value, assigning a dynamic attribute label to the first pixel, and identifying a second pixel that meets the preset static co-point constraint condition, assigning a static attribute label to the second pixel; and determining a dynamic attribute mask based on the dynamic attribute labels and static attribute labels.

[0014] By using the above techniques, the dynamic attribute mask of pixels can be determined based on static co-point constraints. This allows for the identification of dynamic and static background regions based on the geometric consistency between multiple frames of images, improving the accuracy of dynamic target detection, reducing false detections and missed detections, and providing reliable pixel-level constraints for dynamic-static separation modeling.

[0015] Optionally, in one embodiment of this application, reconstructing a 3D scene representation containing static background and dynamic features based on a dynamic attribute mask includes: using motion speed, peak duration, and inherent motion frequency to describe the spatiotemporal characteristics of a first pixel to determine dynamic features; using a second pixel to fit a local plane to determine a static background; determining dynamic features; and reconstructing a 3D scene representation based on the static background and dynamic features.

[0016] By using the above techniques to perform 3D Gaussian modeling on static backgrounds and dynamic features respectively, we can achieve independent representation of static environment and dynamic targets, reduce the impact of dynamic target motion on static scene reconstruction, and improve the accuracy, completeness and temporal consistency of 3D scene representation in complex dynamic scenes.

[0017] A second aspect of this application provides a dynamic 3D reconstruction device for an autonomous driving scene, comprising: a construction module, configured to generate a keyframe image sequence that satisfies preset incremental optimization conditions based on images from a vehicle-mounted camera in an autonomous driving scene, so as to construct an initial 3D point cloud of the autonomous driving scene; a fusion module, configured to extract features of the autonomous driving scene from the keyframe image sequence from multiple perspectives to determine fusion features, and determine the dense depth value of each pixel in the keyframe image sequence and the corresponding 3D Gaussian point cloud of the scene based on the fusion features and the initial 3D point cloud; and a reconstruction module, configured to assign dynamic attribute labels or static attribute labels to each point in the 3D Gaussian point cloud based on the dense depth values ​​to determine a dynamic attribute mask, and reconstruct a 3D scene representation containing a static background and dynamic features based on the dynamic attribute mask.

[0018] Optionally, in one embodiment of this application, the construction module includes: a detection unit, used to detect scale-invariant feature transformation feature points of the image to determine the two-dimensional features of the image; a filtering unit, used to filter image frames that meet preset incremental optimization conditions from the image based on the two-dimensional features and according to at least one of image frame disparity, image overlap rate, and motion baseline; and a generation unit, used to generate incremental optimization information of camera pose and sparse 3D point cloud based on the image frames, with minimizing reprojection error as the optimization objective, to obtain a keyframe image sequence.

[0019] Optionally, in one embodiment of this application, the fusion module includes: a sorting unit, configured to determine the sorting of image frames in the keyframe image sequence based on the timestamps and poses of the keyframe image sequence, so as to determine the current viewpoint and multiple neighboring viewpoints that meet the preset condition of sufficient disparity information based on the sorting; an extraction unit, configured to extract features of the current viewpoint and multiple neighboring viewpoints from the image frames; an interaction unit, configured to calculate the interaction attention weights between the current viewpoint and neighboring viewpoints based on the features; and a fusion unit, configured to determine the fusion features based on the interaction attention weights.

[0020] Optionally, in one embodiment of this application, the fusion module includes: a prediction unit, configured to predict the density depth value of each pixel based on a pre-built feedforward neural network using fusion features, and determine the three-dimensional Gaussian components by combining the initial three-dimensional point cloud with uncertainty, thereby determining the three-dimensional Gaussian point cloud, wherein the loss function of the pre-built feedforward neural network is: , in, For loss function, These are the weight coefficients corresponding to the photometric consistency loss, depth supervision loss, smoothing regularization loss, and confidence correction loss, respectively. This represents the photometric consistency loss used to constrain the consistency of image reprojection results or image feature consistency under different viewpoints. This represents the depth supervision loss used to constrain the difference between predicted depth values ​​and true depth values, lidar depth values, or sparse depth priors. This represents the smooth regularization loss used to constrain the continuity of depth values ​​between adjacent pixels and suppress abrupt changes in depth. This represents the confidence correction loss used to constrain the prediction uncertainty or confidence level to match the actual depth prediction error.

[0021] Optionally, in one embodiment of this application, the reconstruction module includes: an identification unit, configured to identify a first pixel that does not meet the preset static co-point constraint condition based on the dense depth value, assign a dynamic attribute label to the first pixel, and identify a second pixel that meets the preset static co-point constraint condition, assign a static attribute label to the second pixel; and a determination unit, configured to determine a dynamic attribute mask based on the dynamic attribute label and the static attribute label.

[0022] Optionally, in one embodiment of this application, the reconstruction module includes: a description unit, configured to describe the spatiotemporal characteristics of a first pixel using motion speed, peak duration, and inherent motion frequency to determine dynamic features; a fitting unit, configured to fit a local plane using a second pixel to determine a static background; and a reconstruction unit, configured to reconstruct a three-dimensional scene representation based on the static background and dynamic features.

[0023] A third aspect of this application provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the dynamic three-dimensional reconstruction method for autonomous driving scenes as described in the above embodiments.

[0024] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described dynamic 3D reconstruction method for autonomous driving scenarios.

[0025] A fifth aspect of this application provides a computer program product, including a computer program that, when executed, is used to implement the above-described dynamic three-dimensional reconstruction method for autonomous driving scenarios.

[0026] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0027] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a flowchart of a dynamic 3D reconstruction method for an autonomous driving scene provided according to an embodiment of this application; Figure 2 This is a schematic diagram illustrating motion consistency verification according to an embodiment of this application; Figure 3 This is a flowchart of a dynamic 3D reconstruction method for an autonomous driving scene according to an embodiment of this application; Figure 4 This is a block diagram of a dynamic three-dimensional reconstruction device for an autonomous driving scene according to an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application.

[0028] Figure label: 10-Dynamic 3D Reconstruction Device for Autonomous Driving Scenarios; 100-Building Module, 200-Fusion Module, 300-Reconstruction Module; 501-Memory, 502-Processor, 503-Communication Interface. Detailed Implementation

[0029] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.

[0030] The following description, with reference to the accompanying drawings, outlines a method and apparatus for dynamic 3D reconstruction of autonomous driving scenes according to embodiments of this application. Addressing the technical problems mentioned in the background art, which primarily rely on adjusting transparency to achieve simple overlay of foreground and background, making it difficult to realistically reflect the occlusion relationships of multiple dynamic objects, and easily leading to blurred occlusion boundaries, geometric distortion, and insufficient rendering realism, thus affecting the accuracy and visual effects of autonomous driving scene modeling, this application provides a method for dynamic 3D reconstruction of autonomous driving scenes. This method aims to utilize multi-view fusion features, dense depth information, and dynamic / static separation in 3D Gaussian modeling to achieve high-precision 3D reconstruction of complex dynamic scenes in autonomous driving, improving the completeness and accuracy of 3D scene representation and its ability to express dynamic targets and occlusion relationships.

[0031] Specifically, Figure 1 This is a flowchart illustrating a dynamic 3D reconstruction method for an autonomous driving scene provided in an embodiment of this application.

[0032] like Figure 1 As shown, the dynamic 3D reconstruction method for autonomous driving scenarios includes the following steps: In step S101, a key frame image sequence that meets preset incremental optimization conditions is generated based on the images from the vehicle-mounted camera in the autonomous driving scenario, so as to construct the initial three-dimensional point cloud of the autonomous driving scenario.

[0033] It can be explained that as the vehicle-mounted camera continuously acquires new images, this embodiment can process the new images using a recursive visual front-end algorithm, estimate the current camera pose based on historical image information, and generate initial 3D point constraints by combining feature matching relationships. Furthermore, this embodiment can employ Bundle Adjustment (BA) to jointly optimize the camera pose and 3D point coordinates to construct the initial 3D point cloud for the autonomous driving scenario. However, if global BA optimization is performed for each acquired image frame, the number of image frames and 3D points involved in the optimization will continuously increase, leading to a rapid increase in the scale of the nonlinear optimization problem, making it difficult to meet the real-time requirements of autonomous driving scenarios. Therefore, this embodiment can employ a sliding window mechanism for incremental BA, jointly optimizing only newly added keyframes, associated historical keyframes, and their corresponding local 3D points within the sliding window, thereby reducing the scale of variables involved in the optimization and avoiding the real-time bottleneck caused by large-scale nonlinear optimization.

[0034] Therefore, in this embodiment, the new images acquired by the vehicle-mounted camera are first screened. Based on the image overlap rate, parallax size, number of feature matches, and camera pose change, key frame images that meet the incremental BA optimization conditions are determined. These images are then combined with historical key frames to construct a key frame image sequence for incremental BA optimization.

[0035] The preset incremental optimization conditions refer to the fact that image overlap rate, disparity magnitude, number of feature matches, and camera pose change all meet threshold requirements to ensure that newly added images can establish effective multi-view geometric constraints with historical keyframes and participate in incremental BA optimization. The initial 3D point cloud refers to a set of 3D point coordinates, where each point typically only contains position coordinates, but may also include attributes such as color, and is a discrete point set.

[0036] This application employs a sliding window mechanism to perform incremental Business Architecture (BA) processing on new images acquired by the vehicle-mounted camera. New images meeting certain conditions are added as keyframes to the optimization window, and the camera pose and local 3D point coordinates corresponding to the keyframes within the window are jointly optimized to obtain the optimized camera pose and initial 3D (3-Dimensional) point cloud. Because only data within a local window is optimized, compared to global BA, this effectively reduces the problem scale and improves the online processing efficiency and scalability of 3D modeling for autonomous driving scenarios.

[0037] As a specific example, generating a keyframe image sequence that meets preset incremental optimization conditions based on images from vehicle-mounted cameras in an autonomous driving scenario includes: detecting scale-invariant feature transformation feature points of the images to determine the two-dimensional features of the images; based on the two-dimensional features, selecting image frames that meet the preset incremental optimization conditions from the images according to at least one of image frame disparity, image overlap rate, and motion baseline; and based on the image frames, generating incremental optimization information of camera pose and sparse 3D point cloud with minimizing reprojection error as the optimization objective to obtain the keyframe image sequence.

[0038] Scale-invariant feature transform (SIN) feature points refer to local feature points detected in an image using the SIN algorithm that possess position, scale, and orientation information and are stable to changes in image scale, rotation, and a certain degree of illumination. Incremental optimization information refers to the optimization results information, such as camera pose updates and 3D point coordinate updates, generated relative to existing estimation results for newly added keyframes during incremental BA optimization.

[0039] First, in the embodiments of this application, in image frame I t Detect key SIFT (Scale-Invariant Feature Transform) feature points and extract descriptor subsets. F t : Among them, u i d represents the pixel coordinates of a feature point in an image. i This represents a 128-dimensional descriptor for SIFT, where feature points between frames are matched using the descriptor. N t This represents the total number of feature points.

[0040] Secondly, in this embodiment, key frame images in the input image are selected based on thresholds determined by the image frame disparity, image overlap rate, and motion baseline, and added to a key frame image set K. This set K is constructed in a first-in-first-out manner, retaining only the most recent 10 to 20 key frame images to reduce the number of frames involved in the calculation and avoid the computational load from increasing linearly over time.

[0041] Subsequently, embodiments of this application can construct a nonlinear optimization problem within a local sliding window, using the camera pose matrix T and sparse 3D points X. j To optimize the variables and minimize the reprojection error, the camera pose is represented by the homogeneous transformation matrix T. i It can be represented as: Among them, R i Let t represent the rotation matrix of the camera coordinate system relative to the reference coordinate system in the i-th frame. iSE(3) represents the translation vector of the camera coordinate system relative to the reference coordinate system in the i-th frame, and SE(3) represents the pose using a homogeneous transformation matrix belonging to a special three-dimensional Euclidean group.

[0042] Furthermore, embodiments of this application can construct a least squares problem, the structure of which is as follows: Where π is the camera projection equation, ρ is the robust kernel function, and K is the camera intrinsic parameter matrix. During the optimization process, this embodiment can use the LM (Levenberg-Marquardt) algorithm to iteratively optimize the camera pose and 3D points, ensuring the continuity and accuracy of the 3D reconstruction.

[0043] Based on feature matching results and constraints within a local window, this application's embodiments employ nonlinear least squares optimization to recursively update the camera pose, while simultaneously optimizing the positions of sparse 3D points. This enables incremental 3D reconstruction of continuous scenes, providing real-time and accurate scene geometric information for autonomous driving environment perception, dynamic scene understanding, and subsequent 3D reconstruction and decision planning.

[0044] In step S102, features of the autonomous driving scene under multiple perspectives are extracted from the keyframe image sequence to determine the fusion features. Based on the fusion features and the initial 3D point cloud, the density depth value of each pixel in the keyframe image sequence and the corresponding 3D Gaussian point cloud of the scene are determined.

[0045] In the embodiments of this application, the features may include, but are not limited to, Q (Query), K (Key), and V (Value). The query features, key features, and value features can be obtained from the input image features through feature mapping and are used to describe the feature association relationship between different spatial locations or different viewpoints.

[0046] Feature fusion refers to the feature representation obtained by fusing the features of the current keyframe with the image features of the neighboring keyframes. This fully utilizes the contextual information of the current viewpoint and the geometric constraint information of the neighboring viewpoints, thereby solving problems such as large depth prediction errors caused by occlusion areas, weak texture areas and depth ambiguity in complex scenes.

[0047] Dense depth value refers to the depth information corresponding to each pixel in an image, which is used to represent the distance between the scene point corresponding to the pixel and the camera.

[0048] 3D Gaussian point cloud refers to assigning Gaussian distribution parameters, such as position, covariance matrix, opacity, and color, to each point based on an initial 3D point cloud. This gives each point a certain spatial distribution range rather than an ideal geometric point, thus enabling a more continuous and realistic representation of the scene.

[0049] This application embodiment constructs a three-dimensional Gaussian point cloud, which can transform the initial three-dimensional point cloud from a discrete point representation into a continuous three-dimensional Gaussian distribution representation, thereby improving the ability to express the geometric structure and surface properties of the scene and providing a foundation for high-precision three-dimensional reconstruction and realistic scene rendering.

[0050] One possible approach is to extract features of an autonomous driving scene from a keyframe image sequence from multiple perspectives to determine fusion features. This includes: determining the order of image frames in the keyframe image sequence based on the timestamps and poses of the keyframe image sequence, and determining the current perspective and multiple neighboring perspectives that meet the preset condition of sufficient disparity information based on the order; extracting features of the current perspective and multiple neighboring perspectives from the image frames; calculating the interaction attention weights between the current perspective and neighboring perspectives based on the features; and determining the fusion features based on the interaction attention weights.

[0051] Among them, the condition of pre-setting sufficient disparity information can refer to the fact that the change between two viewpoints is large enough to make the position of the same scene point change significantly in different images, so as to recover the depth using triangulation; it can be determined based on the disparity between keyframes, the length of the motion baseline, and the amount of viewpoint change.

[0052] Interactive attention weight refers to the weight calculated based on the correlation between one feature (usually the Query feature of the current viewpoint) and another feature (usually the Key feature of a neighboring viewpoint) when interacting between different feature sources. It is used to measure the degree of important correlation between features from different viewpoints.

[0053] This application embodiment can sort the keyframe image set optimized by BA by timestamp and pose, and select several neighboring viewpoints with sufficient disparity information to the current keyframe as multi-view input. Simultaneously, this application embodiment can perform color normalization on multiple frames to unify image parameters, and construct a pyramid according to multi-scale proportions to obtain image features at different resolutions.

[0054] Subsequently, in this embodiment of the application, an initial feature vector e is constructed for each pixel. j (u,v) is obtained by concatenating image pixel values, ResNet (Residual Network) layer features, and camera pose encoder output, and serves as the input representation for the attention network. This encoding also includes prior camera pose information, facilitating the direct output of a 3D Gaussian point cloud.

[0055] Within each viewpoint, embodiments of this application can use a self-attention module to extract long-range contextual information, thereby enhancing the representation ability of local textures at different scales.

[0056] First, the embodiments of this application can use the standard Query / Key / Value mechanism to calculate attention weights: Where Q, K, and V are obtained by linear transformation of the input embedding e, d is the feature dimension, A represents the obtained attention weight matrix, and each element in the matrix is ​​used to represent the degree of correlation between different feature positions; O represents the attention output feature obtained by weighting and aggregating the value matrix V using the attention weight matrix A.

[0057] To achieve information complementarity between viewpoints, embodiments of this application may introduce a cross-attention module, enabling the query of the current viewpoint to interact with the key / value pairs of neighboring viewpoints to obtain fused features, thereby fusing multi-view geometry and texture information into the representation of the current pixel. Specifically, the cross-attention is weighted according to viewpoint weights and then superimposed onto the self-attention output.

[0058] Further, in one embodiment of this application, determining the dense depth value of each pixel in the keyframe image sequence and the corresponding 3D Gaussian point cloud of the scene based on fusion features and an initial 3D point cloud includes: predicting the dense depth value of each pixel using fusion features based on a pre-built feedforward neural network, and determining the 3D Gaussian component by combining the initial 3D point cloud with uncertainty, thereby determining the 3D Gaussian point cloud, wherein the loss function of the pre-built feedforward neural network is: , in, For loss function, These are the weight coefficients corresponding to the photometric consistency loss, depth supervision loss, smoothing regularization loss, and confidence correction loss, respectively. This represents the photometric consistency loss used to constrain the consistency of image reprojection results or image feature consistency under different viewpoints. This represents the depth supervision loss used to constrain the difference between predicted depth values ​​and true depth values, lidar depth values, or sparse depth priors. This represents the smooth regularization loss used to constrain the continuity of depth values ​​between adjacent pixels and suppress abrupt changes in depth. This represents the confidence correction loss used to constrain the prediction uncertainty or confidence level to match the actual depth prediction error.

[0059] It can be noted that the feedforward neural network may include, but is not limited to, a feature mapping layer, a non-linear activation layer, and an output mapping layer, which are used to perform non-linear transformation and feature enhancement on the fused features; the output mapping layer is connected to a depth regression head to predict the density depth value of each pixel in the keyframe based on the fused features.

[0060] Uncertainty is a quantitative parameter used to describe the reliability of dense depth values ​​or three-dimensional spatial position estimates. It is used to characterize the position error range of the corresponding three-dimensional point, and the spatial distribution range of the three-dimensional Gaussian components can be determined based on the uncertainty.

[0061] This application embodiment can input the fused pixel representation into the depth regression head to directly predict the depth value, thereby obtaining the pixel-to-3D projection: converting the pixel depth into a 3D point in the camera coordinate system. Where K is the camera intrinsic parameter, T cw The pose of the camera in world coordinates.

[0062] Subsequently, embodiments of this application can convert the 3D points and uncertainties of each pixel into 3D Gaussian components, such as per-pixel Gaussians or voxel-level Gaussians: each Gaussian is represented by the mean μ, covariance Σ, weight (or amplitude) α, and color and normal vector attributes c,n. During voxel aggregation, embodiments of this application can perform a weighted average on the set of pixels falling into the same voxel to improve the continuity and robustness of the 3D scene representation and reduce the impact of noise and abnormal depth values ​​on the 3D reconstruction results.

[0063] In the embodiments of this application, the feedforward neural network is trained on a large-scale dataset for multiple tasks. The loss function includes photometric consistency, depth supervision, smoothing regularization term and uncertainty correction term. The photometric term can promote cross-view geometric consistency, LiDAR (Light Detection and Ranging) data can be directly used for depth supervision, the smoothing term ensures the continuity of the depth field, and the uncertainty term encourages reasonable confidence output.

[0064] In step S103, based on the dense depth value, each point in the 3D Gaussian point cloud is assigned a dynamic attribute label or a static attribute label to determine the dynamic attribute mask, and a 3D scene representation containing static background and dynamic features is reconstructed based on the dynamic attribute mask.

[0065] Dynamic attribute labels or static attribute labels refer to classification information used to describe the motion state of scene objects corresponding to three-dimensional Gaussian components. Dynamic attribute labels can be used to identify objects with motion change characteristics, such as vehicles and pedestrians, while static attribute labels can be used to identify environmental objects with stable positions, such as roads and buildings.

[0066] Dynamic attribute masks are pixel-level binary masks generated by dynamic or static attribute tags, used to identify the region to which each pixel belongs in a keyframe image. For example, the mask value of pixels corresponding to moving targets such as vehicles and pedestrians can be set to 1, while the mask value of pixels corresponding to static environments such as roads and buildings can be set to 0.

[0067] The embodiments of this application can reconstruct the fixed environment and moving targets in the scene based on multiple frames of images, camera pose, depth information and target attribute information, forming a three-dimensional spatial representation that includes static environment structure and dynamic target change information. This can realistically reflect the occlusion relationship between multiple dynamic objects and between dynamic objects and static background, improving the integrity and accuracy of three-dimensional environment modeling in complex dynamic scenes.

[0068] In actual execution, based on the dense depth value, each point in the 3D Gaussian point cloud is assigned a dynamic attribute label or a static attribute label to determine the dynamic attribute mask. This includes: based on the dense depth value, identifying the first pixel that does not meet the preset static co-point constraint condition, assigning the first pixel a dynamic attribute label, and identifying the second pixel that meets the preset static co-point constraint condition, assigning the second pixel a static attribute label; and determining the dynamic attribute mask based on the dynamic attribute label and the static attribute label.

[0069] The preset static co-point constraint conditions refer to the requirement that the corresponding pixels of static scene points in multiple keyframe images should meet geometric consistency conditions, including but not limited to reprojection error, 3D position deviation less than a preset threshold, and depth consistency requirements.

[0070] As a specific example, for the same spatial point in a static area such as a road or building, its corresponding pixel position should remain consistent after camera pose transformation in different keyframes; if the condition is not met, it is determined that the pixel may belong to a dynamic attribute label.

[0071] It is understood that the embodiments of this application are responsible for distinguishing between dynamic features and static background of the dense 3DGS (3D Gaussian Splatting) point cloud and corresponding depth / confidence information output by the feedforward neural network, and outputting a point-level dynamic attribute mask.

[0072] Specifically, inter-frame motion consistency detection can be achieved by constructing inter-frame epipolar geometric constraints to quickly detect motion features. For moving points in space... Epipolar geometric relations such as Figure 2 As shown, , They are Time's up The position of the optical center of the camera at any given time, corresponding to and These represent the camera imaging planes at two different times, since the point... Motion occurs simultaneously with camera movement, therefore... The projection point at time also changed from the expected value. become Projection point There is a distance relative to the polar line. For projection distance error Detection can determine whether a point in space is moving.

[0073] As a point The polar lines satisfy the following relationship, where... Let be the essential matrix between frames, satisfying , where d is the inter-frame displacement vector, R is the inter-frame rotation matrix, and K is the camera projection matrix.

[0074]

[0075] point The distance to the polar line is Expressed as:

[0076] Subsequently, this embodiment of the application achieves dynamic feature discrimination by setting an allowable epipolar distance threshold. A binary dynamic mask is generated for each 3DGS point, where 1 represents a dynamic point and 0 represents a static background. Based on the generated mask, temporal smoothing is performed on the mask between consecutive keyframes using an exponentially weighted average to filter out some erroneous classifications. This reduces the impact of single-frame false detections, feature matching errors, or occlusion-related errors, improving the robustness of classification results for dynamic and static regions. This can be expressed as: M represents the image mask, and α is the weighting coefficient.

[0077] Optionally, in one embodiment of this application, reconstructing a 3D scene representation containing static background and dynamic features based on a dynamic attribute mask includes: using motion speed, peak duration, and inherent motion frequency to describe the spatiotemporal characteristics of a first pixel to determine dynamic features; using a second pixel to fit a local plane to determine the static background; determining the dynamic features; and reconstructing a 3D scene representation based on the static background and dynamic features.

[0078] Among them, fitting a local plane refers to establishing a local plane model based on the three-dimensional points corresponding to adjacent pixels in a static background area and calculating the deviation between the three-dimensional points and the local plane. This is done by using local geometric consistency constraints to optimize the reconstruction results, thereby maintaining the structural continuity of background areas such as roads and buildings and reducing surface breaks and geometric distortions caused by depth errors.

[0079] This application embodiment is responsible for optimizing and modeling the 3DGS point cloud after separating dynamic and static elements. Gaussian descriptions are established for both the dynamic target point group and the static background point group, and a continuous, smooth, dense 3D representation is output. This application embodiment may include two parts: descriptor construction and optimization. Static Gaussian modeling and optimization can include: optimizing using local plane fitting constraints under the 3DGS parameters generated by the feedforward network to maintain the continuity of the background structure. Where n is the normal vector of the fitting plane, and P0 is the reference point on the plane. r plane Represents a three-dimensional point X p X is relative to the residual of the fitted local plane. p Let be the spatial coordinates of the p-th static 3D point to be constrained. The objective of this optimization is to minimize the residuals of all static points with respect to the fitted plane, and to update the Gaussian mean and covariance using weighted methods.

[0080] Dynamic Gaussian modeling and optimization can include: using a dynamic Gaussian model to describe the spatiotemporal characteristics of each Gaussian point Xj in the dynamic point cloud set D extracted into the mask.

[0081] Where μ is the mean position of the point in three-dimensional space; Σj is the covariance matrix, describing the positional uncertainty; α is the weight or amplitude; v j t represents the velocity of motion. j θ represents the existence time of the Gaussian peak. j The natural motion frequency is modeled using a sine function, and the last three parameters are dynamic description parameters introduced based on the feedforward network. Let be the parameter of the j-th dynamic three-dimensional Gaussian component.

[0082] The embodiments of this application perform 3D Gaussian modeling on static background and dynamic features respectively, which can effectively separate and independently express static environment structure and dynamic target information, avoid interference of dynamic target movement on static scene reconstruction, and at the same time preserve the spatial position and change characteristics of dynamic targets, thereby improving the integrity, accuracy and temporal consistency of 3D scene representation in complex dynamic scenes.

[0083] The following is a specific example, such as Figure 3 As shown, the dynamic 3D reconstruction method for autonomous driving scenes according to embodiments of this application is further described; it may include the following steps: Step S301, front-end visual recursion.

[0084] First, this embodiment of the application acquires multi-view camera images at the vehicle end and processes the continuously acquired images using a recursive visual front-end algorithm. An incremental BA (Balanced Analytical Architecture) method is employed to jointly optimize the current frame and historical keyframes, forming an initial 3D estimate of the scene. This embodiment of the application can effectively reduce the computational scale of global BA through local incremental updates, avoiding the real-time bottleneck caused by large-scale nonlinear optimization, thereby improving online processing speed and scalability while ensuring accuracy.

[0085] Step S302: Output Gaussian point cloud from feedforward image inference.

[0086] In this embodiment, the keyframe image sequence optimized by BA is input into a feedforward neural network. This network achieves dense depth estimation through multi-view information fusion and further generates a high-quality 3D Gaussian point cloud representation. Dense depth inference not only enhances the integrity and geometric details of the point cloud but also provides a high-fidelity input foundation for subsequent dynamic and static modeling.

[0087] Step S303, motion consistency constraints achieve dynamic and static separation.

[0088] This application introduces a motion consistency constraint mechanism into point cloud data generated from consecutive multi-frame images. Embodiments of this application can construct multi-view geometric constraints based on pixel correspondences, depth values, and camera poses between adjacent frames. Pixels that do not meet static co-point constraints are identified and verified as dynamic points, distinguishing them from static regions that satisfy the constraints. This embodiment can achieve the separation of dynamic and static targets, effectively reducing interference and motion blur caused by dynamic objects in scene modeling. The separation results provide separate dynamic and static modeling for the backend, improving overall reconstruction accuracy.

[0089] Step S304: Optimize the backend 3DGS separately.

[0090] In the backend optimization stage, this embodiment utilizes static and dynamic modeling frameworks to handle dynamic and static features respectively. For static backgrounds, Gaussian background modeling is employed to enhance the stability and continuity of their 3D representation. For dynamic targets, a moving Gaussian model is used to fit and optimize their temporal trajectory by encoding motion parameters, thereby maintaining the spatiotemporal consistency of dynamic objects across different frames. Finally, this embodiment achieves unified optimization using vehicle-side computing resources, generating a 3D scene representation that simultaneously includes a high-fidelity static background and realistic dynamic features.

[0091] The dynamic 3D reconstruction method for autonomous driving scenes proposed in this application generates a 3D Gaussian point cloud by fusing features from multiple perspectives and combining the dense depth value corresponding to each pixel to separate the dynamic features and static background in the scene. Then, the corresponding 3D Gaussian representations are constructed and optimized separately. This avoids ghosting, trailing shadows caused by the movement of dynamic objects and static background pollution. At the same time, by fusing the spatial change information of dynamic targets and the stable structural information of the static environment, the distribution, spatial relationships and occlusion relationships of objects in complex autonomous driving scenes can be described more accurately, improving the integrity and geometric consistency of the 3D scene representation. In addition, the dynamic-static separation modeling can enhance the ability to express scene changes over time, providing a more reliable 3D environment model for autonomous driving environment perception, positioning and navigation and path planning.

[0092] Next, referring to the accompanying drawings, we describe the dynamic three-dimensional reconstruction device for autonomous driving scenarios proposed in the embodiments of this application.

[0093] Figure 4 This is a block diagram of an autonomous driving scene dynamic three-dimensional reconstruction device according to an embodiment of this application.

[0094] like Figure 4 As shown, the dynamic 3D reconstruction device 10 for autonomous driving scenarios includes: a construction module 100, a fusion module 200, and a reconstruction module 300.

[0095] Among them, the construction module 100 is used to generate a key frame image sequence that meets the preset incremental optimization conditions based on the images of the vehicle-end camera in the autonomous driving scenario, so as to construct the initial three-dimensional point cloud of the autonomous driving scenario. The fusion module 200 is used to extract features of the autonomous driving scene from multiple perspectives from the keyframe image sequence to determine the fusion features, and to determine the density depth value of each pixel in the keyframe image sequence and the corresponding three-dimensional Gaussian point cloud of the scene based on the fusion features and the initial three-dimensional point cloud. The reconstruction module 300 is used to assign dynamic or static attribute labels to each point in the 3D Gaussian point cloud based on dense depth values, in order to determine the dynamic attribute mask, and to reconstruct a 3D scene representation containing static background and dynamic features based on the dynamic attribute mask.

[0096] Optionally, in one embodiment of this application, the construction module 100 includes: a detection unit, a filtering unit, and a generation unit.

[0097] The detection unit is used to detect scale-invariant feature transformation feature points of the image in order to determine the two-dimensional features of the image.

[0098] The filtering unit is used to filter image frames that meet preset incremental optimization conditions from the image based on two-dimensional features, according to at least one of image frame disparity, image overlap rate and motion baseline.

[0099] The generation unit is used to generate incremental optimization information of camera pose and sparse 3D point cloud based on image frames, with the optimization objective of minimizing reprojection error, in order to obtain keyframe image sequences.

[0100] Optionally, in one embodiment of this application, the fusion module 200 includes: a sorting unit, an extraction unit, an interaction unit, and a fusion unit.

[0101] The sorting unit is used to determine the sorting of image frames in the keyframe image sequence based on the timestamp and pose of the keyframe image sequence, so as to determine the current viewpoint and multiple neighboring viewpoints that meet the preset condition of sufficient parallax information based on the sorting.

[0102] The extraction unit is used to extract features from the current viewpoint and multiple neighboring viewpoints from an image frame.

[0103] The interaction unit is used to calculate the interaction attention weights between the current viewpoint and neighboring viewpoints based on features.

[0104] The fusion unit is used to determine the fusion features based on the interaction attention weights.

[0105] Optionally, in one embodiment of this application, the fusion module 200 includes a prediction unit.

[0106] The prediction unit, based on a pre-built feedforward neural network, predicts the density depth value of each pixel using fused features, and determines the 3D Gaussian components by combining the initial 3D point cloud with uncertainty, thereby determining the 3D Gaussian point cloud. The loss function of the pre-built feedforward neural network is: , in, For loss function, These are the weight coefficients corresponding to the photometric consistency loss, depth supervision loss, smoothing regularization loss, and confidence correction loss, respectively. This represents the photometric consistency loss used to constrain the consistency of image reprojection results or image feature consistency under different viewpoints. This represents the depth supervision loss used to constrain the difference between predicted depth values ​​and true depth values, lidar depth values, or sparse depth priors. This represents the smooth regularization loss used to constrain the continuity of depth values ​​between adjacent pixels and suppress abrupt changes in depth. This represents the confidence correction loss used to constrain the prediction uncertainty or confidence level to match the actual depth prediction error.

[0107] Optionally, in one embodiment of this application, the reconstruction module 300 includes: an identification unit and a determination unit.

[0108] The identification unit is used to identify a first pixel that does not meet the preset static co-point constraint condition based on the dense depth value, assign a dynamic attribute label to the first pixel, and identify a second pixel that meets the preset static co-point constraint condition, assign a static attribute label to the second pixel.

[0109] The determination unit is used to determine the dynamic attribute mask based on the dynamic attribute tags and static attribute tags.

[0110] Optionally, in one embodiment of this application, the reconstruction module 300 includes: a description unit, a fitting unit, and a reconstruction unit.

[0111] The description unit is used to describe the spatiotemporal characteristics of the first pixel using motion speed, peak duration, and inherent motion frequency, in order to determine dynamic features.

[0112] A fitting unit is used to fit a local plane using the second pixel to determine a static background.

[0113] Reconstruction units are used to reconstruct a 3D scene representation based on a static background and dynamic features.

[0114] It should be noted that the foregoing explanation of the embodiment of the dynamic three-dimensional reconstruction method for autonomous driving scenarios also applies to the dynamic three-dimensional reconstruction device for autonomous driving scenarios in this embodiment, and will not be repeated here.

[0115] The dynamic 3D reconstruction device for autonomous driving scenes proposed in this application generates a 3D Gaussian point cloud by using fusion features from multiple perspectives and combining the dense depth value corresponding to each pixel to separate the dynamic features and static background in the scene. Then, the corresponding 3D Gaussian representations are constructed and optimized separately. This can avoid ghosting, trailing shadows caused by the movement of dynamic objects and static background pollution. At the same time, by fusing the spatial change information of dynamic targets and the stable structural information of static environment, it can more accurately describe the distribution of objects, spatial relationships and occlusion relationships in complex autonomous driving scenes, and improve the integrity and geometric consistency of the 3D scene representation.

[0116] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include: The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0117] When the processor 502 executes the program, it implements the dynamic three-dimensional reconstruction method for autonomous driving scenarios provided in the above embodiments.

[0118] Furthermore, electronic devices also include: Communication interface 503 is used for communication between memory 501 and processor 502.

[0119] The memory 501 is used to store computer programs that can run on the processor 502.

[0120] Memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0121] If the memory 501, processor 502, and communication interface 503 are implemented independently, then the communication interface 503, memory 501, and processor 502 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0122] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0123] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0124] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described dynamic 3D reconstruction method for autonomous driving scenarios.

[0125] This application also provides a computer program product, including a computer program that can run computer instructions. When the computer instructions are executed by a processor, they implement the dynamic three-dimensional reconstruction method for autonomous driving scenes provided in this application.

[0126] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0127] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0128] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0129] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0130] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or more of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0131] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0133] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

Claims

1. A method for dynamic 3D reconstruction of an autonomous driving scene, characterized in that, Includes the following steps: Based on the images from the vehicle-mounted camera in the autonomous driving scenario, a key frame image sequence that meets the preset incremental optimization conditions is generated to construct the initial three-dimensional point cloud of the autonomous driving scenario. Features of the autonomous driving scene from multiple perspectives are extracted from the keyframe image sequence to determine the fusion features. Based on the fusion features and the initial 3D point cloud, the density depth value of each pixel in the keyframe image sequence and the corresponding 3D Gaussian point cloud of the scene are determined. Based on the density depth value, each point in the 3D Gaussian point cloud is assigned a dynamic attribute label or a static attribute label to determine a dynamic attribute mask, and a 3D scene representation containing static background and dynamic features is reconstructed based on the dynamic attribute mask.

2. The method according to claim 1, characterized in that, The step of generating a keyframe image sequence that meets preset incremental optimization conditions based on images from vehicle-mounted cameras in an autonomous driving scenario includes: Detect scale-invariant feature transform feature points of the image to determine the two-dimensional features of the image; Based on the two-dimensional features, image frames that meet the preset incremental optimization conditions are selected from the image according to at least one of image frame disparity, image overlap rate, and motion baseline. Based on the image frames, with the goal of minimizing reprojection error, incremental optimization information for camera pose and sparse 3D point cloud is generated to obtain the keyframe image sequence.

3. The method according to claim 1, characterized in that, The step of extracting features of the autonomous driving scene from the keyframe image sequence from multiple perspectives to determine fused features includes: The order of image frames in the keyframe image sequence is determined based on the timestamp and pose of the keyframe image sequence, so as to determine the current viewpoint and multiple neighboring viewpoints that meet the preset condition of sufficient parallax information based on the order. Extract the features of the current viewpoint and multiple neighboring viewpoints from the image frame; Calculate the interaction attention weights between the current viewpoint and the neighboring viewpoints based on the features; The fusion features are determined based on the interaction attention weights.

4. The method according to claim 1, characterized in that, The step of determining the density depth value of each pixel in the keyframe image sequence and the corresponding 3D Gaussian point cloud of the scene based on the fusion features and the initial 3D point cloud includes: Based on a pre-built feedforward neural network, the dense depth value of each pixel is predicted using the fused features, and the three-dimensional Gaussian components are determined by combining the initial 3D point cloud with uncertainty, thereby determining the 3D Gaussian point cloud. The loss function of the pre-built feedforward neural network is: ， in, For loss function, These are the weight coefficients corresponding to the photometric consistency loss, depth supervision loss, smoothing regularization loss, and confidence correction loss, respectively. This represents the photometric consistency loss used to constrain the consistency of image reprojection results or image feature consistency under different viewpoints. This represents the depth supervision loss used to constrain the difference between predicted depth values ​​and true depth values, lidar depth values, or sparse depth priors. This represents the smooth regularization loss used to constrain the continuity of depth values ​​between adjacent pixels and suppress abrupt changes in depth. This represents the confidence correction loss used to constrain the prediction uncertainty or confidence level to match the actual depth prediction error.

5. The method according to claim 1, characterized in that, The step of assigning dynamic or static attribute labels to each point in the 3D Gaussian point cloud based on the dense depth value to determine the dynamic attribute mask includes: Based on the density depth value, a first pixel that does not meet the preset static co-point constraint condition is identified, and the dynamic attribute label is assigned to the first pixel. A second pixel that meets the preset static co-point constraint condition is identified, and the static attribute label is assigned to the second pixel. The dynamic attribute mask is determined based on the dynamic attribute tag and the static attribute tag.

6. The method according to claim 5, characterized in that, The process of reconstructing a 3D scene representation containing static background and dynamic features based on the dynamic attribute mask includes: The spatiotemporal characteristics of the first pixel are described using motion speed, peak duration, and inherent motion frequency to determine the dynamic features. The static background is determined by fitting a local plane using the second pixel. The 3D scene representation is reconstructed based on the static background and the dynamic features.

7. A dynamic three-dimensional reconstruction device for autonomous driving scenarios, characterized in that, include: The construction module is used to generate a key frame image sequence that meets preset incremental optimization conditions based on the images from the vehicle-mounted camera in the autonomous driving scenario, so as to construct the initial three-dimensional point cloud of the autonomous driving scenario. The fusion module is used to extract features of the autonomous driving scene from the keyframe image sequence from multiple perspectives to determine the fusion features, and to determine the density depth value of each pixel in the keyframe image sequence and the corresponding three-dimensional Gaussian point cloud based on the fusion features and the initial three-dimensional point cloud. The reconstruction module is used to assign dynamic or static attribute labels to each point in the 3D Gaussian point cloud based on the dense depth value, so as to determine the dynamic attribute mask, and reconstruct a 3D scene representation containing static background and dynamic features according to the dynamic attribute mask.

8. An electronic device, characterized in that, include: The system includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the program to implement the dynamic three-dimensional reconstruction method for autonomous driving scenes as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, The program is executed by the processor to implement the dynamic three-dimensional reconstruction method for autonomous driving scenarios as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, The computer program is executed to implement the dynamic three-dimensional reconstruction method for autonomous driving scenarios as described in any one of claims 1-6.

Images