Dynamic scene modeling method and device and electronic equipment

Abstract

The invention provides a dynamic scene modeling method and device and electronic equipment, and the method comprises the steps: carrying out the pedestrian target detection of an image sequence after the three-dimensional point cloud of a scene is aligned with the image sequence, and generating a bounding box; separating a dynamic pedestrian point cloud set and a static background point cloud set; encoding the static background point cloud set into Gaussian point parameters by using a Hash network, and initializing a Gaussian point cluster based on the dynamic pedestrian point cloud set to construct a Gaussian scene representation model; carrying out dynamic modeling and physical shielding effect simulation on a pedestrian movement track in the Gaussian scene representation model by adopting Kalman filtering so as to update the Gaussian scene representation model; performing heterogeneous feature fusion on the static region and the dynamic region of the updated Gaussian scene representation model to generate fusion features; and generating a dynamic scene image based on the fusion features and the multi-view camera parameters. The dynamic scene three-dimensional reconstruction method can solve the problem that the credibility, the real-time performance and the fluency of an existing dynamic scene three-dimensional reconstruction technical scheme are not ideal.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a dynamic scene modeling method, apparatus, and electronic device. Background Technology

[0002] Current dynamic scene 3D reconstruction technologies have significant limitations in applications such as autonomous driving simulation and smart city management. Traditional methods that fuse LiDAR and visual data struggle to effectively distinguish moving targets from static backgrounds, leading to incorrect fusion of dynamic objects and static environments, resulting in geometric distortion and texture anomalies. Furthermore, existing technologies oversimplify the modeling of dynamic targets such as pedestrians, failing to capture fine-grained behavior and easily causing trajectory distortion in high-density scenes. Mainstream rendering schemes are inefficient: voxel mesh methods sacrifice detail fidelity, while implicit representations such as neural radiation fields consume enormous computational resources, making them unsuitable for real-time applications. These shortcomings severely restrict the reliability, real-time performance, and smoothness of applications related to dynamic scene 3D reconstruction technologies. Summary of the Invention

[0003] In view of this, it is necessary to provide a dynamic scene modeling method, device and electronic device to solve the technical problems that the reliability, real-time performance and smoothness of existing dynamic scene 3D reconstruction technology solutions are not ideal.

[0004] To address the aforementioned problems, in a first aspect, the present invention provides a dynamic scene modeling method, comprising: After aligning the 3D point cloud of the scene acquired by the LiDAR with the image sequence acquired by the multi-view camera, pedestrian target detection is performed on the image sequence to generate bounding boxes; Based on the bounding box, the dynamic pedestrian point cloud and the static background point cloud in the scene's 3D point cloud are separated. A hash network is used to encode the static background point cloud into Gaussian point parameters, and the spatial distribution of Gaussian point clusters is initialized based on the dynamic pedestrian point cloud to construct a Gaussian scene representation model. Kalman filtering is used to dynamically model and simulate physical occlusion effects on the pedestrian trajectory in the Gaussian scene representation model in order to update the Gaussian scene representation model. Heterogeneous features are fused between the static and dynamic regions of the updated Gaussian scene representation model to generate fused features; Based on the fused features and multi-view camera parameters, dynamic scene images are generated.

[0005] In one possible implementation, the 3D point cloud of the scene acquired by LiDAR is aligned with the image sequence acquired by multi-view cameras, including: Determine the calibration matrix between the lidar and the multi-view camera; The calibration matrix parameters are optimized based on the iterative nearest point algorithm, and the scene 3D point cloud and the image sequence are aligned by homogeneous transformation based on the optimized calibration matrix.

[0006] In one possible implementation, pedestrian target detection is performed on the image sequence to generate bounding boxes, including: Pedestrian target detection is performed on the image sequence based on the YOLO network to generate original bounding boxes; The Mask R-CNN network performs pixel-level instance segmentation within the original bounding box, generating bounding boxes carrying binary masks.

[0007] In one possible implementation, the pedestrian trajectory in the Gaussian scene representation model is dynamically modeled and the physical occlusion effect is simulated using Kalman filtering to update the Gaussian scene representation model, including: The parameters of Gaussian points corresponding to the static background point cluster in the Gaussian scene representation model are frozen. Based on the spatial distribution of Gaussian point clusters and the pedestrian motion law model, the pedestrian motion speed is predicted by combining the Kalman filter method, and the position of the Gaussian points is updated based on the predicted pedestrian motion speed. The covariance matrix of Gaussian points is dynamically expanded based on the predicted pedestrian movement speed to simulate motion blur effect and dynamically adjust the occlusion perception transparency.

[0008] In one possible implementation, the occlusion-aware transparency is dynamically adjusted, including: When the depth of the target Gaussian point is less than the preset occlusion threshold, the occlusion perception transparency is reduced proportionally to simulate visual occlusion.

[0009] In one possible implementation, the dynamic scene modeling method also includes: When updating the Gaussian scene representation model, a sparse gradient update mechanism is used to reset the gradient of the Gaussian point parameters corresponding to the static background point cloud to zero.

[0010] In one possible implementation, heterogeneous feature fusion is performed on the static and dynamic regions of the updated Gaussian scene representation model to generate fused features, including: In the static region of the updated Gaussian scene representation model, Gaussian point geometric features and color features generated by Gaussian rendering interpolation are fused together, while in the dynamic region, pedestrian motion trajectory features and texture features are fused together to generate fused features.

[0011] In one possible implementation, a dynamic scene image is generated based on the fused features and multi-view camera parameters, including: Based on the fusion features and multi-view camera parameters, a material base color is generated; Dynamic scene images are generated based on the material's base color and the specular component used to simulate the reflection from a camera lens.

[0012] Secondly, the present invention also provides a dynamic scene modeling device, comprising: The pedestrian detection module is used to perform pedestrian target detection on the image sequence after aligning the scene 3D point cloud collected by LiDAR with the image sequence collected by multi-view cameras, and generate bounding boxes. The dynamic-static separation module is used to separate the dynamic pedestrian point cloud and the static background point cloud in the scene 3D point cloud based on the bounding box; The model building module is used to encode the static background point cloud into Gaussian point parameters using a hash network, and initialize the spatial distribution of Gaussian point clusters based on the dynamic pedestrian point cloud to construct a Gaussian scene representation model. The model update module is used to dynamically model the pedestrian motion trajectory in the Gaussian scene representation model using Kalman filtering and simulate the physical occlusion effect, so as to update the Gaussian scene representation model. The feature fusion module is used to perform heterogeneous feature fusion on the static and dynamic regions of the updated Gaussian scene representation model to generate fused features; The scene generation module is used to generate dynamic scene images based on the fused features and multi-view camera parameters.

[0013] Thirdly, the present invention also provides an electronic device, including a memory and a processor, wherein, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps of the dynamic scene modeling method as described in any of the preceding claims.

[0014] The beneficial effects of the above implementation are as follows: The dynamic scene modeling method, apparatus, and electronic device provided by this invention can eliminate spatiotemporal errors by aligning point clouds with image sequences, providing a foundation for subsequent separation of dynamic pedestrian point clouds and static background point clouds. Furthermore, this invention uses a hash network to encode the static background point cloud into Gaussian point parameters and initializes Gaussian point clusters based on the dynamic pedestrian point cloud. Based on the separated dynamic and static point cloud data, it aims to construct a physically reasonable mixed Gaussian scene representation, supporting kinematic modeling of dynamic targets, reducing computational resources, accelerating feature querying and rendering, significantly shortening model construction time and improving efficiency, thereby meeting real-time requirements and improving smoothness. The dynamic-static decoupled Gaussian scene representation model can effectively distinguish moving targets from static backgrounds, avoiding geometric distortion and texture abnormalities caused by incorrect fusion of dynamic objects and static environments.

[0015] Furthermore, by employing Kalman filtering for dynamic modeling of pedestrian trajectories and simulating physical occlusion effects, high-precision physically compliant modeling of pedestrian trajectories is achieved. This outputs high-precision dynamic scene images, i.e., high-precision dynamic scene models, which can support real-time interactive visualization in autonomous driving simulation and smart city management. It can capture fine-grained behaviors, avoid trajectory distortion problems in high-density scenes, and improve model reliability. Therefore, this invention can solve the technical problems of unsatisfactory reliability, real-time performance, and smoothness in existing dynamic scene 3D reconstruction technologies. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart of an embodiment of the dynamic scene modeling method provided by the present invention; Figure 2 A flowchart illustrating another embodiment of the dynamic scene modeling method provided by the present invention; Figure 3 A schematic diagram of an embodiment of the dynamic scene modeling device provided by the present invention; Figure 4 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0019] In the description of the embodiments of this application, unless otherwise stated, "a plurality of" means two or more.

[0020] In this embodiment of the invention, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, apparatus, product or device that includes a series of steps or modules is not necessarily limited to those steps or modules that are explicitly listed, but may include other steps or modules that are not explicitly listed or that are inherent to such process, method, product or device.

[0021] The naming or numbering of steps in the embodiments of the present invention does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering. The execution order of the named or numbered process steps can be changed according to the technical purpose to be achieved, as long as the same or similar technical effect can be achieved.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] This invention provides a dynamic scene modeling method, apparatus, and electronic device, which are described below.

[0024] This invention provides a dynamic scene modeling method, which can be implemented by executing an application on a terminal or server. The terminal can be a computer or a vehicle-mounted terminal, and the server can be an edge server or a cloud server, such as... Figure 1 As shown, the method includes: S101. After aligning the 3D point cloud of the scene acquired by the lidar with the image sequence acquired by the multi-view camera, pedestrian target detection is performed on the image sequence to generate bounding boxes.

[0025] Understandably, the LiDAR and multi-view camera acquire data synchronously, using calibration parameters to achieve precise projection alignment of point cloud coordinates to pixel coordinates in the image sequence, eliminating spatiotemporal misalignment errors. This step, based on the control signal of the hardware synchronization trigger device, aims to achieve sub-millimeter-level spatiotemporal alignment of data acquired by the LiDAR and multi-view camera, providing a geometric basis for dynamic-static separation.

[0026] S102. Based on the bounding box, separate the dynamic pedestrian point cloud set and the static background point cloud set in the scene's 3D point cloud.

[0027] Understandably, spatiotemporally aligned multimodal data aims to accurately separate dynamic pedestrian point clouds through visual detection and geometric mapping, thereby resolving depth ambiguity caused by occlusion.

[0028] S103. The static background point cloud is encoded into Gaussian point parameters using a hash network, and the spatial distribution of Gaussian point clusters is initialized based on the dynamic pedestrian point cloud to construct a Gaussian scene representation model.

[0029] It is understandable that a Gaussian scene representation model with decoupled dynamic and static elements is constructed (hereinafter referred to as Gaussian representation model): the static background directly generates Gaussian unit parameters (Gaussian point parameters) based on the point cloud coordinates, while the dynamic pedestrian initializes the spatial distribution of Gaussian point clusters (hereinafter referred to as Gaussian clusters) based on the three-dimensional size of the detection box; This step, based on separate dynamic and static point cloud data, aims to construct a physically plausible mixture of Gaussian scene representation to support kinematic modeling of dynamic targets.

[0030] S104. The pedestrian motion trajectory in the Gaussian scene representation model is dynamically modeled and the physical occlusion effect is simulated using Kalman filtering to update the Gaussian scene representation model.

[0031] It is understandable that a dynamic target time-series optimization mechanism is designed: Kalman filtering is used to predict pedestrian movement speed, Gaussian cell positions are iteratively updated to simulate trajectory continuity, and the covariance matrix is ​​dynamically expanded to express motion ambiguity effects.

[0032] S105. Perform heterogeneous feature fusion on the static and dynamic regions of the updated Gaussian scene representation model to generate fused features.

[0033] It is understandable that the static area refers to the area corresponding to the static background point cluster, while the dynamic area refers to the area corresponding to the dynamic pedestrian point cluster.

[0034] Implement a heterogeneous feature fusion strategy: fuse geometric coding features and color interpolation features for static regions, and fuse trajectory temporal features and texture extraction features for dynamic regions to form a unified input vector, which is the fused feature.

[0035] S106. Based on the fusion features and multi-view camera parameters, generate dynamic scene images.

[0036] Understandably, this process involves using a view-aware multilayer perceptron to decompose the material's primary color and specular reflectance components, combining this with a differentiable rendering pipeline to output dynamic scene images, and then using a multi-objective loss function to optimize the reconstruction's physical plausibility. This step, based on fused features and camera parameters, aims to generate high-fidelity rendered images using a physically based lighting model.

[0037] In some embodiments, aligning the 3D point cloud of the scene acquired by the LiDAR with the image sequence acquired by the multi-view camera includes: Determine the calibration matrix between the lidar and the multi-view camera; The calibration matrix parameters are optimized based on the iterative nearest point algorithm, and the scene 3D point cloud and the image sequence are aligned by homogeneous transformation based on the optimized calibration matrix.

[0038] It is understandable that a hardware synchronization triggering device controls the LiDAR and multi-view high-definition camera system to simultaneously acquire 3D point cloud data of the scene. and image sequences Using a pre-calibrated lidar-camera calibration matrix To achieve data alignment, the homogeneous transformation form is as follows:

[0039] in The rotation matrix describes the rotational transformation between the sensor coordinate systems; Let be the translation vector, representing the offset of the coordinate system origin. Alignment accuracy is constrained by reprojection error.

[0040] Here This is a 3D to 2D projection function. To preset the alignment error threshold, For point cloud coordinates, This step uses the iterative nearest neighbor (ICP) algorithm to optimize calibration parameters, ensuring spatial consistency of the dynamic target in the point cloud and image, thus providing a geometric basis for subsequent dynamic-static separation. If the error exceeds a threshold, a real-time calibration compensation module is triggered, employing an incremental correction strategy based on feature matching.

[0041] In some embodiments, pedestrian target detection is performed on the image sequence to generate bounding boxes, including: The image sequence is used to detect pedestrian targets based on the YOLO (You Only Live Once) network to generate original bounding boxes; The Mask R-CNN (Region-Based Convolutional Neural Networks) network is used to perform pixel-level instance segmentation within the original bounding box, generating bounding boxes carrying binary masks.

[0042] It is understandable that high-confidence bounding boxes are generated based on the YOLOv7 network:

[0043] As a confidence threshold, its backbone network extracts multi-scale features through a cross-stage local connectivity structure, and a path aggregation mechanism fuses deep and shallow semantic information to output bounding box coordinates. x min 、y min 、x max 、x minUsing the Mask R-CNN network in the bounding box Perform pixel-level instance segmentation to generate a binary mask. The model is achieved through... The operation preserves the details of the human body outline and solves the problem of blurred boundaries caused by occlusion.

[0044] In some embodiments, the pedestrian motion trajectory in the Gaussian scene representation model is dynamically modeled and the physical occlusion effect is simulated using Kalman filtering to update the Gaussian scene representation model, including: The parameters of Gaussian points corresponding to the static background point cluster in the Gaussian scene representation model are frozen. Based on the spatial distribution of Gaussian point clusters and the pedestrian motion law model, the pedestrian motion speed is predicted by combining the Kalman filter method, and the position of the Gaussian points is updated based on the predicted pedestrian motion speed. The covariance matrix of Gaussian points is dynamically expanded based on the predicted pedestrian movement speed to simulate motion blur effect and dynamically adjust the occlusion perception transparency.

[0045] It is understandable to implement a dynamic-static difference update strategy: Static parameter freezing: setting gradient mask Reverse propagation of updates is prohibited.

[0046] Dynamic position iteration: updating the Gaussian point position based on Kalman filter prediction speed.

[0047] Motion fuzzy simulation: Expanding the covariance matrix along the velocity direction:

[0048] coefficient With velocity modulus Adaptive, high-speed targets produce a greater blurring effect.

[0049] In some embodiments, dynamically adjusting the occlusion perception transparency includes: When the depth of the target Gaussian point is less than the preset occlusion threshold, the occlusion perception transparency is reduced proportionally to simulate visual occlusion.

[0050] Understandably, the occlusion perception transparency is dynamically adjusted based on depth testing.

[0051] The attenuation coefficient is used to simulate visual occlusion when the target point depth is less than the occlusion threshold.

[0052] In some embodiments, the dynamic scene modeling method further includes: When updating the Gaussian scene representation model, a sparse gradient update mechanism is used to reset the gradient of the Gaussian point parameters corresponding to the static background point cloud to zero.

[0053] It is understandable that a sparse gradient update mechanism is adopted:

[0054] in This is a dynamic region binary mask. This operation makes the gradient of static parameters zero, which greatly reduces the computational load.

[0055] In some embodiments, heterogeneous feature fusion is performed on the static and dynamic regions of the updated Gaussian scene representation model to generate fused features, including: In the static region of the updated Gaussian scene representation model, Gaussian point geometric features and color features generated by Gaussian rendering interpolation are fused together, while in the dynamic region, pedestrian motion trajectory features and texture features are fused together to generate fused features.

[0056] It is understandable that a heterogeneous feature fusion strategy is implemented: static regions fuse geometric coding features and color interpolation features, and dynamic regions fuse trajectory temporal features and texture extraction features to form a unified input vector; This step leverages the differences in physical properties between static and dynamic regions, aiming to fuse geometric, textural, and motion features to enhance rendering realism. This step covers the following main aspects: Geometric and color features in static regions:

[0057] in Generated via Gaussian rendering interpolation:

[0058] The color of a Gaussian distribution function is weighted and fused from the colors of neighboring Gaussian points.

[0059] Fusing trajectory temporal features and texture features in dynamic regions:

[0060] The LSTM (Long Short-Term Memory) module extracts pedestrian historical trajectory features. The hidden state stores the dynamic characteristics of motion.

[0061] From the current frame image I k detection box B kHigh-resolution textures are extracted from the interior, and bilinear interpolation preserves spatial details.

[0062] In some embodiments, generating dynamic scene images based on the fused features and multi-view camera parameters includes: Based on the fusion features and multi-view camera parameters, a material base color is generated; Dynamic scene images are generated based on the material's base color and the specular component used to simulate the reflection from a camera lens.

[0063] Understandably, fusion features and camera angle Input viewpoint dependency Decompose the base color and highlight components:

[0064] Basic colors The inherent color of the material (viewpoint independent), highlight Simulate viewpoint-dependent specular reflections. Synthesize dynamic scene images using volumetric rendering.

[0065] Multi-objective loss function: Joint optimization of static reconstruction and dynamic trajectory physical rationality:

[0066] Color consistency loss: Constraints on rendered images With real images Pixel-level differences:

[0067] The dynamic scene modeling method provided by this invention can be referenced. Figure 2 This is a dynamic 3D Gaussian splash reconstruction and dense pedestrian trajectory modeling method based on multimodal data fusion, which includes the following steps: The S201 uses a hardware-synchronized LiDAR and a multi-view camera system to simultaneously acquire 3D point cloud data and high-definition image sequences of the scene. It uses calibration parameters to achieve precise projection alignment from point cloud coordinates to pixel coordinates, eliminating spatiotemporal misalignment errors. This step, based on the control signals of a hardware synchronization triggering device, aims to achieve sub-millimeter-level spatiotemporal alignment of data acquired by the LiDAR and multi-view cameras, providing a geometric basis for dynamic-static separation. This step covers the following main contents: The LiDAR and multi-view high-definition camera system are controlled by a hardware synchronization triggering device to synchronously acquire 3D point cloud data of the scene. and image sequences Using a pre-calibrated lidar-camera calibration matrix To achieve data alignment, the homogeneous transformation form is as follows:

[0068] in The rotation matrix describes the rotational transformation between the sensor coordinate systems; Let be the translation vector, representing the offset of the coordinate system origin. Alignment accuracy is constrained by reprojection error.

[0069] Here This is a 3D to 2D projection function. To preset the alignment error threshold, For point cloud coordinates, The corresponding pixel coordinates in the image are used. This step optimizes the calibration parameters using the Iterative Closest Point (ICP) algorithm to ensure spatial consistency of dynamic targets in the point cloud and image, providing a geometric basis for subsequent dynamic-static separation. If the error exceeds the threshold, a real-time calibration compensation module is triggered, employing an incremental correction strategy based on feature matching. In step S202, a YOLOv7 and Mask R-CNN cascaded network is used to perform pedestrian target detection on the image sequence, generating high-confidence bounding boxes with masks, and separating the dynamic pedestrian point cloud through geometric inverse projection. This step, based on spatiotemporally aligned multimodal data, aims to accurately separate dynamic pedestrian point clouds through visual detection and geometric mapping, resolving depth ambiguity caused by occlusion. This step covers the following main components: Pedestrian object detection is achieved using a cascaded deep learning framework: Step 2.1: Generate high-confidence bounding boxes based on the YOLOv7 network:

[0070] As the confidence threshold, its backbone network extracts multi-scale features through a cross-stage local connectivity structure, and the path aggregation mechanism fuses deep and shallow semantic information to output bounding box coordinates. Step 2.2: Utilize the Mask R-CNN network to define the bounding box coordinates. Perform pixel-level instance segmentation to generate a binary mask. The model is achieved through... The operation preserves the details of the human body contour and resolves the boundary blurring problem caused by occlusion. Step 2.3: Using the inverse projection function. Mapping the 2D detection results (i.e., bounding boxes with masks) to 3D space to separate dynamic pedestrian point cloud subsets:

[0071] A static background dot cloud is This process relies on the S201 calibration matrix to ensure projection geometry consistency and avoid dynamic target misfusion caused by depth ambiguity.

[0072] S203, Construct a Gaussian representation model that decouples static and dynamic elements: For static backgrounds, Gaussian unit parameters (Gaussian point parameters) are directly generated based on point cloud coordinates, while for dynamic pedestrians, the spatial distribution of Gaussian clusters (Gaussian point clusters) is initialized based on the three-dimensional dimensions of the detection box. This step, based on separate static and dynamic point cloud data, aims to construct a physically plausible Gaussian mixture scene representation to support kinematic modeling of dynamic targets. This step covers the following main components: Step 3.1: Static background modeling: Directly based on Generate Gaussian point parameters. Location Inherited point cloud coordinates; covariance matrix Calculate the local point cloud distribution characteristics using Principal Component Analysis (PCA):

[0073] in This is the eigenvector matrix (main direction). The eigenvalue diagonal matrix (distribution scale) is obtained by SVD decomposition of the covariance matrix of a local neighborhood of points with a predetermined radius. Transparency Initialized to fixed values ​​to balance rendering contribution with background visibility. Geometric features are encoded into 128-dimensional vectors using a hash grid.

[0074] Dynamic pedestrian modeling: Based on the detection box (bounding box) Initialize the Gaussian point cluster. Location Calculated from the center of the bounding box:

[0075] in Given the set of point cloud depth values ​​within the bounding box, median filtering is used to mitigate ground noise interference. The covariance matrix is ​​diagonalized as follows:

[0076] Characterizes the spatial distribution range of pedestrians along the width, height, and depth axes. Kalman filtering is used to predict movement speed. :

[0077] in This is the state transition matrix (uniform velocity model). To control the input matrix, S204 is the state vector of the previous moment. Design a dynamic target time-series optimization mechanism: fuse Kalman filter to predict pedestrian movement speed, iteratively update Gaussian cell position to simulate trajectory continuity, and dynamically expand the covariance matrix to express motion ambiguity effect; This step, based on the kinematic characteristics of dynamic targets, aims to achieve trajectory continuity modeling and physical occlusion effect simulation through temporal optimization. This step covers the following main contents: Implement a dynamic / static difference update strategy: Static parameter freezing: setting gradient mask Reverse propagation of updates is prohibited.

[0078] Dynamic position iteration: updating the Gaussian point position based on Kalman filter prediction speed.

[0079] Motion fuzzy simulation: Expanding the covariance matrix along the velocity direction:

[0080] coefficient With velocity modulus Adaptive, high-speed targets produce a greater blurring effect.

[0081] Occlusion-aware transparency: dynamically adjusted based on depth testing.

[0082] The attenuation coefficient is used to proportionally reduce transparency when the target point depth is less than the occlusion threshold, simulating visual occlusion. A sparse gradient update mechanism is employed.

[0083] in For dynamic region binary masking, this operation reduces the gradient of static parameters to zero, significantly reducing the computational load. S205, implement a heterogeneous feature fusion strategy: static regions fuse geometric coding features and color interpolation features, dynamic regions fuse trajectory temporal features and texture extraction features, forming a unified input vector; This step leverages the differences in physical properties between static and dynamic regions, aiming to fuse geometric, textural, and motion features to enhance rendering realism. This step covers the following main aspects: Step 5.1: Geometric and color features in the static region:

[0084] in Generated via Gaussian rendering interpolation:

[0085] The color of a Gaussian distribution function is weighted and fused from the colors of neighboring Gaussian points.

[0086] Step 5.1: Fuse trajectory temporal features and texture features in the dynamic region:

[0087] The LSTM (Long Short-Term Memory) module extracts pedestrian historical trajectory features. The hidden state stores the dynamic characteristics of motion.

[0088] From the current frame image detection box High-resolution textures are extracted internally, and bilinear interpolation preserves spatial details. The S206 uses a view-aware multilayer perceptron to decompose the material's primary color and specular reflection components, combines a differentiable rendering pipeline to output dynamic scene images, and optimizes the physical plausibility of the reconstruction using a multi-objective loss function.

[0089] This step, based on fused features and camera parameters, aims to generate high-fidelity rendered images using a physically based lighting model. This step covers the following main components: Fusion features and camera angle Input viewpoint dependency Decompose the base color and highlight components:

[0090] Basic colors The inherent color of the material (viewpoint independent), highlight Simulate viewpoint-dependent specular reflections. Synthesize dynamic scene images using volumetric rendering.

[0091] Multi-objective loss function: Joint optimization of static reconstruction and dynamic trajectory physical rationality:

[0092] Color consistency loss: Constraints on rendered images With real images Pixel-level differences:

[0093] Geometric alignment loss: Matching Gaussian points with the spatial distribution of LiDAR point clouds using chamfer distance.

[0094] Trajectory smoothing loss: penalizes abrupt changes in the dynamic target position.

[0095] Motion fuzzy constraints: controlling the rate of change of the covariance matrix.

[0096] The RAdam optimizer is used to dynamically update parameters, and the regularization term improves the model's generalization ability.

[0097] In summary, the dynamic 3D Gaussian splash reconstruction framework proposed in this invention achieves significant improvements: it optimizes scene initialization accuracy by utilizing geometric priors from LiDAR point clouds, and accelerates feature querying and rendering by combining multi-resolution hash grid encoding, greatly shortening model construction time and improving efficiency; through a dynamic-static decoupling strategy and a sparse gradient update mechanism, it effectively eliminates motion target trailing artifacts, achieving high-precision physical compliance modeling of pedestrian trajectories. The high-precision dynamic scene model output by this system supports real-time interactive visualization in autonomous driving simulation and smart city management, providing reliable 3D environmental information for critical tasks; simultaneously, its excellent scene adaptability, relying on heterogeneous feature fusion and occlusion compensation mechanisms, maintains reconstruction robustness under complex lighting and occlusion conditions, powerfully promoting the practical application of dynamic 3D reconstruction technology.

[0098] like Figure 3 As shown, the present invention also provides a dynamic scene modeling device 300, comprising: The pedestrian detection module 301 is used to perform pedestrian target detection on the image sequence after aligning the scene 3D point cloud collected by the lidar with the image sequence collected by the multi-view camera, and generate bounding boxes. The dynamic-static separation module 302 is used to separate the dynamic pedestrian point cloud set and the static background point cloud set in the scene 3D point cloud based on the bounding box; The model building module 303 is used to encode the static background point cloud into Gaussian point parameters using a hash network, and initialize the spatial distribution of Gaussian point clusters based on the dynamic pedestrian point cloud to build a Gaussian scene representation model. The model update module 304 is used to perform dynamic modeling and physical occlusion effect simulation on the pedestrian motion trajectory in the Gaussian scene representation model using Kalman filtering, so as to update the Gaussian scene representation model. The feature fusion module 305 is used to perform heterogeneous feature fusion on the static and dynamic regions of the updated Gaussian scene representation model to generate fused features. The scene generation module 306 is used to generate dynamic scene images based on the fused features and multi-view camera parameters.

[0099] The dynamic scene modeling device provided in the above embodiments can realize the technical solutions described in the above dynamic scene modeling method embodiments. The specific implementation principles of each module or unit can be found in the corresponding content in the above dynamic scene modeling method embodiments, which will not be repeated here.

[0100] like Figure 4 As shown, the present invention also provides an electronic device 400. The electronic device 400 includes a processor 401, a memory 402, and a display 403. Figure 4 Only some components of the electronic device 400 are shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead.

[0101] In some embodiments, memory 402 may be an internal storage unit of electronic device 400, such as a hard disk or memory of electronic device 400. In other embodiments, memory 402 may also be an external storage device of electronic device 400, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. equipped on electronic device 400.

[0102] Furthermore, the memory 402 may include both internal storage units of the electronic device 400 and external storage devices. The memory 402 is used to store application software and various types of data installed on the electronic device 400.

[0103] In some embodiments, processor 401 may be a central processing unit (CPU), microprocessor, or other data processing chip, used to run program code stored in memory 402 or process data, such as the dynamic scene modeling method of the present invention.

[0104] In some embodiments, display 403 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. Display 403 is used to display information from electronic device 400 and to display a visual user interface. Components 401-403 of electronic device 400 communicate with each other via a system bus.

[0105] In some embodiments of the present invention, when the processor 401 executes the dynamic scene modeling program in the memory 402, the following steps can be implemented: After aligning the 3D point cloud of the scene acquired by the LiDAR with the image sequence acquired by the multi-view camera, pedestrian target detection is performed on the image sequence to generate bounding boxes; Based on the bounding box, the dynamic pedestrian point cloud and the static background point cloud in the scene's 3D point cloud are separated. A hash network is used to encode the static background point cloud into Gaussian point parameters, and the spatial distribution of Gaussian point clusters is initialized based on the dynamic pedestrian point cloud to construct a Gaussian scene representation model. Kalman filtering is used to dynamically model and simulate physical occlusion effects on the pedestrian trajectory in the Gaussian scene representation model in order to update the Gaussian scene representation model. Heterogeneous features are fused between the static and dynamic regions of the updated Gaussian scene representation model to generate fused features; Based on the fused features and multi-view camera parameters, dynamic scene images are generated.

[0106] It should be understood that when the processor 401 executes the dynamic scene modeling program in the memory 402, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0107] Furthermore, the embodiments of the present invention do not specifically limit the type of electronic device 400 mentioned. Electronic device 400 can be a mobile phone, tablet computer, personal digital assistant (PDA), wearable device, laptop computer, or other portable electronic device. Exemplary embodiments of portable electronic devices include, but are not limited to, portable electronic devices running iOS, Android, Microsoft, or other operating systems. The aforementioned portable electronic device can also be other portable electronic devices, such as a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present invention, electronic device 400 may not be a portable electronic device, but rather a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0108] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the dynamic scene modeling methods provided by the methods described above, the method comprising: After aligning the 3D point cloud of the scene acquired by the LiDAR with the image sequence acquired by the multi-view camera, pedestrian target detection is performed on the image sequence to generate bounding boxes; Based on the bounding box, the dynamic pedestrian point cloud and the static background point cloud in the scene's 3D point cloud are separated. A hash network is used to encode the static background point cloud into Gaussian point parameters, and the spatial distribution of Gaussian point clusters is initialized based on the dynamic pedestrian point cloud to construct a Gaussian scene representation model. Kalman filtering is used to dynamically model and simulate physical occlusion effects on the pedestrian trajectory in the Gaussian scene representation model in order to update the Gaussian scene representation model. Heterogeneous features are fused between the static and dynamic regions of the updated Gaussian scene representation model to generate fused features; Based on the fused features and multi-view camera parameters, dynamic scene images are generated.

[0109] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0110] The dynamic scene modeling method, apparatus, and electronic device provided by the present invention have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A dynamic scene modeling method, characterized in that, include: After aligning the 3D point cloud of the scene acquired by the LiDAR with the image sequence acquired by the multi-view camera, pedestrian target detection is performed on the image sequence to generate bounding boxes; Based on the bounding box, the dynamic pedestrian point cloud and the static background point cloud in the scene's 3D point cloud are separated. A hash network is used to encode the static background point cloud into Gaussian point parameters, and the spatial distribution of Gaussian point clusters is initialized based on the dynamic pedestrian point cloud to construct a Gaussian scene representation model. Kalman filtering is used to dynamically model and simulate physical occlusion effects on the pedestrian trajectory in the Gaussian scene representation model in order to update the Gaussian scene representation model. Heterogeneous features are fused between the static and dynamic regions of the updated Gaussian scene representation model to generate fused features; Based on the fused features and multi-view camera parameters, dynamic scene images are generated.

2. The dynamic scene modeling method according to claim 1, characterized in that, Aligning the 3D point cloud of the scene acquired by LiDAR with the image sequence acquired by multi-view cameras, including: Determine the calibration matrix between the lidar and the multi-view camera; The calibration matrix parameters are optimized based on the iterative nearest point algorithm, and the scene 3D point cloud and the image sequence are aligned by homogeneous transformation based on the optimized calibration matrix.

3. The dynamic scene modeling method according to claim 1, characterized in that, Perform pedestrian target detection on the image sequence and generate bounding boxes, including: Pedestrian target detection is performed on the image sequence based on the YOLO network to generate original bounding boxes; The Mask R-CNN network performs pixel-level instance segmentation within the original bounding box, generating bounding boxes carrying binary masks.

4. The dynamic scene modeling method according to claim 1, characterized in that, The pedestrian trajectory in the Gaussian scene representation model is dynamically modeled and the physical occlusion effect is simulated using Kalman filtering to update the Gaussian scene representation model, including: The parameters of Gaussian points corresponding to the static background point cluster in the Gaussian scene representation model are frozen. Based on the spatial distribution of Gaussian point clusters and the pedestrian motion law model, the pedestrian motion speed is predicted by combining the Kalman filter method, and the position of the Gaussian points is updated based on the predicted pedestrian motion speed. The covariance matrix of Gaussian points is dynamically expanded based on the predicted pedestrian movement speed to simulate motion blur effect and dynamically adjust the occlusion perception transparency.

5. The dynamic scene modeling method according to claim 4, characterized in that, Dynamically adjust the transparency of occlusion perception, including: When the depth of the target Gaussian point is less than the preset occlusion threshold, the occlusion perception transparency is reduced proportionally to simulate visual occlusion.

6. The dynamic scene modeling method according to claim 4, characterized in that, Also includes: When updating the Gaussian scene representation model, a sparse gradient update mechanism is used to reset the gradient of the Gaussian point parameters corresponding to the static background point cloud to zero.

7. The dynamic scene modeling method according to claim 1, characterized in that, Heterogeneous feature fusion is performed on the static and dynamic regions of the updated Gaussian scene representation model to generate fused features, including: In the static region of the updated Gaussian scene representation model, Gaussian point geometric features and color features generated by Gaussian rendering interpolation are fused together, while in the dynamic region, pedestrian motion trajectory features and texture features are fused together to generate fused features.

8. The dynamic scene modeling method according to any one of claims 1-7, characterized in that, Based on the fused features and multi-view camera parameters, a dynamic scene image is generated, including: Based on the fusion features and multi-view camera parameters, a material base color is generated; Dynamic scene images are generated based on the material's base color and the specular component used to simulate the reflection from a camera lens.

9. A dynamic scene modeling device, characterized in that, include: The pedestrian detection module is used to perform pedestrian target detection on the image sequence after aligning the scene 3D point cloud collected by LiDAR with the image sequence collected by multi-view cameras, and generate bounding boxes. The dynamic-static separation module is used to separate the dynamic pedestrian point cloud and the static background point cloud in the scene 3D point cloud based on the bounding box; The model building module is used to encode the static background point cloud into Gaussian point parameters using a hash network, and initialize the spatial distribution of Gaussian point clusters based on the dynamic pedestrian point cloud to construct a Gaussian scene representation model. The model update module is used to dynamically model the pedestrian motion trajectory in the Gaussian scene representation model using Kalman filtering and simulate the physical occlusion effect, so as to update the Gaussian scene representation model. The feature fusion module is used to perform heterogeneous feature fusion on the static and dynamic regions of the updated Gaussian scene representation model to generate fused features; The scene generation module is used to generate dynamic scene images based on the fused features and multi-view camera parameters.

10. An electronic device, characterized in that, Including memory and processor, among which, The memory is used to store programs; The processor, coupled to the memory, is used to execute the program stored in the memory to implement the steps of the dynamic scene modeling method as described in any one of claims 1 to 8.

Images