A pose optimization and external parameter compensation method and device for 4D Gaussian sputtering scene reconstruction

Abstract

The application provides a pose optimization and external parameter compensation method and device for 4D Gaussian sputtering scene reconstruction, relates to the technical field of three-dimensional reconstruction, and comprises the following steps: acquiring current frame image data collected by a multi-view camera, initial trajectory parameters of a host vehicle, initial external parameters of each camera and an initial positioning pose of the host vehicle, and establishing a 4D Gaussian sputtering scene state to be optimized; determining current camera external parameters according to the initial external parameters and a six-degree-of-freedom external parameter increment, and determining a current host vehicle positioning pose according to the initial positioning pose and a host vehicle positioning increment; performing continuous six-dimensional rotational manifold parameterization on a trajectory rotation component to obtain to-be-optimized manifold trajectory parameters; constructing a global total loss based on a photometric error, a re-projection difference, a host vehicle positioning loss and a trajectory smoothing constraint; solving the gradient of the loss and synchronously updating each parameter and the scene state, and outputting a reconstruction result, so that the reconstruction accuracy and stability of a 4D Gaussian sputtering space-time scene model can be improved.

Description

Technical Field

[0001] This application relates to the field of 3D reconstruction technology, and in particular to a pose optimization and extrinsic parameter compensation method and apparatus for 4D Gaussian sputtering scene reconstruction. Background Technology

[0002] 4D Gaussian sputtering scene reconstruction typically uses continuous images acquired by vehicle-mounted multi-view cameras, vehicle positioning and pose, and camera extrinsic parameters as input. By projecting the Gaussian representation of the dynamic scene onto different camera viewpoints and comparing it with real images, a spatiotemporal scene model that can be used for autonomous driving perception, path planning, and environmental understanding is gradually formed. This type of reconstruction process is highly dependent on the continuity and consistency of the pose and extrinsic parameter links: rotation and translation errors in the main vehicle trajectory directly change the projected position of the Gaussian points in the camera viewpoints, and the offset of the extrinsic parameters from multiple cameras can disrupt the correspondence of the same scene structure within overlapping fields of view.

[0003] Under conditions of long-distance vehicle travel, bumpy roads, and continuous parallel acquisition by multiple cameras, the trajectory parameters and camera extrinsic parameters obtained in the relevant technologies are prone to deviation over time. Trajectory errors, camera extrinsic parameter errors, and vehicle positioning errors are often processed separately and are decoupled from each other, making it difficult to promptly feed the correction results of one link back to the 4D Gaussian sputtering scene state and other parameter links. When these deviations collectively participate in the rendering and reconstruction of the current frame, inconsistencies in the image correspondence within the overlapping viewport will occur, and the scene structure between consecutive frames will also experience cumulative misalignment.

[0004] Therefore, when performing 4D Gaussian sputtering reconstruction for dynamic urban scenes, related technologies are prone to problems such as discontinuous trajectory rotation optimization, drift in the reprojection relationship between cameras, and mismatch between the vehicle's positioning pose and the scene state. Ultimately, this leads to a continuous increase in the luminosity difference between the rendered image and the current frame image, resulting in ghosting, stretching, or local misalignment in the reconstruction results, making it difficult for the autonomous driving system to obtain a stable and consistent dynamic scene representation. Summary of the Invention

[0005] To overcome the problems existing in related technologies, this application provides a pose optimization and extrinsic parameter compensation method and apparatus for 4D Gaussian sputtering scene reconstruction, which can ensure the spatial reprojection consistency of multi-view panoramic scenes, eliminate ghosting and local misalignment in the reconstruction results, and improve the reconstruction accuracy and stability of 4D Gaussian sputtering spatiotemporal scene models.

[0006] According to a first aspect of the embodiments of this application, a method for pose optimization and extrinsic parameter compensation for 4D Gaussian sputtering scene reconstruction is provided, the method comprising: Acquire the current frame image data collected by the multi-camera during vehicle movement, the initial trajectory parameters of the main vehicle corresponding to the current frame, the initial extrinsic parameters of each camera, and the initial positioning pose of the main vehicle, and establish the 4D Gaussian sputtering scene state to be optimized. The current camera extrinsic parameters are determined based on the initial extrinsic parameters of each camera and the corresponding six-degree-of-freedom extrinsic parameter increments. The current master vehicle positioning pose is determined based on the initial positioning pose of the master vehicle and the corresponding master vehicle positioning increments. The master vehicle positioning increments and the six-degree-of-freedom extrinsic parameter increments adopt the same manifold perturbation update parameterization form. The two are configured independently and updated synchronously under the same global total loss. The rotational components in the initial trajectory parameters are subjected to continuous six-dimensional rotational manifold parameterization to obtain the trajectory parameters of the manifold to be optimized. Based on the current camera extrinsic parameters, the current vehicle positioning pose, the manifold trajectory parameters to be optimized, and the translation component in the initial trajectory parameters, the 4D Gaussian sputtering scene state is projected and rendered to the camera view of the current frame to obtain a rendered image. Based on the photometric error between the rendered image and the corresponding current frame image data, the reprojection difference in the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint, the global total loss of the current frame is constructed. The gradient of the global total loss is calculated, and the trajectory parameters of the manifold to be optimized and the translation components are updated and corrected according to the gradient to obtain the trajectory correction amount. At the same time, the six-degree-of-freedom extrinsic parameter increment, the main vehicle positioning increment and the 4D Gaussian sputtering scene state are updated synchronously according to the gradient, and the 4D Gaussian sputtering scene reconstruction result of the current frame is output.

[0007] In one exemplary embodiment of this application, the step of constructing the global total loss of the current frame based on the photometric error between the rendered image and the corresponding current frame image data, the reprojection difference within the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint includes: The photometric error is determined based on the pixel residual between the rendered image and the corresponding current frame image data within the effective observation area; The reprojection difference within the overlapping field of view of adjacent cameras is determined based on the projection position deviation of the same 4D Gaussian sputtering scene state under different camera perspectives. The main vehicle positioning loss is calculated based on the current main vehicle positioning pose and the main vehicle positioning increment. The trajectory smoothing constraint is determined based on the manifold trajectory parameters to be optimized, the translation component, and the pose change between the main vehicle trajectory parameters and those of historical adjacent frames; The photometric error, the reprojection difference within the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint are normalized and weighted to obtain the global total loss of the current frame.

[0008] In one exemplary embodiment of this application, determining the reprojection difference within the overlapping field of view of adjacent cameras based on the projection position deviation of the same 4D Gaussian sputtering scene state under different camera viewpoints includes: The 4D Gaussian sputtering scene state is projected onto the viewpoint of a first camera and the viewpoint of a second camera that overlaps with the viewpoint of the first camera, respectively, to obtain a first projection result and a second projection result; based on the current camera extrinsic parameters, the first projection result is transformed into the coordinate system of the second camera viewpoint to obtain a reference projection result; Based on the positional residual between the reference projection result and the second projection result, the reprojection difference within the overlapping field of view of the adjacent cameras is determined; Wherein, the first camera viewpoint is used to characterize the observation viewpoint of any camera, and the second camera viewpoint is used to characterize the observation viewpoint of an adjacent camera that has an overlapping field of view with the first camera.

[0009] In one exemplary embodiment of this application, the step of determining the current camera extrinsic parameters based on the initial extrinsic parameters of each camera and the corresponding six-degree-of-freedom extrinsic parameter increments, and determining the current master vehicle positioning pose based on the initial positioning pose of the master vehicle and the corresponding master vehicle positioning increments, includes: The initial extrinsic parameters of each camera, the increment of each six-degree-of-freedom extrinsic parameter, and the preset perturbation superposition operator are used to perform manifold exponential mapping calculation to obtain the current camera extrinsic parameters of each camera. The current vehicle positioning pose is obtained by performing manifold index mapping calculation on the main vehicle positioning increment, the initial positioning pose, and the disturbance superposition operator.

[0010] In one exemplary embodiment of this application, the step of performing continuous six-dimensional rotational manifold parameterization on the rotational components in the initial trajectory parameters to obtain the manifold trajectory parameters to be optimized includes: Extract the three-dimensional rotation component from the initial trajectory parameters; The three-dimensional rotational components are projected into the continuous rotating manifold representation space to obtain the trajectory parameters of the manifold to be optimized, which are composed of six-dimensional manifold vectors.

[0011] In one exemplary embodiment of this application, the step of solving the gradient of the global total loss, updating and correcting the trajectory parameters of the manifold to be optimized and the translation component according to the gradient to obtain the trajectory correction amount, and simultaneously updating the six-degree-of-freedom extrinsic parameter increment, the main vehicle positioning increment, and the 4D Gaussian sputtering scene state according to the gradient, and outputting the 4D Gaussian sputtering scene reconstruction result of the current frame, includes: The global gradient vector is obtained by calculating the first-order partial derivative of the global total loss. In the current optimization iteration, the global gradient vector is decomposed into a first gradient, a second gradient, a third gradient, and a fourth gradient that are independent of each other. The first gradient acts on the manifold trajectory parameters to be optimized and the translation component, the second gradient acts on the six-degree-of-freedom extrinsic parameter increment, the third gradient acts on the vehicle positioning increment, and the fourth gradient acts on the 4D Gaussian sputtering scene state. Based on the first gradient, the trajectory correction amount is obtained, and the second gradient and the third gradient are used to update the six-degree-of-freedom extrinsic parameter increment and the main vehicle positioning increment simultaneously. At the same time, the fourth gradient is used to update the 4D Gaussian sputtering scene state simultaneously, so as to obtain the multi-parameter synchronous joint optimization result.

[0012] In one exemplary embodiment of this application, the simultaneous updating of the 4D Gaussian sputtering scene state using the fourth gradient synchronization to obtain a multi-parameter synchronous joint optimization result includes: Based on the fourth gradient, calculate the position, opacity, and scaling parameter update amount of the scene Gaussian point within the visible area of ​​the current frame; The numerical update is superimposed and updated into the state tensor of the 4D Gaussian sputtering scene state to obtain the updated 4D Gaussian sputtering scene state. Based on the updated 4D Gaussian sputtering scene state, image rendering is performed in the visible view of the current frame to obtain the 4D Gaussian sputtering scene reconstruction result of the current frame.

[0013] According to a second aspect of the embodiments of this application, a pose optimization and extrinsic parameter compensation device for 4D Gaussian sputtering scene reconstruction is provided, the device comprising: The acquisition module is used to acquire the current frame image data collected by the multi-view camera during the vehicle's movement, the initial trajectory parameters of the main vehicle corresponding to the current frame, the initial external parameters of each camera, and the initial positioning pose of the main vehicle, and to establish the 4D Gaussian sputtering scene state to be optimized. The determination module is used to determine the current camera extrinsic parameters based on the initial extrinsic parameters of each camera and the corresponding six-degree-of-freedom extrinsic parameter increments, and to determine the current master vehicle positioning pose based on the initial positioning pose of the master vehicle and the corresponding master vehicle positioning increments. The master vehicle positioning increments and the six-degree-of-freedom extrinsic parameter increments adopt the same manifold perturbation update parameterization form, and the two are configured independently and updated synchronously under the same global total loss. The processing module is used to perform continuous six-dimensional rotational manifold parameterization on the rotational components in the initial trajectory parameters to obtain the trajectory parameters of the manifold to be optimized. The construction module is used to project and render the 4D Gaussian sputtering scene state to the camera view of the current frame to obtain a rendered image based on the current camera extrinsic parameters, the current vehicle positioning pose, the manifold trajectory parameters to be optimized, and the translation component in the initial trajectory parameters. It also constructs the global total loss of the current frame based on the photometric error between the rendered image and the corresponding current frame image data, the reprojection difference in the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint. The output module is used to calculate the gradient of the global total loss, update and correct the trajectory parameters of the manifold to be optimized and the translation component according to the gradient to obtain the trajectory correction amount, and simultaneously update the six-degree-of-freedom extrinsic parameter increment, the main vehicle positioning increment and the 4D Gaussian sputtering scene state according to the gradient, and output the 4D Gaussian sputtering scene reconstruction result of the current frame.

[0014] According to a third aspect of the embodiments of this application, an electronic device is provided, the electronic device comprising: Processor; and A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in the first aspect.

[0015] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein computer program code instructions are stored on the computer-readable storage medium, and when the computer program code instructions are executed by a processor, the method described in the first aspect is implemented.

[0016] Compared to related technologies that process trajectory error, camera extrinsic parameter error, and vehicle positioning error separately, this application incorporates the current frame image data, the initial trajectory parameters of the vehicle, the initial extrinsic parameters of each camera, the initial positioning pose of the vehicle, and the 4D Gaussian sputtering scene state to be optimized into the same optimization link during the current frame reconstruction process. This allows the error between the rendered image and the current frame image data to be fed back to the trajectory, extrinsic parameters, positioning, and scene state, avoiding coordinate mismatch caused by other links remaining offset after single parameter link correction. This reduces the drift of correspondence within the overlapping field of view and the cumulative misalignment between consecutive frames.

[0017] Furthermore, when determining the current camera extrinsic parameters and the current vehicle positioning pose, this application uses six-degree-of-freedom extrinsic parameter increments and vehicle positioning increments to consistently express camera extrinsic parameter drift and vehicle positioning offset. During trajectory optimization, this application performs continuous six-dimensional rotating manifold parameterization on the rotation component and constructs the global total loss for the current frame by combining photometric error, reprojection differences within the overlapping fields of view of adjacent cameras, and trajectory smoothing constraints. As a result, trajectory rotation error, trajectory translation error, camera extrinsic parameter drift, vehicle positioning offset, and 4D Gaussian sputtering scene state misalignment can be co-optimized and jointly converged in the same gradient solution, making the 4D Gaussian sputtering scene reconstruction result of the current frame more stable and consistent with multi-view camera observations, thereby reducing the probability of ghosting, stretching, and local misalignment.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this application, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] Figure 1 This is a flowchart illustrating a pose optimization and extrinsic parameter compensation method for 4D Gaussian sputtering scene reconstruction according to an exemplary embodiment of this application.

[0021] Figure 2 This is a schematic diagram of the pose optimization and extrinsic parameter compensation device for 4D Gaussian sputtering scene reconstruction according to an exemplary embodiment of this application.

[0022] Figure 3 This is a hardware structure diagram of a computer device illustrated in an exemplary embodiment of this application.

[0023] Figure 4 This is a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of this application. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0025] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0026] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0027] The embodiments of this application will now be described in detail.

[0028] This application relates to a pose optimization and extrinsic parameter compensation method for 4D Gaussian Splatting (GS) scene reconstruction. 4D Gaussian Splatting scene reconstruction can be understood as a reconstruction process involving Gaussian representation, projection rendering, and parameter updates of a time-varying 3D scene. This process can receive continuous frame images acquired by multiple cameras, vehicle trajectory parameters, extrinsic parameters of each camera, and the vehicle's positioning pose, and generate a rendered image from the current frame's camera viewpoint. The difference between the rendered image and the actual acquired image reflects the degree of inconsistency between the current scene state, trajectory parameters, extrinsic parameters, and positioning pose.

[0029] In some embodiments, this application can be deployed in an in-vehicle computing platform. The in-vehicle computing platform may include an image acquisition interface, an inertial measurement data interface, a combined navigation data interface, a processor, a graphics processing unit, a memory, and a data bus. The image acquisition interface is used to receive multi-view camera images from the vehicle's forward, backward, left, right, or surround-view positions. The inertial measurement data interface is used to receive acceleration, angular velocity, or attitude-related data. The combined navigation data interface is used to receive the initial positioning pose of the main vehicle. The processor is used to perform timing synchronization, coordinate transformation, loss construction, and parameter scheduling. The graphics processing unit can be used to perform parallel operations in 4D Gaussian sputtering rendering, visible area filtering, and gradient calculation. The memory can store the current frame image data, historical adjacent frame trajectory parameters, initial camera extrinsic parameters, six-DOF extrinsic parameter increments, main vehicle positioning increments, and 4D Gaussian sputtering scene states.

[0030] In other embodiments, this application can also be deployed in a vehicle-to-cloud collaborative system. The vehicle can be responsible for multi-camera image acquisition, current frame preprocessing, initial pose loading, and low-latency reconstruction output. The cloud or roadside computing device can receive keyframe data, low-frequency extrinsic parameter correction results, or historical trajectory segments uploaded by the vehicle and verify the parameter stability across time periods. When optimizing the current frame in the next iteration, the vehicle can use the correction results returned by the cloud or roadside computing device as an initialization reference for extrinsic parameter increments or trajectory parameters. This deployment method does not change the joint optimization logic within the current frame; it only changes some data sources and the location of computing resources.

[0031] The application scenarios of this application can include dynamic urban scene reconstruction, highway driving scene reconstruction, park autonomous driving scene reconstruction, underground parking lot or tunnel scene reconstruction, closed test scene reconstruction, and multi-view vision reconstruction of robot mobile platforms. In dynamic urban scenes, vehicles, pedestrians, non-motorized vehicles, and static buildings appear together in the field of view of multiple cameras, and the difference between the current frame image and the 4D Gaussian sputtering rendered image is easily affected by the combined influence of the main vehicle pose and extrinsic parameter drift. In highway scenes, the long-distance travel of vehicles leads to more significant cumulative errors in the main vehicle trajectory, and the combined use of trajectory smoothing constraints and photometric errors helps to suppress jumps in projection positions between consecutive frames. In underground parking lot or tunnel scenes, the availability of satellite positioning decreases, and the uncertainty of the initial positioning pose of the main vehicle increases. Homogeneous incremental updates can reduce mismatches between the positioning pose and the scene state. In robot mobile platform scenarios, multi-view cameras can be mounted on the mobile robot body, the vehicle can be replaced by the robot body, and the main vehicle positioning pose can be replaced by the mobile platform positioning pose.

[0032] The input objects in this embodiment may include the current frame image data of the multi-camera system, the initial trajectory parameters of the main vehicle, the initial extrinsic parameters of the cameras, the initial positioning pose of the main vehicle, and the 4D Gaussian sputtering scene state to be optimized. The current frame image data of the multi-camera system may include camera identifiers, frame timestamps, image matrices, exposure information, effective observation area masks, and camera intrinsic parameters. The initial trajectory parameters of the main vehicle may include rotational components, translational components, trajectory timestamps, and historical adjacent frame indices. The initial extrinsic parameters of the cameras may include the initial pose of each camera relative to the vehicle coordinate system or the world coordinate system. The initial positioning pose of the main vehicle may include the initial position and attitude of the main vehicle in the world coordinate system. The 4D Gaussian sputtering scene state may include the position, opacity, scaling parameters, color-related parameters, time-related parameters, and visibility index of the scene's Gaussian points.

[0033] In practical engineering implementation, the current frame optimization task can be organized using a structured task data stream. This data stream includes fields such as frame identifier, vehicle identifier, camera set, image data index, initial trajectory parameters, camera initial extrinsic parameters, six-DOF extrinsic parameter increment, master vehicle initial localization pose, master vehicle localization increment, 4D Gaussian sputtering scene state, historical adjacent frame index, and optimization state. The frame identifier field distinguishes the current frame from historical frames. The camera set field records the number and identifiers of cameras participating in the current frame optimization. The image data index field points to the image matrix or image feature cache in memory. The six-DOF extrinsic parameter increment field stores the extrinsic parameter offset for each camera. The master vehicle localization increment field stores the correction amount for the master vehicle's initial localization pose in the current frame. The optimization state field records whether the current frame is in the initialization, loss construction, gradient update, scene state update, or output stage.

[0034] For a multi-camera system, the cameras may include a forward-looking camera, a rear-looking camera, a left-looking camera, a right-looking camera, and an oblique blind-spot camera. Each camera has its own initial extrinsic parameters. During vehicle operation, vibrations, minor structural displacements, or mechanical shifts after long-term use may cause differences between the actual extrinsic parameters and the initial extrinsic parameters of each camera. Therefore, embodiments of this application configure a six-DOF extrinsic parameter increment for each camera. This six-DOF extrinsic parameter increment can be used as a variable to be updated in the joint optimization of the current frame. The current camera extrinsic parameters of each camera are jointly determined by the corresponding initial extrinsic parameters and the corresponding six-DOF extrinsic parameter increment. In this way, extrinsic parameter drift is not considered a fixed error but is incorporated into the optimizable parameter chain of the current frame reconstruction.

[0035] For the main vehicle localization link, this embodiment also designs the main vehicle localization increment as a six-dimensional tangent space representation that is homologous to the camera's six-DOF extrinsic parameter increment. By using the same manifold perturbation to update the parameterization form for both the main vehicle localization increment and the six-DOF extrinsic parameter increment, and configuring them independently, this setting allows extrinsic parameter drift and main vehicle localization offset to participate in the loss solution within the same current frame based on the same manifold perturbation superposition logic. When the camera extrinsic parameters experience a slight shift, the main vehicle localization pose will not be fixed in the initial state, but can be synchronously corrected with the gradient feedback of the global total loss. Thus, the coordinate link between the current camera extrinsic parameters, the current main vehicle localization pose, and the 4D Gaussian sputtering scene state can maintain good consistency.

[0036] For the trajectory link, this embodiment of the application performs continuous six-dimensional rotational manifold parameterization on the rotation component in the initial trajectory parameters to obtain the trajectory parameters of the manifold to be optimized. The translation component can be decoupled from the manifoldized rotation component to participate in the optimization. This process makes the rotation update independent of the local representation of Euler angles, reducing discontinuous changes in the attitude update process.

[0037] In this embodiment, the overall processing flow can be executed cyclically according to the current frame task. The processor first acquires the current frame image data, the initial trajectory parameters of the main vehicle, the initial extrinsic parameters of each camera, and the initial positioning pose of the main vehicle, and establishes or reads the 4D Gaussian sputtering scene state to be optimized. Then, the processor determines the current camera extrinsic parameters and the current positioning pose of the main vehicle for each camera, and converts the rotation components in the initial trajectory parameters into manifold trajectory parameters to be optimized. Subsequently, the processor projects and renders the 4D Gaussian sputtering scene state to the camera view of the current frame based on the current camera extrinsic parameters, the current main vehicle positioning pose, the manifold trajectory parameters to be optimized, and the translation components, obtaining a rendered image. The processor then constructs the global total loss for the current frame based on the photometric error between the rendered image and the current frame image data, the reprojection difference within the overlapping view of adjacent cameras, the main vehicle positioning loss, and the trajectory smoothing constraint. Finally, the processor calculates the gradient of the global total loss and synchronously updates the trajectory parameters, extrinsic parameter increments, main vehicle positioning increments, and the 4D Gaussian sputtering scene state, outputting the 4D Gaussian sputtering scene reconstruction result for the current frame.

[0038] See Figure 1 , Figure 1 This can be a flowchart illustrating a pose optimization and extrinsic parameter compensation method for 4D Gaussian sputtering scene reconstruction, as shown in an exemplary embodiment of this application. The method can be executed by an onboard computing platform, a roadside computing device, a cloud server, or a combination thereof. In an onboard real-time reconstruction scenario, the executing entity can be a processor and graphics processing unit on the vehicle. The method may include steps 110 to 150 as follows.

[0039] Step 110: Obtain the current frame image data collected by the multi-camera during vehicle movement, the initial trajectory parameters of the main vehicle corresponding to the current frame, the initial extrinsic parameters of each camera, and the initial positioning pose of the main vehicle, and establish the 4D Gaussian sputtering scene state to be optimized.

[0040] Step 120: Determine the current camera extrinsic parameters based on the initial extrinsic parameters of each camera and the corresponding six-degree-of-freedom extrinsic parameter increments, and determine the current master vehicle positioning pose based on the initial positioning pose of the master vehicle and the corresponding master vehicle positioning increments.

[0041] Step 130: Perform continuous six-dimensional rotating manifold parameterization on the rotation component in the initial trajectory parameters to obtain the trajectory parameters of the manifold to be optimized.

[0042] Step 140: Based on the current camera extrinsic parameters, the current vehicle positioning pose, the manifold trajectory parameters to be optimized, and the translation components in the initial trajectory parameters, the 4D Gaussian sputtering scene state is projected and rendered to the camera view of the current frame to obtain a rendered image. Based on the photometric error between the rendered image and the corresponding current frame image data, the reprojection difference in the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint, the global total loss of the current frame is constructed.

[0043] Step 150: Solve for the gradient of the global total loss, and update and correct the trajectory parameters and translation components of the manifold to be optimized according to the gradient to obtain the trajectory correction amount. At the same time, update the six-degree-of-freedom extrinsic parameter increment, the main vehicle positioning increment and the 4D Gaussian sputtering scene state synchronously according to the gradient, and output the 4D Gaussian sputtering scene reconstruction result of the current frame.

[0044] The pose optimization and extrinsic parameter compensation method for 4D Gaussian sputtering scene reconstruction provided in this application establishes the state of the 4D Gaussian sputtering scene to be optimized and records it in a unified manner with the multi-source input parameters, providing a unified spatiotemporal data benchmark for subsequently incorporating various parameters into the same optimization link.

[0045] Furthermore, addressing the extrinsic parameter and positioning deviations generated by the vehicle during long-distance travel and road bumps, the current camera extrinsic parameters and the current master vehicle positioning pose are determined in real time by utilizing the six-DOF extrinsic parameter increments of each camera and the master vehicle positioning increment. This achieves rigid geometric constraints and consistent expression of extrinsic parameter drift and positioning offset under the same incremental expression logic. Simultaneously, by performing continuous six-dimensional rotational manifold parameterization on the rotational components in the master vehicle's initial trajectory parameters, the discontinuity in pose updates during iteration, as seen in traditional local parameterization expressions, is effectively overcome, ensuring the temporal smoothness of trajectory rotation optimization.

[0046] Based on this, the complete pose link parameters are integrated with the 4D Gaussian sputtering scene state for unified projection rendering. The global total loss for the current frame is constructed by combining photometric error, reprojection differences within the overlapping fields of view of adjacent cameras, vehicle positioning loss, and trajectory smoothing constraints. This step imposes loss constraints on the spatial geometric consistency and temporal trajectory continuity of multi-view surround view, ensuring that the errors between the rendered image and the current frame image data can collectively contribute to subsequent feedback adjustments.

[0047] Finally, by calculating the gradient of the global total loss, and using this gradient to synchronously update the correction trajectory parameters, six-DOF extrinsic parameter increments, vehicle positioning increments, and 4D Gaussian sputtering scene state in the same optimization iteration, this application overcomes the limitations of related technologies where various errors are handled and decoupled for optimization. It enables the coordinated optimization and joint convergence of vehicle trajectory rotation error, trajectory translation error, camera extrinsic parameter drift, vehicle positioning offset, and 4D Gaussian sputtering scene state misalignment in the same gradient calculation. This application eliminates asynchronous drift and coordinate mismatch between parameter links, ensuring more stable consistency between the reconstruction results and multi-view camera observations. This effectively reduces the probability of ghosting, stretching, and local misalignment in the reconstruction results, improving the reconstruction accuracy and stability of the spatiotemporal scene model.

[0048] The following provides a detailed explanation of steps 110 to 150.

[0049] In step 110, the current frame image data collected by the multi-camera during the vehicle's movement, the initial trajectory parameters of the main vehicle corresponding to the current frame, the initial extrinsic parameters of each camera, and the initial positioning pose of the main vehicle are obtained, and the 4D Gaussian sputtering scene state to be optimized is established.

[0050] In one example embodiment of this application, the current frame image data may include an image matrix acquired by multiple cameras under the same vehicle time reference, frame timestamps, camera identifiers, exposure information, effective observation area masks, and camera intrinsic parameters. The initial trajectory parameters of the main vehicle may include rotational components, translational components, and trajectory timestamps. The initial extrinsic parameters of each camera may include the initial pose of each camera relative to the vehicle coordinate system or the world coordinate system. The initial positioning pose of the main vehicle may come from integrated navigation, a visual odometry front-end, or map matching results. The 4D Gaussian sputtering scene state may include the position, opacity, scaling parameters, color correlation parameters, time index, and visibility index of the scene's Gaussian points. By organizing the above data in the same current frame task record, subsequent photometric errors, reprojection differences, and trajectory smoothing constraints can be fed back into the same parameter chain.

[0051] In step 120, the current camera extrinsic parameters are determined based on the initial extrinsic parameters of each camera and the corresponding six-degree-of-freedom extrinsic parameter increments, and the current master vehicle positioning pose is determined based on the initial positioning pose of the master vehicle and the corresponding master vehicle positioning increments.

[0052] In one example embodiment of this application, the current camera extrinsic parameters and the main vehicle positioning increment are determined based on the initial extrinsic parameters of each camera and the corresponding six-degree-of-freedom extrinsic parameter increments. Specifically, the processing method includes the following: First, obtain the six-DOF extrinsic parameter increments for each camera. These increments can then be assembled into an extrinsic parameter tensor with a preset dimension. This tensor is structured as a multi-dimensional tensor, cascaded and concatenated, including a camera index dimension, a six-DOF increment dimension, and a current frame index dimension. The camera index dimension distinguishes between forward-looking, backward-looking, left-looking, right-looking, or oblique-looking cameras; the six-DOF increment dimension represents 3D translational and rotational increments; and the current frame index dimension supports online updates of the current and historical frames, enabling temporal backtracking and reuse of cross-frame data through online updates.

[0053] Subsequently, the current extrinsic parameters of each camera and the vehicle positioning increment after spatial coordinate system alignment are determined. Specifically, to achieve online self-calibration and dynamic drift compensation, the extrinsic parameter offset of each camera is embedded as an optimizable variable for any camera in the multi-camera system. (Its camera index is determined by the dimension or camera identifier corresponding to the extrinsic tensor), its current camera extrinsic parameters in the world coordinate system. Based on initial external parameters With the corresponding six-degree-of-freedom external parameter increment Combining the preset perturbation superposition operator in the special Euclidean group SE(3) manifold space, adaptive updates are performed using the formula shown in formula (1): in, Used to characterize the superposition operator of perturbations in the manifold space of special orthogonal groups or special Euclidean groups. Each monocular camera is configured with independent six-DOF incremental optimization parameters to compensate for extrinsic parameter micro-drifts caused by vehicle vibration or thermal deformation in real time. Current camera extrinsic parameters... It was then used as a known geometric prior for subsequent projection rendering and reprojection difference calculation.

[0054] In one example embodiment of this application, the current positioning pose of the main vehicle is determined based on the initial positioning pose and the positioning increment of the main vehicle, specifically including the following processing method: First, extract the initial positioning pose of the main vehicle from the current frame task record. And obtain the incremental location of the main vehicle. The initial positioning pose is represented as a transformation matrix under the special Euclidean group SE(3), and the vehicle positioning increment is represented as a six-dimensional tangent space vector with the same geometric dimension as the camera extrinsic parameter increment (i.e., ).

[0055] Subsequently, using the same source manifold perturbation superposition logic as the camera extrinsic parameter incremental update, the same perturbation superposition operator as in formula (1) is invoked, utilizing the main vehicle positioning increment. For the initial positioning pose Corrections are made to maintain alignment within the same tangent space geometry, thereby determining the current master vehicle positioning pose. The above incremental update process is shown in formula (2): Determine the current position and orientation of the main vehicle. Then, the calculated current translation and rotation matrix components are written into the current pose field of the main vehicle in the current frame task record, and this value is used as known input to propagate to the subsequent projection rendering and trajectory smoothing constraint construction chain. In subsequent optimization iterations, if the global total loss of the current frame does not reach the preset convergence threshold, the gradient will be continuously solved in the same optimization iteration, and the gradient will be used to dynamically backpropagate and correct the main vehicle positioning increment. The update calculations shown in formulas (1) and (2) above are repeatedly executed to refresh and update the current master vehicle positioning pose in real time. This continues until the multi-parameter synchronous joint optimization converges. This reduces the spatiotemporal mismatch drift between the vehicle's positioning and pose and the 4D Gaussian sputtering scene state.

[0056] In step 130, the rotational components in the initial trajectory parameters are subjected to continuous six-dimensional rotating manifold parameterization to obtain the trajectory parameters of the manifold to be optimized.

[0057] In one example embodiment of this application, when performing continuous six-dimensional rotational manifold parameterization on the rotational component of the initial trajectory parameters, a three-dimensional rotational component can be extracted from the initial trajectory parameters first. The three-dimensional rotational component is used to characterize the three-dimensional attitude change matrix of the vehicle during its three-dimensional motion. At the data structure organization level, this three-dimensional rotational component is stored separately from the translational component, ensuring that the attitude update operator and the translation update operator remain independently decoupled at the optimization variable level. The translational component is stored as a three-dimensional position vector or trajectory segment translation, while the three-dimensional rotational component serves as the original input to be parameterized.

[0058] Subsequently, based on special orthogonal groups The topological structure projects the three-dimensional rotational components into a continuous six-dimensional rotational manifold representation space, resulting in the manifold trajectory parameters to be optimized, which consist of continuous six-dimensional manifold vectors containing rotational information. This solves the gimbal lock and gradient singularity problems. Specifically, a continuous parameterized representation of the six-dimensional rotational manifold is adopted, and a bidirectional transformation between the continuous six-dimensional manifold vectors and the orthogonal rotation matrix is ​​achieved using continuous manifold mappings, such as exponential-logarithmic mappings. The mathematical mapping relationship of the transformation is shown in formula (3): in, For the corresponding six-dimensional manifold vector, Here is the corresponding rotation matrix. This represents the exponential manifold mapping operator that maps a six-dimensional continuous manifold vector to an orthogonal rotation matrix. This represents the logarithmic manifold inverse mapping operator that inversely maps an orthogonal rotation matrix to a continuous six-dimensional manifold vector.

[0059] Through this bidirectional mapping, the rotational attitude with non-orthogonal singularity characteristics in three-dimensional space is projected into a continuous manifold space in a high dimension, so that the trajectory parameters of the manifold to be optimized, representing the rotational components, are transformed into a vector form that is unconstrained, singularity-free, and continuously differentiable in real space.

[0060] Specifically, the manifold trajectory parameters to be optimized, composed of continuous six-dimensional manifold vectors, are batch-organized according to temporal trajectory segments. For example, for a single-frame joint optimization task in the current frame, the manifold trajectory parameters to be optimized include a set of continuous six-dimensional rotating manifold parameters corresponding to the vehicle's pose in the current frame; for a sliding window optimization task, the manifold trajectory parameters to be optimized include multiple sets of continuous six-dimensional rotating manifold parameters corresponding to the poses of the current frame and at least one historical adjacent frame. Each set of continuous six-dimensional rotating manifold parameters in the current frame task record is mapped to the translation component of the corresponding frame, the frame timestamp field in the temporal sequence, and the joint optimization mask field. The frame timestamp field is used to align the current frame image data with the trajectory parameters of historical adjacent frames on the time axis; the joint optimization mask field is used to indicate whether the set of continuous six-dimensional rotating manifold parameters is in a gradient activation state and participates in the construction of the global total loss of the current frame.

[0061] The above processing flow does not introduce a new scene reconstruction target, but rather changes the underlying representation and parameterization of the rotation component in the joint optimization chain. This decoupled representation mechanism ensures that in subsequent gradient solutions, the geometric transformation caused by rotation updates will not interfere with translation correction, effectively eliminating the jump in the Gaussian point projection position caused by discontinuous attitude updates, and improving the reconstruction accuracy and stability of the 4D Gaussian sputtering spatiotemporal scene model.

[0062] In step 140, based on the current camera extrinsic parameters, the current vehicle positioning pose, the manifold trajectory parameters to be optimized, and the translation components in the initial trajectory parameters, the 4D Gaussian sputtering scene state is projected and rendered to the camera view of the current frame to obtain a rendered image. Based on the photometric error between the rendered image and the corresponding current frame image data, the reprojection difference in the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint, the global total loss of the current frame is constructed.

[0063] In one example embodiment of this application, the 4D Gaussian sputtering scene state can be projected and rendered to the camera viewpoint of the current frame based on the current camera extrinsic parameters, the current vehicle positioning pose, the manifold trajectory parameters to be optimized, and the translation component in the initial trajectory parameters, to obtain a rendered image. Specifically, the manifold trajectory parameters to be optimized can be combined with the translation component to construct the trajectory pose of the vehicle in the current frame; then, combined with the latest current vehicle positioning pose and the current camera extrinsic parameters of each camera, the projection matrix of all camera channels, including forward-looking, backward-looking, left-looking, right-looking, and oblique blind spot filling, can be calculated. Subsequently, using this projection matrix, the 4D Gaussian scene is batch-projected and rendered onto the images of each camera, outputting the corresponding rendered image.

[0064] In one example embodiment of this application, the global total loss of the current frame is constructed based on the photometric error between the rendered image and the corresponding current frame image data, the reprojection difference within the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint. Specifically, the processing method includes the following: First, the photometric error is determined based on the pixel residuals between the rendered image and the corresponding current frame image data within the effective observation area.

[0065] For example, an effective observation area mask can be obtained by combining the image boundary, dynamic object mask, and exposure reliability mask. Then, within the range defined by the effective observation area mask, the pixel values ​​of the rendered image are numerically subtracted from the pixel values ​​of the actual acquired current frame image data to obtain the pixel residual. The residual is then calculated by combining the pixel grayscale loss operator and the structural similarity loss operator to output the photometric error of the current frame.

[0066] Secondly, based on the projection position deviation of the same 4D Gaussian sputtering scene state under different camera views within the overlapping field of view of adjacent cameras, the reprojection difference within the overlapping field of view of adjacent cameras is determined.

[0067] For example, the 4D Gaussian sputtering scene state can be projected onto the two-dimensional image plane corresponding to different camera channels, and a cross-view relative geometric transformation can be established under a unified master vehicle relative coordinate system. The degree of three-dimensional projection alignment between adjacent cameras with overlapping fields of view can be calculated, thereby solving the reprojection difference, which is used to constrain the spatial consistency of adjacent multi-camera parameter links.

[0068] Next, the vehicle positioning loss is determined based on the vehicle positioning increment and the vehicle's initial positioning pose. For example, the vehicle positioning increment can be obtained from the current frame task record, where the vehicle positioning increment is a six-dimensional tangent space vector corresponding to the vehicle's initial positioning pose. Then, the squared L2 norm of the vehicle positioning increment in the six-dimensional tangent space is calculated to obtain the positioning increment regularization loss. Alternatively, the manifold deviation between the current vehicle positioning pose after the perturbation superposition operator and the vehicle's initial positioning pose can be calculated, and the vehicle positioning loss is output. This loss is used to limit the deviation of the vehicle pose from the initial positioning information during joint optimization, ensuring the convergence stability of multi-parameter synchronous sampling.

[0069] Next, based on the manifold trajectory parameters to be optimized, the translation components, and the pose change between the main vehicle trajectory parameters and those of historical adjacent frames, trajectory smoothing constraints are determined.

[0070] For example, the historical master vehicle trajectory parameters pointed to by the historical adjacent frame index field can be obtained, and the first or second order kinematic change between the target trajectory pose solved in the current frame and the master vehicle trajectory parameters in the historical adjacent frames can be calculated. The relative change error can be calculated by logarithmic mapping in Lie algebra space, thereby outputting trajectory smoothing constraints to ensure the smooth continuity of the vehicle's temporal motion trajectory.

[0071] Finally, the photometric error, the reprojection difference in the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint are normalized and weighted to obtain the global total loss of the current frame.

[0072] For example, the configured normalized scaling factor is called to combine the photometric error, reprojection difference, main vehicle positioning loss and trajectory smoothing constraint into a unified value, and this value is written into the specified loss field of the current frame task record for subsequent steps to directly call and execute global gradient backpropagation.

[0073] In one example embodiment of this application, the reprojection difference within the overlapping field of view of adjacent cameras can be determined as follows: First, the photometric error is determined based on the pixel residuals between the rendered image and the corresponding current frame image data within the effective observation area. To establish a feedback loop in the 4DGS global photometric loss and reverse-correct trajectory parameters to suppress accumulated error, this embodiment utilizes the rendered image, the acquired image, the Gaussian point projection function, and pixel weights at the current moment to construct a trajectory-optimized global photometric loss, which characterizes the photometric error; the corresponding trajectory-optimized global photometric loss function... As shown in formula (4): in, Indicates the first The trajectory parameter set corresponding to each camera (i.e., the trajectory parameters and translation components of the manifold to be optimized). Indicates the first 4DGS rendered images from one camera. For the corresponding real captured image, The Gaussian projection function. For pixel weights, The state is a Gaussian point. Indicates the first The set of pixels within the effective observation area corresponding to each camera. This trajectory optimizes the global photometric loss function. It can reverse-align the spatiotemporal luminosity features of the current frame.

[0074] Secondly, based on the projection position deviation of the same 4D Gaussian sputtering scene state under different camera viewpoints within the overlapping field of view of adjacent cameras, the reprojection difference within the overlapping field of view of adjacent cameras is determined. Furthermore, based on the pose change between the manifold trajectory parameters to be optimized, the translation components, and the vehicle trajectory parameters in historical adjacent frames, trajectory smoothing constraints are determined.

[0075] Specifically, the system obtains the historical master vehicle trajectory parameters pointed to by the historical adjacent frame indices, calculates the first or second order kinematic changes between the target trajectory pose in the current frame and the master vehicle trajectory parameters in the historical adjacent frames, calculates the relative change error through logarithmic mapping in Lie algebra space, and thus outputs the trajectory smoothing constraint. This is used to restrict the physical continuity of the vehicle's temporal trajectory.

[0076] Finally, regarding the photometric error (i.e., the global photometric loss in trajectory optimization) ), reprojection differences within the overlapping field of view of adjacent cameras (i.e., global consistency loss of extrinsic calibration) Losses from master vehicle positioning optimization and trajectory smoothing constraints Perform weighted fusion to construct the global collaborative optimization total loss function for the current frame. The specific weighting formula (5) is shown below: in, , , , These are their respective weighting coefficients. The independent vehicle localization optimization loss is characterized by the vehicle pose optimization process, which shares the same incremental optimization mechanism as the extrinsic parameter calibration. This corresponds to the trajectory smoothing regularization term. The calculated global co-optimization total loss function can be used as a reference. The values ​​are written into the specified loss field of the current frame task record so that the calibration parameters can be dynamically updated in subsequent optimizations.

[0077] In one example embodiment of this application, the reprojection difference within the overlapping field of view of adjacent cameras is determined, specifically as shown in the following manner: First, the 4D Gaussian sputtering scene state is projected onto the first camera view and the second camera view, which has an overlapping field of view with the first camera view, respectively, to obtain the first projection result and the second projection result.

[0078] For example, firstly, the perspective projection matrix corresponding to the first camera viewpoint and the perspective projection matrix corresponding to the second camera viewpoint are extracted from the current camera extrinsic parameter field and scene state field recorded in the current frame task. The first camera viewpoint represents the observation viewpoint of any camera, and the second camera viewpoint represents the observation viewpoint of an adjacent camera with an overlapping field of view with the first camera. Then, the 3D center coordinates of the same scene Gaussian point within the overlapping region in the 4D Gaussian sputtering scene state are obtained, and these 3D center coordinates are projected onto its 2D image plane using the perspective projection matrix of the first camera viewpoint, outputting the corresponding 2D pixel coordinates, denoted as the first projection result. Similarly, the same projection process is performed using the perspective projection matrix of the second camera viewpoint, outputting the corresponding other 2D pixel coordinates, denoted as the second projection result.

[0079] Secondly, based on the current camera extrinsic parameters, the first projection result is transformed into the coordinate system of the second camera's viewpoint to obtain the reference projection result.

[0080] For example, the current camera extrinsic parameters obtained from the current frame optimization are acquired, and the rigid relative transformation matrix between two adjacent camera channels in the current iteration is calculated using the current extrinsic parameter matrix of the first camera and the current extrinsic parameter matrix of the second camera. Subsequently, using this rigid relative transformation matrix as a geometric transformation operator, the two-dimensional pixel coordinates contained in the first projection result are subjected to inverse parallax projection to restore them to local three-dimensional spatial points in the main vehicle relative coordinate system or the local vehicle center coordinate system. Then, the points are cascaded and projected onto the two-dimensional image plane of the second camera's viewpoint to generate a reference projection result.

[0081] Based on the positional residual between the reference projection result and the second projection result, the reprojection difference within the overlapping field of view of adjacent cameras is determined.

[0082] Specifically, in order to ensure the alignment of multi-view data fusion and surround view stitching, and to eliminate the physical damage to the consistency of multi-camera extrinsic parameters caused by vehicle bumps, this embodiment introduces a global geometric consistency constraint. In the three-dimensional coordinate system space, the square of the second norm of the difference between the true position of the scene Gaussian point in the overlapping field of view and its position after cross-view cascaded reprojection transformation is used as an optimization term. A second norm regularization penalty for the six-degree-of-freedom extrinsic parameter increment is also introduced to construct the global consistency loss function for extrinsic parameter calibration as shown in formula (6). : (6) in, This represents the global consistency loss of extrinsic parameter calibration (used to characterize reprojection differences). and These are the current camera extrinsic parameters (i.e., the values ​​determined by updating the initial extrinsic parameters and their increments in the manifold space perturbation) for the first and second camera perspectives, respectively, under the current optimization iteration. This represents the corresponding cross-view cascaded reprojection transformation function. Indicates the overlapping field of view of the first and second cameras. The state of the Gaussian point in the same scene within, For the corresponding regularization coefficient, For camera The corresponding six-degree-of-freedom extrinsic parameter increments. The resulting extrinsic parameter calibration global consistency loss function. It is then output to the global total loss function for subsequent weighted fusion and global gradient backpropagation.

[0083] By inversely projecting the first projection result using the current camera extrinsic parameters and transforming it across the viewing angle to the second camera's viewpoint, pixel-level geometric constraint lines are directly established on the image plane. Any relative pose deviation will be mapped into a two-dimensional position residual during the reprojection process, thereby providing a partial derivative supervision signal for the joint optimizer in subsequent steps, ensuring the spatiotemporal geometric consistency of the multi-view surround system during dynamic driving.

[0084] In step 150, the gradient of the global total loss is calculated, and the trajectory parameters and translation components of the manifold to be optimized are updated and corrected according to the gradient to obtain the trajectory correction amount. At the same time, the six-degree-of-freedom extrinsic parameter increment, the main vehicle positioning increment, and the 4D Gaussian sputtering scene state are updated synchronously according to the gradient, and the 4D Gaussian sputtering scene reconstruction result of the current frame is output.

[0085] In one example embodiment of this application, the first-order partial derivative of the global total loss can be calculated to obtain the global gradient vector. Exemplarily, the global total loss in the current iteration can be obtained. Subsequently, along the computational graph chain formed by the cascaded constraints of photometric error, reprojection difference, and trajectory smoothing, the first-order partial derivatives of all variables to be optimized are uniformly solved, thereby outputting a global gradient vector containing gradient vector information. This global gradient vector characterizes the gradient update direction between the current spatiotemporal network model and the actual observation data.

[0086] After obtaining the global gradient vector, it can be decomposed into independent first gradient, second gradient, third gradient and fourth gradient in the same optimization iteration.

[0087] For example, the global gradient vector can be decomposed and mapped to divide it into four independent gradient branches that do not interfere with each other in the variable space. The first gradient is generated by taking the partial derivative of the global total loss with respect to the trajectory variables, and acts on the trajectory parameters and translation components of the manifold to be optimized; the second gradient is generated by taking the partial derivative of the global total loss with respect to the extrinsic variables, and acts on the six-degree-of-freedom extrinsic parameter increments; the third gradient is generated by taking the partial derivative of the global total loss with respect to the positioning variables, and acts on the vehicle positioning increments; the fourth gradient is generated by taking the partial derivative of the global total loss with respect to the Gaussian 3D geometry and appearance variables, and acts on the 4D Gaussian sputtering scene state. This parallel gradient decomposition method ensures that different variables obtain homogeneous partial derivative information within the same iteration step.

[0088] Furthermore, the trajectory correction amount is obtained based on the first gradient, and the six-degree-of-freedom extrinsic parameter increment and the main vehicle positioning increment are updated synchronously using the second and third gradients. At the same time, the 4D Gaussian sputtering scene state is updated synchronously using the fourth gradient, resulting in a multi-parameter synchronous joint optimization result.

[0089] For example, within the current optimization iteration cycle, the first gradient can be used to make numerical adjustments along the negative gradient direction of the manifold space to calculate the trajectory correction amount used to compensate for the initial trajectory deviation. Then, the second and third gradients are used to refresh the numerical matrix of the six-degree-of-freedom extrinsic parameter increment and the main vehicle positioning increment in the extrinsic parameter tensor to synchronously compensate for installation deviation and calibration drift. The fourth gradient is used to perform high-concurrency numerical updates of the position, opacity, scaling, color, and time index parameters of each Gaussian point in the scene. By executing the above gradient applications in the same computation loop, the final output is the multi-parameter synchronous joint optimization result corresponding to each calibration parameter and the reconstruction state, thereby deriving the 4D Gaussian sputtering scene reconstruction result of the current frame.

[0090] By directly solving for the first-order partial derivatives of the unified global loss in the same optimization iteration and decomposing it into four independent gradient branches acting on different variables, the extrinsic parameter updates, localization correction, trajectory correction, and Gaussian reconstruction are jointly updated based on the unified global loss function in each iteration step. The gradients automatically allocate their contribution based on the partial derivative matrix during backpropagation, reducing the reliance on manual prior parameter tuning. This not only ensures the spatial geometric consistency of the multi-view surround view system's parameter space but also improves the collaborative convergence speed and multi-view scene reconstruction accuracy under complex and turbulent conditions.

[0091] In one example embodiment of this application, when performing local fine-grained numerical updates and viewpoint rendering reconstruction for a 4D Gaussian sputtering scene state using the fourth gradient, the position, opacity, and scaling parameter update amount of the scene Gaussian points within the visible area of ​​the current frame can be calculated first based on the fourth gradient.

[0092] For example, the fourth gradient acting on the 4D Gaussian sputtering scene state can be extracted from the backpropagation data of the current iteration. At the data composition level, the fourth gradient is a structured partial derivative vector matrix, which can be composed of partial derivative values ​​with respect to the 3D center coordinates of each Gaussian point within the visible region of the current frame, partial derivative values ​​with respect to the opacity scalar, and partial derivative values ​​with respect to the 3D anisotropic scaling matrix. Subsequently, the aforementioned partial derivative values ​​contained in the fourth gradient are obtained and multiplied by a preset learning step size to obtain the numerical update amount including geometric deformation compensation and appearance correction features.

[0093] Furthermore, the numerical update is superimposed on the state tensor of the 4D Gaussian sputtering scene state to obtain the updated 4D Gaussian sputtering scene state.

[0094] For example, the calculated numerical update is superimposed on the global state tensor storing the 4D Gaussian sputtering scene state. During the specific update process, the position change in the numerical update directly corrects the position field in the state tensor, while the opacity change and scaling change synchronously overwrite the corresponding fields, so that the old scene state that was in the previous iteration cycle is directly refreshed into the updated 4D Gaussian sputtering scene state that incorporates the latest multi-view geometric observation information of the current frame.

[0095] Then, based on the updated 4D Gaussian sputtering scene state, image rendering can be performed in the visible view of the current frame to obtain the 4D Gaussian sputtering scene reconstruction result of the current frame.

[0096] For example, the updated 4D Gaussian sputtering scene state is used as the sole real reference source for the current spatiotemporal scene. The previously determined current camera extrinsic parameters and the current vehicle positioning pose are cascaded and reprojected, and Alpha blending and overlay rendering is performed. Finally, a high-definition 3D perspective rendering atlas is output in real time, which serves as the 4D Gaussian sputtering scene reconstruction result for the current frame.

[0097] This embodiment, through the fourth gradient, no longer reconstructs global parameters from the entire global Gaussian map. Instead, it calculates the numerical updates of the position, opacity, and scaling parameters of active Gaussian points within the visible region of the current frame and updates the state tensor. This ensures that geometric distortions caused by multi-view relative calibration drift can be corrected for geometric and appearance parameters through gradient updates of the Gaussian state tensor. By performing real-time image rendering based on the updated state tensor, the rendering result converges to the true image features in the same optimization iteration, improving the timeliness and texture fidelity of online high-precision reconstruction in dynamic bumpy environments for the vehicle-side system.

[0098] like Figure 2 As shown, Figure 2This application discloses a pose optimization and extrinsic parameter compensation device for 4D Gaussian sputtering scene reconstruction according to an exemplary embodiment. The device includes: an acquisition module 210, a determination module 220, a processing module 230, a construction module 240, and an output module 250. The acquisition module 210 is used to acquire the current frame image data collected by the multi-view camera during the vehicle's movement, the initial trajectory parameters of the main vehicle corresponding to the current frame, the initial external parameters of each camera, and the initial positioning pose of the main vehicle, and to establish the 4D Gaussian sputtering scene state to be optimized. The determination module 220 is used to determine the current camera extrinsic parameters based on the initial extrinsic parameters of each camera and the corresponding six-degree-of-freedom extrinsic parameter increments, and to determine the current master vehicle positioning pose based on the initial positioning pose of the master vehicle and the corresponding master vehicle positioning increments. The master vehicle positioning increments and the six-degree-of-freedom extrinsic parameter increments adopt the same manifold perturbation update parameterization form, and the two are configured independently and updated synchronously under the same global total loss. Processing module 230 is used to perform continuous six-dimensional rotating manifold parameterization on the rotation component in the initial trajectory parameters to obtain the trajectory parameters of the manifold to be optimized. The construction module 240 is used to project and render the 4D Gaussian sputtering scene state to the camera view of the current frame to obtain a rendered image based on the current camera extrinsic parameters, the current vehicle positioning pose, the manifold trajectory parameters to be optimized, and the translation components in the initial trajectory parameters. It also constructs the global total loss of the current frame based on the photometric error between the rendered image and the corresponding current frame image data, the reprojection difference in the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint. Output module 250 is used to solve the gradient of the global total loss, and update and correct the trajectory parameters and translation components of the manifold to be optimized according to the gradient to obtain the trajectory correction amount. At the same time, it synchronously updates the six-degree-of-freedom extrinsic parameter increment, the main vehicle positioning increment and the 4D Gaussian sputtering scene state according to the gradient, and outputs the 4D Gaussian sputtering scene reconstruction result of the current frame.

[0099] The specific implementation process of the functions and roles of each module in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0100] The embodiment of the pose optimization and extrinsic parameter compensation device for 4D Gaussian sputtering scene reconstruction in this application can be applied to computer devices, such as servers or terminal devices. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor reading the corresponding computer program instructions from non-volatile memory into memory and executing them. From a hardware perspective, such as... Figure 3The diagram shown is a hardware structure diagram of a computer device for pose optimization and extrinsic parameter compensation for 4D Gaussian sputtering scene reconstruction according to an embodiment of this application. Except for... Figure 3 In addition to the processor 310, memory 330, network interface 320, and non-volatile memory 340 shown, the server or electronic device where the pose optimization and extrinsic parameter compensation device for 4D Gaussian sputtering scene reconstruction is located in the embodiment may also include other hardware depending on the actual function of the computer device, which will not be described in detail here.

[0101] In one example embodiment of this application, a computer-readable storage medium is also provided, on which a program 400 capable of implementing the methods described above is stored. In some possible embodiments, various aspects of this application may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this application.

[0102] In the description of this application, the terms "first" and "second" are used for description only and do not indicate relative importance or imply the number of technical features. Therefore, the features referred to as "first" or "second" may explicitly or implicitly include at least one of those features. "A plurality of" means at least two, unless otherwise expressly defined.

[0103] Through the description of the above embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0104] Finally, the above preferred embodiments are only used to illustrate the technical solutions of this application and are not restrictive. Although this application has been described in detail, those skilled in the art should understand that changes in form and detail can be made without departing from the scope defined by the claims of this application. The dimensions in the drawings are not related to the specific physical object, and the physical object dimensions can be arbitrarily changed.

Claims

1. A pose optimization and extrinsic parameter compensation method for 4D Gaussian sputtering scene reconstruction, characterized in that, The method includes: Acquire the current frame image data collected by the multi-camera during vehicle movement, the initial trajectory parameters of the main vehicle corresponding to the current frame, the initial extrinsic parameters of each camera, and the initial positioning pose of the main vehicle, and establish the 4D Gaussian sputtering scene state to be optimized. The current camera extrinsic parameters are determined based on the initial extrinsic parameters of each camera and the corresponding six-degree-of-freedom extrinsic parameter increments. The current master vehicle positioning pose is determined based on the initial positioning pose of the master vehicle and the corresponding master vehicle positioning increments. The master vehicle positioning increments and the six-degree-of-freedom extrinsic parameter increments adopt the same manifold perturbation update parameterization form. The two are configured independently and updated synchronously under the same global total loss. The rotational components in the initial trajectory parameters are subjected to continuous six-dimensional rotational manifold parameterization to obtain the trajectory parameters of the manifold to be optimized. Based on the current camera extrinsic parameters, the current vehicle positioning pose, the manifold trajectory parameters to be optimized, and the translation component in the initial trajectory parameters, the 4D Gaussian sputtering scene state is projected and rendered to the camera view of the current frame to obtain a rendered image. Based on the photometric error between the rendered image and the corresponding current frame image data, the reprojection difference in the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint, the global total loss of the current frame is constructed. The gradient of the global total loss is calculated, and the trajectory parameters of the manifold to be optimized and the translation components are updated and corrected according to the gradient to obtain the trajectory correction amount. At the same time, the six-degree-of-freedom extrinsic parameter increment, the main vehicle positioning increment and the 4D Gaussian sputtering scene state are updated synchronously according to the gradient, and the 4D Gaussian sputtering scene reconstruction result of the current frame is output.

2. The method according to claim 1, characterized in that, The step of constructing the global total loss for the current frame based on the photometric error between the rendered image and the corresponding current frame image data, the reprojection difference within the overlapping field of view of adjacent cameras, the vehicle positioning loss, and trajectory smoothing constraints includes: The photometric error is determined based on the pixel residual between the rendered image and the corresponding current frame image data within the effective observation area; The reprojection difference within the overlapping field of view of adjacent cameras is determined based on the projection position deviation of the same 4D Gaussian sputtering scene state under different camera perspectives. The main vehicle positioning loss is calculated based on the current main vehicle positioning pose and the main vehicle positioning increment. The trajectory smoothing constraint is determined based on the manifold trajectory parameters to be optimized, the translation component, and the pose change between the main vehicle trajectory parameters and those of historical adjacent frames; The photometric error, the reprojection difference within the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint are normalized and weighted to obtain the global total loss of the current frame.

3. The method according to claim 2, characterized in that, The step of determining the reprojection difference within the overlapping field of view of adjacent cameras based on the projection position deviation of the same 4D Gaussian sputtering scene state under different camera viewpoints includes: The 4D Gaussian sputtering scene state is projected onto the first camera view and the second camera view, which has an overlapping field of view with the first camera view, respectively, to obtain the first projection result and the second projection result. Based on the current camera extrinsic parameters, the first projection result is transformed into the coordinate system of the second camera's viewpoint to obtain a reference projection result; Based on the positional residual between the reference projection result and the second projection result, the reprojection difference within the overlapping field of view of the adjacent cameras is determined; Wherein, the first camera viewpoint is used to characterize the observation viewpoint of any camera, and the second camera viewpoint is used to characterize the observation viewpoint of an adjacent camera that has an overlapping field of view with the first camera.

4. The method according to claim 1, characterized in that, The step of determining the current camera extrinsic parameters based on the initial extrinsic parameters of each camera and the corresponding six-degree-of-freedom extrinsic parameter increments, and determining the current master vehicle positioning pose based on the initial positioning pose of the master vehicle and the corresponding master vehicle positioning increments, includes: The initial extrinsic parameters of each camera, the increment of each six-degree-of-freedom extrinsic parameter, and the preset perturbation superposition operator are used to perform manifold exponential mapping calculation to obtain the current camera extrinsic parameters of each camera. The current vehicle positioning pose is obtained by performing manifold index mapping calculation on the main vehicle positioning increment, the initial positioning pose, and the disturbance superposition operator.

5. The method according to claim 1, characterized in that, The step of performing continuous six-dimensional rotating manifold parameterization on the rotational components in the initial trajectory parameters to obtain the trajectory parameters of the manifold to be optimized includes: Extract the three-dimensional rotation component from the initial trajectory parameters; The three-dimensional rotational components are projected into the continuous rotating manifold representation space to obtain the trajectory parameters of the manifold to be optimized, which are composed of six-dimensional manifold vectors.

6. The method according to claim 1, characterized in that, The gradient of the global total loss is calculated, and the trajectory parameters of the manifold to be optimized and the translation components are updated and corrected according to the gradient to obtain the trajectory correction amount. At the same time, the six-degree-of-freedom extrinsic parameter increment, the main vehicle positioning increment, and the 4D Gaussian sputtering scene state are updated synchronously according to the gradient, and the 4D Gaussian sputtering scene reconstruction result of the current frame is output, including: The global gradient vector is obtained by calculating the first-order partial derivative of the global total loss. In the current optimization iteration, the global gradient vector is decomposed into a first gradient, a second gradient, a third gradient, and a fourth gradient that are independent of each other. The first gradient acts on the manifold trajectory parameters to be optimized and the translation component, the second gradient acts on the six-degree-of-freedom extrinsic parameter increment, the third gradient acts on the vehicle positioning increment, and the fourth gradient acts on the 4D Gaussian sputtering scene state. Based on the first gradient, the trajectory correction amount is obtained, and the second gradient and the third gradient are used to update the six-degree-of-freedom extrinsic parameter increment and the main vehicle positioning increment simultaneously. At the same time, the fourth gradient is used to update the 4D Gaussian sputtering scene state simultaneously, so as to obtain the multi-parameter synchronous joint optimization result.

7. The method according to claim 6, characterized in that, The process of updating the 4D Gaussian sputtering scene state using the fourth gradient synchronization to obtain multi-parameter synchronization joint optimization results includes: Based on the fourth gradient, calculate the position, opacity, and scaling parameter update amount of the scene Gaussian point within the visible area of ​​the current frame; The numerical update is superimposed and updated into the state tensor of the 4D Gaussian sputtering scene state to obtain the updated 4D Gaussian sputtering scene state. Based on the updated 4D Gaussian sputtering scene state, image rendering is performed in the visible view of the current frame to obtain the 4D Gaussian sputtering scene reconstruction result of the current frame.

8. A pose optimization and extrinsic parameter compensation device for 4D Gaussian sputtering scene reconstruction, characterized in that, include: The acquisition module is used to acquire the current frame image data collected by the multi-view camera during the vehicle's movement, the initial trajectory parameters of the main vehicle corresponding to the current frame, the initial external parameters of each camera, and the initial positioning pose of the main vehicle, and to establish the 4D Gaussian sputtering scene state to be optimized. The determination module is used to determine the current camera extrinsic parameters based on the initial extrinsic parameters of each camera and the corresponding six-degree-of-freedom extrinsic parameter increments, and to determine the current master vehicle positioning pose based on the initial positioning pose of the master vehicle and the corresponding master vehicle positioning increments. The master vehicle positioning increments and the six-degree-of-freedom extrinsic parameter increments adopt the same manifold perturbation update parameterization form, and the two are configured independently and updated synchronously under the same global total loss. The processing module is used to perform continuous six-dimensional rotational manifold parameterization on the rotational components in the initial trajectory parameters to obtain the trajectory parameters of the manifold to be optimized. The construction module is used to project and render the 4D Gaussian sputtering scene state to the camera view of the current frame to obtain a rendered image based on the current camera extrinsic parameters, the current vehicle positioning pose, the manifold trajectory parameters to be optimized, and the translation component in the initial trajectory parameters. It also constructs the global total loss of the current frame based on the photometric error between the rendered image and the corresponding current frame image data, the reprojection difference in the overlapping field of view of adjacent cameras, the vehicle positioning loss, and the trajectory smoothing constraint. The output module is used to calculate the gradient of the global total loss, update and correct the trajectory parameters of the manifold to be optimized and the translation component according to the gradient to obtain the trajectory correction amount, and simultaneously update the six-degree-of-freedom extrinsic parameter increment, the main vehicle positioning increment and the 4D Gaussian sputtering scene state according to the gradient, and output the 4D Gaussian sputtering scene reconstruction result of the current frame.

9. An electronic device, characterized in that, include: processor; as well as A memory storing computer-readable instructions that, when executed by the processor, implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program code instructions, which, when executed by a processor, implement the method as described in any one of claims 1 to 7.

Images