Method and system for converting road end view to vehicle end view based on three-dimensional Gaussian splashing

Abstract

The invention belongs to the technical field of image processing, and particularly relates to a method and system for converting a road end view to a vehicle end view based on three-dimensional Gaussian splashing, and the method comprises the steps: generating a pseudo view through radar depth completion and self-adaptive depth deformation according to a small number of real road end views, carrying out the hole completion, taking the optimized pseudo view as an image at a source view angle, and carrying out the self-adaptive depth deformation based on the pseudo view. And the pseudo view is continuously generated and the holes are complemented, so that a large number of vehicle end views are obtained, the view angles are dense, the trueness is high, the data collection efficiency and the reliability of the synthesized data set are improved, and the requirement of automatic driving on data is met.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to a method and system for converting a road-end view to a vehicle-end view based on three-dimensional Gaussian splashing. Background Art

[0002] In recent years, end-to-end autonomous driving systems have made significant progress and can directly convert sensor inputs into control signals. However, one of the main challenges faced by end-to-end autonomous driving is the need for a large amount of training data to achieve reliable performance. These large datasets are crucial for training models that can handle complex and dynamic environments.

[0003] Currently, the main methods for obtaining autonomous driving data can be divided into dedicated data collection fleets, production vehicle fleets, and synthetic datasets. However, each of these methods faces significant challenges. Although the datasets provided by dedicated data collection fleets offer real environmental data, data collection is relatively expensive due to the high operating costs of vehicles, sensors, and safety drivers. Although production vehicle fleets can generate a large amount of real-world driving data, they face data privacy and high data transmission cost issues. These limitations have led to an increasing adoption of synthetic dataset technologies, which are not only cost-effective and efficient but also can generate diverse scenarios. Synthetic datasets can generally be divided into the following categories: data rendering using game engines, generating driving scenarios using generative models, and view transformation or adding additional objects based on real-world datasets. However, there are significant challenges in transitioning from synthetic data to practical applications because algorithms must handle the authenticity gap to ensure robust performance in real-world environments. In addition, some methods propose synthesizing datasets by transforming perspectives of real datasets. However, this method is limited by the scale of the original collected data and has limited scalability. Summary of the Invention

[0004] The present invention provides a method and system for converting a road-end view to a vehicle-end view based on three-dimensional Gaussian splashing.

[0005] The technical solution of the present invention is as follows:

[0006] The present invention provides a method for converting a road-end view to a vehicle-end view based on three-dimensional Gaussian splashing, including the following steps:

[0007] S1: Collect a road-end view, use monocular depth estimation to obtain a monocular disparity map, calculate the monocular depth value, and complete the radar depth according to the monocular depth value and optimizable parameters to obtain an image with corrected depth;

[0008] S2: Process the image with corrected depth through three-dimensional Gaussian splashing to obtain an image under the source view;

[0009] S3: Project the image from the source perspective to obtain the three-dimensional coordinates in the source perspective; perform adaptive depth deformation based on the three-dimensional coordinates in the source perspective to obtain the three-dimensional coordinates in the pseudo perspective. The operations are as follows:

[0010] In the spatial coordinate system, if the change amount of the z-axis is 0, set the z-axis value of this point to the minimum depth value of the depth image in the source perspective, and control the displacement deformation amounts of the x-axis and y-axis according to the z-axis value of this point, the camera internal parameters, and the preset deformation amount;

[0011] If the change amount of the z-axis is not 0, set the change amount of the z-axis of this point to the preset value, and control the displacement deformation amounts of the x-axis and y-axis according to the change amount of the z-axis of this point, the three-dimensional coordinates in the source perspective, the camera internal parameters, and the preset deformation amount;

[0012] Based on the three-dimensional coordinates in the source perspective and the corresponding displacement deformation amounts of the x-axis and y-axis, obtain the three-dimensional coordinates in the pseudo perspective;

[0013] Project the three-dimensional coordinates in the pseudo perspective onto the camera plane to generate the image in the pseudo perspective;

[0014] S4: The image in the pseudo perspective is successively subjected to forward noise addition and reverse noise removal to obtain an optimized pseudo view; use the optimized pseudo view as the image in the source perspective, and execute S3 until the preset execution times are reached;

[0015] S5: Calculate the loss between the image in the source perspective and the image after depth correction, and the loss between the optimized pseudo view and its corresponding rendered image. After subjecting the optimized pseudo view that meets the preset conditions to three-dimensional Gaussian splashing processing, use it as the vehicle-end view.

[0016] In the above S1, according to the monocular depth value and the optimizable parameters, complete the radar depth to obtain the image after depth correction, specifically:

[0017] According to the formula: Complete the radar depth; where D align is the radar depth value after completion, d mono is the monocular disparity map, and c1, c2, c3 are optimizable parameters;

[0018] Calculate the consistency between the monocular depth value and the radar depth value after completion. If the consistency is not greater than the preset value, optimize the optimizable parameters by using the non-linear least squares method with Huber loss, and use the optimized optimizable parameters to complete the radar depth to obtain the image after depth correction.

[0019] Furthermore, the optimization of the optimizable parameters by using the non-linear least squares method with Huber loss is based on the formula:

[0020] Implementation;

[0021] Wherein, is the Huber loss function, D lidar is the radar depth, is the data pair of the radar depth and the monocular disparity map, and (i, j) is the pixel coordinate.

[0022] In step S3, according to the z-axis value of this point, the camera internal parameters and the preset deformation amount, control the displacement deformation amounts of the x-axis and y-axis, which is implemented according to the formula: Implementation;

[0023] Wherein, ε is the preset deformation amount, f x , f y are the focal lengths in the camera internal parameters, Δt x , Δt y are the displacement deformation amounts of the x-axis and y-axis respectively, and z is the z-axis value of this point under the source view.

[0024] In step S3, according to the change amount of the z-axis of this point, the three-dimensional coordinates under the source view, the camera internal parameters and the preset deformation amount, control the displacement deformation amounts of the x-axis and y-axis, which is implemented according to the formula: Implementation;

[0025] Wherein, ε is the preset deformation amount, f x , f y are the focal lengths in the camera internal parameters, c x , c y are the principal point coordinates in the camera internal parameters, Δt x , Δt y , Δt z are the displacement deformation amounts of the x-axis, y-axis and z-axis respectively, and x, y are the x-axis and y-axis values of this point under the source view.

[0026] In step S5, calculate the loss between the image under the source view and the image after depth correction, and the loss between the optimized pseudo-view and its corresponding rendered image, which is implemented according to the formula: and Implementation;

[0027] Wherein, is the loss between the image under the source view and the image after depth correction, λ is the loss weight, I are the rendered image and the image after depth correction under the source view respectively, D are the rendered image depth and the image depth after depth correction under the source view respectively, SSIM is the SSIM loss function, W is the confidence weight, is the loss between the optimized pseudo-view and its corresponding rendered image, is the mathematical expectation, λ1 is the weight of the loss function, and L2 is the L2-norm loss function. I' are the rendered image and the optimized pseudo-view respectively.

[0028] In step S4, the image under the pseudo-view is successively subjected to forward noise addition and reverse denoising to obtain an optimized pseudo-view. The operations are as follows:

[0029] S41: Based on the image under the pseudo-view, obtain the image and the mask representing the holes, map the image and the mask to the latent space, and obtain the latent space image;

[0030] S42: After adding noise to the latent space image using a random noise image, in each time step, fuse the non-mask region and the mask region in the noise map to obtain a new noise map;

[0031] S43: Denoise the new noise map according to the time step until the denoising is completed to obtain an optimized pseudo-view.

[0032] In step S1, use monocular depth estimation to obtain a monocular disparity map and calculate the monocular depth value. Specifically:

[0033] According to the formula: Implement;

[0034] In the formula, D mono is the monocular depth value, d mono is the monocular disparity map, b train , f train are the baseline and focal length of the monocular depth estimation model respectively.

[0035] The present invention also provides a road-end view to vehicle-end view conversion system based on three-dimensional Gaussian splash, including:

[0036] Depth completion module: used to collect the road-end view, use monocular depth estimation to obtain a monocular disparity map, calculate the monocular depth value, and complete the radar depth according to the monocular depth value and the optimizable parameters to obtain an image with corrected depth;

[0037] Pre-training module: used to perform three-dimensional Gaussian splash processing on the image with corrected depth to obtain an image under the source view;

[0038] Adaptive depth deformation module: project the image under the source view to obtain the three-dimensional coordinates under the source view; perform adaptive depth deformation according to the three-dimensional coordinates under the source view to obtain the three-dimensional coordinates under the pseudo-view. The operations are as follows:

[0039] In the space coordinate system, if the change amount of the z-axis is 0, set the z-axis value of this point to the minimum depth value of the depth image under the source view, and control the displacement deformation amounts of the x-axis and y-axis according to the z-axis value of this point, the camera internal parameters, and the preset deformation amount;

[0040] If the change amount of the z-axis is not 0, set the change amount of the z-axis of this point to a preset value, and control the displacement deformation amounts of the x-axis and y-axis according to the change amount of the z-axis of this point, the three-dimensional coordinates in the source view, the camera internal parameters, and the preset deformation amount;

[0041] Based on the three-dimensional coordinates in the source view and the corresponding displacement deformation amounts of the x-axis and y-axis, obtain the three-dimensional coordinates in the pseudo view;

[0042] Project the three-dimensional coordinates in the pseudo view onto the camera plane to generate an image in the pseudo view;

[0043] Hole filling module: The image in the pseudo view is sequentially subjected to forward noise addition and reverse denoising to obtain an optimized pseudo view; Use the optimized pseudo view as the image in the source view and enter the adaptive depth deformation module until the preset number of executions is reached;

[0044] Optimization module: Used to calculate the loss between the image in the source view and the image after corrected depth, and the loss between the optimized pseudo view and its corresponding rendered image. After the optimized pseudo view that meets the preset conditions is processed by three-dimensional Gaussian splashing, it is used as the vehicle-end view.

[0045] Beneficial effects

[0046] According to a small number of real roadside views, the present invention generates pseudo views through radar depth completion, adaptive depth deformation, and after hole filling, uses the optimized pseudo view as the image in the source view, continues to generate pseudo views and hole filling, so as to obtain a large number of vehicle-end views, with dense perspectives and high authenticity, improving the efficiency of data collection and the reliability of the synthetic data set, and meeting the data requirements of autonomous driving. Description of the drawings

[0047] Figure 1 It is a schematic flow chart of a method for converting a roadside view to a vehicle-end view based on three-dimensional Gaussian splashing according to the present invention. Specific implementation manners

[0048] The following embodiments are intended to illustrate the present invention rather than further limit the present invention.

[0049] The present invention provides a method for converting a roadside view to a vehicle-end view based on three-dimensional Gaussian splashing, as Figure 1 shown, including the following steps:

[0050] S1: Collect roadside views, use monocular depth estimation to obtain a monocular disparity map, calculate the monocular depth value, and complete the radar depth according to the monocular depth value and the optimizable parameters to obtain an image after corrected depth.

[0051] In the first stage, dense monocular depth is used to complement sparse radar depth to provide supervision for subsequent training. Specifically as follows:

[0052] First, a monocular disparity map is obtained using monocular depth estimation, and the monocular depth value is calculated. Specifically:

[0053] According to the formula: Implement;

[0054] In the formula, D mono is the monocular depth value, d mono is the monocular disparity map, b train , f train are the baseline and focal length of the monocular depth estimation model respectively.

[0055] Then, after obtaining the monocular depth value, its scale needs to be changed to the real scale to complement the radar depth. This process requires constructing a conversion formula from the monocular disparity map to the depth map. Preferably, according to the monocular depth value and the optimizable parameters, the radar depth is complemented to obtain an image with corrected depth. Specifically:

[0056] According to the formula: Complement the radar depth; in the formula, D align is the complemented radar depth value, d mono is the monocular disparity map, and c1, c2, c3 are optimizable parameters;

[0057] Calculate the consistency between the monocular depth value and the complemented radar depth value. If the consistency is not greater than the preset value, the optimizable parameters are optimized by non-linear least squares with Huber loss, and the radar depth is complemented using the optimized optimizable parameters to obtain an image with corrected depth.

[0058] In addition, the optimization of the optimizable parameters by non-linear least squares with Huber loss is according to the formula:

[0059] Implement;

[0060] In the formula, is the Huber loss function, D lidar is the radar depth, is the data pair of the radar depth and the monocular disparity map, and (i, j) is the pixel coordinate.

[0061] For example, when the consistency between the monocular depth value and the complemented radar depth value is not greater than 80%, the optimizable parameters are optimized.

[0062] S2: The image with corrected depth is processed by three-dimensional Gaussian splashing to obtain the image under the source view angle.

[0063] In the second stage, the 3D Gaussian Splatting (3DGS) can be pre-trained from the sparse four end-view images so that it can learn the structure and color of the scene.

[0064] Calculate the loss between the image (rendered image and depth) in the source view and the image after depth correction (real image and depth), specifically:

[0065]

[0066] In the formula, is the loss between the image in the source view and the image after depth correction, λ is the loss weight, for example, λ = 0.8, I are the rendered image and the image after depth correction in the source view respectively, D are the depth of the rendered image and the depth of the image after depth correction in the source view respectively, and SSIM is the SSIM loss function, which is used for the operation of structural similarity index measurement.

[0067] S3: Project the image in the source view to obtain the 3D coordinates in the source view; perform adaptive depth deformation according to the 3D coordinates in the source view to obtain the 3D coordinates in the pseudo-view.

[0068] In the third stage, considering that training with only four images will cause overfitting of 3DGS, resulting in poor rendering results in new views. Therefore, the depth warp technology is used to generate pseudo-views near the training views. Specifically:

[0069] In the spatial coordinate system, if the change amount of the z-axis is 0, then set the z-axis value of this point to the minimum depth value of the depth image in the source view, and control the displacement deformation amounts of the x-axis and y-axis according to the z-axis value of this point, the camera internal parameters, and the preset deformation amount;

[0070] If the change amount of the z-axis is not 0, then set the change amount of the z-axis of this point to the preset value, and control the displacement deformation amounts of the x-axis and y-axis according to the change amount of the z-axis of this point, the 3D coordinates in the source view, the camera internal parameters, and the preset deformation amount;

[0071] Based on the 3D coordinates in the source view and the corresponding displacement deformation amounts of the x-axis and y-axis, obtain the 3D coordinates in the pseudo-view;

[0072] Project the 3D coordinates in the pseudo-view onto the camera plane to generate the image in the pseudo-view.

[0073] Preferably, in one case, control the displacement deformation amounts of the x-axis and y-axis according to the z-axis value of this point, the camera internal parameters, and the preset deformation amount,

[0074] is implemented according to the formula: ;

[0075] In the formula, ε is the preset deformation amount, f x , f y is the focal length in the camera internal parameters, Δt x , Δt y are the displacement deformation amounts of the x-axis and y-axis respectively, and z is the z-axis value of this point under the source view angle.

[0076] In one case, according to the change amount of the z-axis of this point, the three-dimensional coordinates under the source view angle, the camera internal parameters and the preset deformation amount, control the displacement deformation amounts of the x-axis and y-axis, which is realized according to the formula: Realize;

[0077] In the formula, ε is the preset deformation amount, f x , f y is the focal length in the camera internal parameters, c x , c y are the principal point coordinates in the camera internal parameters, Δt x , Δt y , Δt z are the displacement deformation amounts of the x-axis, y-axis and z-axis respectively, and x and y are the x-axis and y-axis values of this point under the source view angle.

[0078] The specific derivation process of the above formula is as follows:

[0079] (1) Project the 3D point p = (x, y, z) rendered under the source view angle V into a new view V', and the formula is: p' = KR'R -1 K -1 p + K(T' - R'R -1 T);

[0080] Among them, p is the point cloud under the source view angle V obtained by depth map projection, and p', R' and T' are the three-dimensional point, rotation matrix and translation vector of the new view V' respectively. K is the camera internal parameter, T is the translation vector of the source view angle V, and R is the rotation matrix of the source view angle V.

[0081] (2) The 3D point under the new view V' can be projected onto the camera plane through the projection formula to generate the image under the new view V', and the formula is: w norm is the coefficient of the normalized homogeneous coordinate system.

[0082] However, directly applying depth deformation with fixed displacement and rotation usually leads to two error situations: 1) Excessive deformation due to excessive displacement, thus magnifying geometric errors and causing a distorted illusion; 2) Insufficient deformation due to insufficient displacement, thus limiting the change of the view angle. To balance this trade-off, adaptive depth deformation is used, which is an adaptive depth deformation strategy restricted by pixel-level spatial consistency.

[0083] (3) For the three-dimensional points observed in the source view V, let represent the pixel displacement in the new view V'. Replace in the formula in (2) with to obtain the following displacement relationship:

[0084]

[0085] where represents the displacement distance difference between the two views, and f x , f y , c x , c y are the camera internal parameters.

[0086] (4) Given an ideal deformation ε such that all pixel displacements do not exceed ε to achieve the effect of controlling the deformation:

[0087] (5) After combining the formulas in (3) and (4), obtain inequalities regarding ε, Δt x , Δt y , Δt z , x, y, and z:

[0088]

[0089] (6) Solving this inequality requires considering two cases:

[0090] One is that Δt z = 0, and the formula can be simplified to:

[0091]

[0092] Given that f x and f y are fixed, ε is predefined, and z is set to the minimum value in the depth map, the relative transformation Δt x and Δt y can be controlled to set an appropriate depth deformation. In actual processing, after controlling the z-axis value, camera internal parameters, and predefined deformation amount ε, there will be a range of values for the displacement deformations of the x-axis and y-axis. Usually, the maximum value of this range of values is taken.

[0093] One is that Δt z ≠ 0, and the formula can be simplified to:

[0094]

[0095] Since only one variation can be controlled in each inequality, Δt z can be set to a fixed value to control Δt x and Δty To set an appropriate depth warp.

[0096] Through the above operations, find an appropriate displacement T', generate a new view V', and provide dense training views for 3DGS.

[0097] S4: After the images under the pseudo-view are successively subjected to forward noise addition and reverse denoising, optimized pseudo-views are obtained; the optimized pseudo-views are used as the images under the source view, and S3 is executed until a preset number of executions is reached.

[0098] Considering that when generating pseudo-views, many holes will be generated due to occlusion relationships. These holes will affect obtaining a larger range of pseudo-views. Therefore, in the fourth stage, these holes are filled by forward noise addition and reverse denoising, and a cascading method is also used to fill them to make the content between the holes consistent.

[0099] Preferably, after the images under the pseudo-view are successively subjected to forward noise addition and reverse denoising, optimized pseudo-views are obtained, and the operations are as follows:

[0100] S41: Based on the images under the pseudo-view, obtain the images and masks representing the holes, and a pre-trained Variational Auto-Encoder (VAE) can be used to map the images and masks to the latent space to obtain latent space images;

[0101] S42: After adding noise to the latent space images using random noise images, in each time step, the non-mask regions and the mask regions in the noise map are fused to obtain a new noise map;

[0102] For example, apply a fixed noise to the latent space image F to obtain a noise-added picture M1; for the regions in the mask, in each time step, generate corresponding noise-added image features M2 according to the time step; in each time step, fuse the two noise-added images through the mask: M = M1 * mask + (1.0 - mask) * M2. That is, the non-mask regions in the original image and the mask regions in the noise map are fused to obtain a new noise map M. This ensures that only the mask regions are reconstructed, while the other regions remain unchanged with the original information.

[0103] S43: Denoise the new noise map according to the time step until the denoising is completed to obtain optimized pseudo-views.

[0104] S5: Calculate the loss between the images under the source view and the images after depth correction, and the loss between the optimized pseudo-views and their corresponding rendered images. After subjecting the optimized pseudo-views that meet the preset conditions to three-dimensional Gaussian splashing, they are used as the vehicle-side views.

[0105] Although the virtual image can be effectively completed through forward noise addition and reverse denoising, the random generation process may lead to semantic inconsistencies with the actual scene. Therefore, this application adopts an optimization method guided by confidence, and uses multi-view consensus to weight the supervision signals. The L2 norm loss function is used to detect pixel alignment, and the SSIM loss is used to evaluate the perceptual similarity at the structural level.

[0106] Preferably, the loss between the optimized pseudo-view and its corresponding rendered image is calculated according to the formula:

[0107]

[0108] implemented;

[0109] where W is the confidence weight, is the loss between the optimized pseudo-view and its corresponding rendered image, is the mathematical expectation, λ1 is the loss function weight, which can be set to 0.8, L2 is the L2 norm loss function, I' is the optimized pseudo-view, is the rendered image, which is an image obtained by three-dimensional Gaussian splashing with the same camera extrinsic parameters as the pseudo-view.

[0110] Therefore, during the 3DGS training to obtain the vehicle-end view, the total loss function is:

[0111] According to a small number of real roadside views, the present invention generates pseudo-views through radar depth completion, adaptive depth deformation, and after hole completion, takes the optimized pseudo-views as the images in the source view, and continues to generate pseudo-views and hole completion, thereby obtaining a large number of vehicle-end views with dense perspectives and high authenticity, improving the efficiency of data collection and the reliability of the synthetic dataset, and meeting the data requirements for autonomous driving.

[0112] The present invention also provides a roadside view to vehicle-end view conversion system based on three-dimensional Gaussian splashing, including:

[0113] Depth completion module: used to collect roadside views, obtain a monocular disparity map using monocular depth estimation, calculate the monocular depth value, and complete the radar depth according to the monocular depth value and optimizable parameters to obtain an image with corrected depth;

[0114] Pre-training module: used to process the image with corrected depth through three-dimensional Gaussian splashing to obtain an image in the source view;

[0115] Adaptive depth deformation module: project the image in the source view to obtain the three-dimensional coordinates in the source view; perform adaptive depth deformation according to the three-dimensional coordinates in the source view to obtain the three-dimensional coordinates in the pseudo-view, and the operations are as follows:

[0116] In the spatial coordinate system, if the change in the z-axis is 0, the z-axis value of this point is set to the minimum depth value of the depth image in the source view. According to the z-axis value of this point, the camera internal parameters, and the preset deformation amount, the displacement deformation amounts of the x-axis and y-axis are controlled;

[0117] If the change in the z-axis is not 0, the change in the z-axis of this point is set to the preset value. According to the change in the z-axis of this point, the three-dimensional coordinates in the source view, the camera internal parameters, and the preset deformation amount, the displacement deformation amounts of the x-axis and y-axis are controlled;

[0118] Based on the three-dimensional coordinates in the source view and the corresponding displacement deformation amounts of the x-axis and y-axis, the three-dimensional coordinates in the pseudo view are obtained;

[0119] The three-dimensional coordinates in the pseudo view are projected onto the camera plane to generate an image in the pseudo view;

[0120] Hole filling module: The image in the pseudo view is successively subjected to forward noise addition and reverse denoising to obtain an optimized pseudo view; the optimized pseudo view is used as the image in the source view and enters the adaptive depth deformation module until the preset number of executions is reached;

[0121] Optimization module: Used to calculate the loss between the image in the source view and the image after depth correction, and the loss between the optimized pseudo view and its corresponding rendered image. The optimized pseudo view that meets the preset conditions is processed by three-dimensional Gaussian splashing and used as the vehicle-end view.

Claims

1. A method for converting a road-end view to a vehicle-end view based on three-dimensional Gaussian splashing, characterized in that It includes the following steps: S1: Collect the road-end view, use monocular depth estimation to obtain a monocular disparity map, calculate the monocular depth value, and complete the radar depth according to the monocular depth value and the optimizable parameters to obtain an image with corrected depth; S2: The image with corrected depth is processed by three-dimensional Gaussian splashing to obtain an image in the source view; S3: Project the image in the source view to obtain the three-dimensional coordinates in the source view; perform adaptive depth deformation according to the three-dimensional coordinates in the source view to obtain the three-dimensional coordinates in the pseudo view. The operations are as follows: In the space coordinate system, if the change amount of the z-axis is 0, set the z-axis value of this point to the minimum depth value of the depth image in the source view, and control the displacement deformation amounts of the x-axis and y-axis according to the z-axis value of this point, the camera internal parameters, and the preset deformation amount; If the change amount of the z-axis is not 0, set the change amount of the z-axis of this point to the preset value, and control the displacement deformation amounts of the x-axis and y-axis according to the change amount of the z-axis of this point, the three-dimensional coordinates in the source view, the camera internal parameters, and the preset deformation amount; Based on the three-dimensional coordinates in the source view and the corresponding displacement deformation amounts of the x-axis and y-axis, obtain the three-dimensional coordinates in the pseudo view; Project the three-dimensional coordinates in the pseudo view onto the camera plane to generate an image in the pseudo view; S4: The image in the pseudo view is successively subjected to forward noise addition and reverse denoising to obtain an optimized pseudo view; use the optimized pseudo view as the image in the source view and execute S3 until the preset execution times are reached; S5: Calculate the loss between the image in the source view and the image with corrected depth, and the loss between the optimized pseudo view and its corresponding rendered image. After the optimized pseudo view that meets the preset conditions is processed by three-dimensional Gaussian splashing, it is used as the vehicle-end view.

2. The method for converting a roadside view to a vehicle-side view based on three-dimensional Gaussian splashing according to claim 1, wherein In the above S1, to complete the radar depth according to the monocular depth value and the optimizable parameters to obtain an image with corrected depth, specifically: According to the formula: Complete the radar depth; where D align is the value of the completed radar depth, d mono is the single-view parallax map, and c1, c2, c3 are optimizable parameters; Calculate the consistency between the monocular depth value and the completed radar depth value. If the consistency is not greater than the preset value, optimize the optimizable parameters by non-linear least squares with Huber loss, and use the optimized optimizable parameters to complete the radar depth to obtain an image with corrected depth.

3. The method for converting a road-end view to a vehicle-end view based on three-dimensional Gaussian splashing according to claim 2, wherein The optimization of the optimizable parameters by the nonlinear least squares method with Huber loss is based on the formula: Implementation; In the formula, is the Huber loss function, D lidar is the radar depth, is the data pair of the radar depth and the monocular disparity map, and (i, j) is the pixel coordinate.

4. The method for converting a road-end view to a vehicle-end view based on three-dimensional Gaussian splashing according to claim 1, wherein The step S3 controls the displacement deformation amounts of the x-axis and the y-axis according to the z-axis value of this point, the internal parameters of the camera, and a preset deformation amount, which is implemented according to the formula: be realized; where ε is the preset deformation amount, f x , f y are the focal lengths in the camera internal parameters, Δt x , Δt y are the displacement deformation amounts of the x-axis and y-axis respectively, and z is the z-axis value of this point under the source viewing angle.

5. The method for converting a road-end view to a vehicle-end view based on three-dimensional Gaussian splashing according to claim 1, wherein S3. According to the change amount of the z-axis at this point, the three-dimensional coordinates under the source view, the camera internal parameters, and the preset deformation amount, control the displacement deformation amounts of the x-axis and the y-axis, which is implemented according to the formula: be realized; In the formula, ε is the preset deformation amount, f x , f y is the focal length in the camera internal parameters, c x , c y is the principal point coordinate in the camera internal parameters, Δt x , Δt y , Δt z are the displacement deformation amounts of the x-axis, y-axis, and z-axis respectively, and x and y are the x-axis and y-axis values of this point under the source viewing angle.

6. The method for converting a road-end view to a vehicle-end view based on three-dimensional Gaussian splashing according to claim 1, wherein In the above S5, to calculate the loss between the image in the source view and the image with corrected depth, and the loss between the optimized pseudo view and its corresponding rendered image, according to the formula: and Implement; In the formula, is the loss between the image from the source view and the image after depth correction, λ is the loss weight, I represents the rendered image from the source view and the image after depth correction respectively, D represents the depth of the rendered image from the source view and the depth of the image after depth correction respectively. SSIM is the SSIM loss function, and W is the confidence weight, is the loss between the optimized pseudo-view and its corresponding rendered image, is the mathematical expectation, λ1 is the weight of the loss function, and L2 is the L2 norm loss function, I' represents the rendered image and the optimized pseudo-view respectively.

7. The method for converting a roadside view to a vehicle-side view based on three-dimensional Gaussian splashing according to claim 1, characterized in that In the above S4, the image in the pseudo view is successively subjected to forward noise addition and reverse denoising to obtain an optimized pseudo view. The operations are as follows: S41: Based on the image in the pseudo view, obtain the image and a mask representing the hole, map the image and the mask to the latent space to obtain a latent space image; S42: After adding noise to the latent space image using a random noise image, in each time step, fuse the non-mask area and the mask area in the noise map to obtain a new noise map; S43: Denoise the new noise map according to the time step until the denoising ends to obtain an optimized pseudo view.

8. The method for converting a roadside view to a vehicle-side view based on three-dimensional Gaussian splashing according to claim 1, characterized in that In the above S1, to use monocular depth estimation to obtain a monocular disparity map and calculate the monocular depth value, specifically: According to the formula: Implement; where D mono is the monocular depth value, d mono is the monocular disparity map, b train , f train are the baseline and focal length of the monocular depth estimation model, respectively.

9. A road-end view to vehicle-end view conversion system based on three-dimensional Gaussian splashing, characterized in that, It includes: Depth Completion Module: It is used to collect road - end views, obtain a monocular disparity map using monocular depth estimation, calculate the monocular depth value, and complete the radar depth according to the monocular depth value and optimizable parameters to obtain an image with corrected depth; Pre - training Module: It is used to perform three - dimensional Gaussian splashing on the image with corrected depth to obtain an image under the source view; Adaptive Depth Deformation Module: Project the image under the source view to obtain three - dimensional coordinates under the source view; perform adaptive depth deformation according to the three - dimensional coordinates under the source view to obtain three - dimensional coordinates under the pseudo - view. The operations are as follows: In the spatial coordinate system, if the change amount of the z - axis is 0, set the z - axis value of this point to the minimum depth value of the depth image under the source view, and control the displacement deformation amounts of the x - axis and y - axis according to the z - axis value of this point, the camera internal parameters, and the preset deformation amount; If the change amount of the z - axis is not 0, set the change amount of the z - axis of this point to the preset value, and control the displacement deformation amounts of the x - axis and y - axis according to the change amount of the z - axis of this point, the three - dimensional coordinates under the source view, the camera internal parameters, and the preset deformation amount; Based on the three - dimensional coordinates under the source view and the corresponding displacement deformation amounts of the x - axis and y - axis, obtain the three - dimensional coordinates under the pseudo - view; Project the three - dimensional coordinates under the pseudo - view onto the camera plane to generate an image under the pseudo - view; Hole Completion Module: The image under the pseudo - view is successively subjected to forward noise addition and reverse denoising to obtain an optimized pseudo - view; Take the optimized pseudo - view as the image under the source view and enter the adaptive depth deformation module until the preset number of executions is reached; Optimization Module: It is used to calculate the loss between the image under the source view and the image with corrected depth, and the loss between the optimized pseudo - view and its corresponding rendered image. After processing the optimized pseudo - view that meets the preset conditions through three - dimensional Gaussian splashing, it is used as the vehicle - end view.

Images