Method and system for converting a road end view to a car end view based on three-dimensional gaussian splatting

Abstract

The present application belongs to the technical field of image processing, and particularly relates to a road-end view to vehicle-end view conversion method and system based on three-dimensional Gaussian splash, wherein according to a small amount of real road-end view, a pseudo view is generated through radar depth completion and adaptive depth deformation, and after hole completion, the optimized pseudo view is taken as an image under a source view angle to continue generating a pseudo view and performing hole completion, so that a large amount of vehicle-end views are obtained, the view angles are dense, the real degree is high, the efficiency of data collection is improved, and the reliability of a synthetic data set is improved, thereby meeting the demand of automatic driving for data.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method and system for converting roadside views to vehicle-side views based on three-dimensional Gaussian splashing. Background Technology

[0002] In recent years, end-to-end autonomous driving systems have made significant progress, enabling them to directly convert sensor inputs into control signals. However, one of the major challenges facing end-to-end autonomous driving is the need for large amounts of training data to achieve reliable performance. These large datasets are crucial for training models capable of handling complex and dynamic environments.

[0003] Currently, the main methods for acquiring autonomous driving data can be categorized into dedicated data collection fleets, production vehicle fleets, and synthetic datasets. However, each of these methods faces significant challenges. While dedicated data collection fleets provide datasets with realistic environmental data, data acquisition is relatively expensive due to the high operating costs of vehicles, sensors, and safety drivers. Production vehicle fleets, while capable of generating large amounts of real-world driving data, face issues of data privacy and high data transmission costs. These limitations have spurred the increasing adoption of synthetic dataset technologies, which are not only cost-effective and efficient but also capable of generating diverse scenarios. Synthetic datasets can generally be categorized as follows: data rendering using game engines, generating driving scenarios using generative models, and view transformations or adding additional objects based on real-world datasets. However, transitioning from synthetic data to practical applications presents significant challenges because algorithms must handle the gap in realism to ensure robust performance in real-world environments. Furthermore, some methods propose synthesizing datasets by transforming perspectives from real-world datasets. However, this approach is limited by the scale of the original collected data, resulting in limited scalability. Summary of the Invention

[0004] This invention provides a method and system for converting roadside views to vehicle-side views based on three-dimensional Gaussian splashing.

[0005] The technical solution of the present invention is as follows:

[0006] This invention provides a method for converting a roadside view to a vehicle-side view based on three-dimensional Gaussian splashing, comprising the following steps:

[0007] S1: Acquire roadside views, obtain monocular disparity maps using monocular depth estimation, calculate monocular depth values, and complete radar depth based on monocular depth values ​​and optimizable parameters to obtain images with corrected depth.

[0008] S2: The depth-corrected image is processed by 3D Gaussian splashing to obtain the image from the source viewpoint;

[0009] S3: Project the image from the source viewpoint to obtain the 3D coordinates from the source viewpoint; perform adaptive depth deformation based on the 3D coordinates from the source viewpoint to obtain the 3D coordinates from the pseudo viewpoint, as follows:

[0010] In the spatial coordinate system, if the change in the z-axis is 0, then the z-axis value of that point is set as the minimum depth value of the depth image under the source view. Based on the z-axis value of that point, the camera intrinsic parameters, and the preset deformation amount, the displacement deformation of the x-axis and y-axis is controlled.

[0011] If the change in the z-axis is not 0, the change in the z-axis of that point is set to a preset value. Based on the change in the z-axis of that point, the three-dimensional coordinates under the source viewpoint, the camera intrinsic parameters, and the preset deformation, the displacement and deformation of the x-axis and y-axis are controlled.

[0012] Based on the three-dimensional coordinates from the source viewpoint and the corresponding displacement and deformation on the x and y axes, the three-dimensional coordinates from the pseudo viewpoint are obtained.

[0013] The 3D coordinates under the pseudo-viewpoint are projected onto the camera plane to generate an image under the pseudo-viewpoint.

[0014] S4: The image under the pseudo-view is subjected to forward noise addition and reverse noise reduction in sequence to obtain an optimized pseudo-view; the optimized pseudo-view is used as the image under the source view, and S3 is executed until the preset number of executions is reached;

[0015] S5: Calculate the loss between the image at the source view and the image after depth correction, as well as the loss between the optimized pseudo-view and its corresponding rendered image. The optimized pseudo-view that meets the preset conditions is then processed by 3D Gaussian splashing and used as the vehicle end view.

[0016] S1, based on the monocular depth value and optimizable parameters, completes the radar depth to obtain a depth-corrected image, specifically as follows:

[0017] According to the formula: Complete the radar depth; where D align To complete the radar depth value, d mono This is a monocular disparity map, where c1, c2, and c3 are optimizable parameters.

[0018] The consistency between the monocular depth value and the completed radar depth value is calculated. If the consistency is not greater than a preset value, the optimizable parameters are optimized using a nonlinear least squares method with Huber loss. The optimized optimizable parameters are then used to complete the radar depth, resulting in an image with corrected depth.

[0019] Furthermore, the optimizable parameters are optimized using a nonlinear least squares method with Huber loss, according to the formula:

[0020] accomplish;

[0021] In the formula, Let D be the Huber loss function. lidar For radar depth, For radar depth and monocular parallax map data pairs, (i, j) are pixel coordinates.

[0022] S3, based on the z-axis value of that point, camera intrinsic parameters, and preset deformation amount, controls the displacement deformation amounts of the x-axis and y-axis, according to the formula: accomplish;

[0023] In the formula, ε is the preset deformation amount, and f x f y Δt is the focal length in the camera's intrinsic parameters. x , Δt y , respectively, represent the displacement deformation along the x-axis and y-axis, and z is the z-axis value of that point from the source viewpoint.

[0024] S3, based on the change in the z-axis of the point, the three-dimensional coordinates under the source viewpoint, the camera intrinsic parameters, and the preset deformation amount, controls the displacement deformation of the x-axis and y-axis, according to the formula: accomplish;

[0025] In the formula, ε is the preset deformation amount, and f x f y c is the focal length in the camera's intrinsic parameters. x c y The coordinates of the principal point in the camera's intrinsic parameters, Δt x , Δt y , Δt z These represent the displacement deformation along the x-axis, y-axis, and z-axis, respectively, where x and y are the x-axis and y-axis values ​​of that point from the source viewpoint.

[0026] S5 calculates the loss between the image at the source viewpoint and the image after depth correction, as well as the loss between the optimized pseudo-view and its corresponding rendered image, based on the formula: and accomplish;

[0027] In the formula, Let λ be the loss between the image at the source viewpoint and the image after depth correction, and let λ be the loss weight. I represents the rendered image from the source viewpoint and the image after depth correction, respectively. D represents the rendered image depth from the source viewpoint, and the image depth after depth correction, respectively; SSIM is the SSIM loss function; and W is the confidence weight. To optimize the loss of the pseudo-view and its corresponding rendered image, Let λ be the mathematical expectation, λ1 be the weight of the loss function, and L2 be the L2 norm loss function. I' represents the rendered image and the optimized pseudo-view, respectively.

[0028] In step S4, the image under the pseudo-viewpoint is sequentially subjected to forward noise addition and inverse noise reduction to obtain an optimized pseudo-viewpoint, as follows:

[0029] S41: Based on the image under the pseudo-view, obtain the image and the mask representing the hole, and map the image and the mask to the latent space to obtain the latent space image;

[0030] S42: After adding noise to the latent space image using random noise image, in each time step, the non-masked region and the masked region in the noise image are fused to obtain a new noise image;

[0031] S43: Based on the time step, denoise the new noise map until the denoising is completed, and obtain the optimized pseudo-view.

[0032] S1 involves obtaining a monocular disparity map using monocular depth estimation and calculating the monocular depth value, specifically as follows:

[0033] According to the formula: accomplish;

[0034] In the formula, D mono d is the monocular depth value. mono For monocular disparity maps, b train f train These are the baseline and focal length of the monocular depth estimation model, respectively.

[0035] The present invention also provides a roadside view to vehicle-side view conversion system based on three-dimensional Gaussian splashing, comprising:

[0036] Depth Completion Module: Used to acquire roadside views, obtain monocular disparity maps using monocular depth estimation, calculate monocular depth values, and complete radar depth based on monocular depth values ​​and optimizable parameters to obtain images with corrected depth.

[0037] Pre-training module: Used to process the depth-corrected image using 3D Gaussian splashing to obtain the image from the source viewpoint;

[0038] Adaptive Depth Deformation Module: Projecting the image from the source viewpoint yields the 3D coordinates from that viewpoint; then performing adaptive depth deformation based on these 3D coordinates to obtain the 3D coordinates from the pseudo-viewpoint. The operation is as follows:

[0039] In the spatial coordinate system, if the change in the z-axis is 0, then the z-axis value of that point is set as the minimum depth value of the depth image under the source view. Based on the z-axis value of that point, the camera intrinsic parameters, and the preset deformation amount, the displacement deformation of the x-axis and y-axis is controlled.

[0040] If the change in the z-axis is not 0, the change in the z-axis of that point is set to a preset value. Based on the change in the z-axis of that point, the three-dimensional coordinates under the source viewpoint, the camera intrinsic parameters, and the preset deformation, the displacement and deformation of the x-axis and y-axis are controlled.

[0041] Based on the three-dimensional coordinates from the source viewpoint and the corresponding displacement and deformation on the x and y axes, the three-dimensional coordinates from the pseudo viewpoint are obtained.

[0042] The 3D coordinates under the pseudo-viewpoint are projected onto the camera plane to generate an image under the pseudo-viewpoint.

[0043] Hole completion module: The image under the pseudo view is sequentially subjected to forward noise addition and inverse noise reduction to obtain an optimized pseudo view; the optimized pseudo view is used as the image under the source view and enters the adaptive depth deformation module until the preset number of executions is reached;

[0044] Optimization module: used to calculate the loss between the image under the source view and the image after depth correction, as well as the loss between the optimized pseudo-view and its corresponding rendered image. The optimized pseudo-view that meets the preset conditions is processed by 3D Gaussian splashing and used as the vehicle end view.

[0045] Beneficial effects

[0046] This invention uses a small number of real roadside views to generate pseudo-views through radar depth completion, adaptive depth deformation, and hole completion. The optimized pseudo-views are then used as the source view images to continue generating pseudo-views and completing hole completion, thereby obtaining a large number of vehicle-side views with dense viewpoints and high realism. This improves the efficiency of data collection and the reliability of the synthesized dataset, meeting the data requirements of autonomous driving. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating a method for converting a roadside view to a vehicle-side view based on three-dimensional Gaussian splashing according to the present invention. Detailed Implementation

[0048] The following examples are intended to illustrate the present invention, and not to further limit the invention.

[0049] This invention provides a method for converting roadside views to vehicle-side views based on three-dimensional Gaussian splashing, such as... Figure 1 As shown, it includes the following steps:

[0050] S1: Acquire roadside views, obtain monocular disparity maps using monocular depth estimation, calculate monocular depth values, and complete radar depth based on monocular depth values ​​and optimizable parameters to obtain images with corrected depth.

[0051] In the first phase, dense monocular depth data is used to supplement the sparse radar depth data, providing supervision for subsequent training. Specifically:

[0052] First, a monocular disparity map is obtained using monocular depth estimation, and then the monocular depth value is calculated, specifically as follows:

[0053] According to the formula: accomplish;

[0054] In the formula, D mono d is the monocular depth value. mono For monocular disparity maps, b train f train These are the baseline and focal length of the monocular depth estimation model, respectively.

[0055] Then, after obtaining the monocular depth value, its scale needs to be converted to the true scale to complete the radar depth map. This process requires constructing a conversion formula from a monocular disparity map to a depth map. Preferably, based on the monocular depth value and optimizable parameters, the radar depth is completed to obtain the depth-corrected image, specifically as follows:

[0056] According to the formula: Complete the radar depth; where D align To complete the radar depth value, d mono This is a monocular disparity map, where c1, c2, and c3 are optimizable parameters.

[0057] The consistency between the monocular depth value and the completed radar depth value is calculated. If the consistency is not greater than a preset value, the optimizable parameters are optimized using a nonlinear least squares method with Huber loss. The optimized optimizable parameters are then used to complete the radar depth, resulting in an image with corrected depth.

[0058] In addition, the optimization of optimizable parameters is achieved using a nonlinear least squares method with Huber loss, based on the formula:

[0059] accomplish;

[0060] In the formula, Let D be the Huber loss function. lidar For radar depth, For radar depth and monocular parallax map data pairs, (i, j) are pixel coordinates.

[0061] For example, if the consistency between the monocular depth value and the completed radar depth value is no more than 80%, then the optimizable parameters are optimized.

[0062] S2: The depth-corrected image is processed by 3D Gaussian splashing to obtain the image from the source viewpoint.

[0063] In the second stage, a 3D Gaussian splash (3DGS) can be pre-trained from four sparse roadside views to learn the structure and color of the scene.

[0064] The loss between the image from the source viewpoint (the rendered image and depth) and the image after depth correction (the real image and depth) is calculated as follows:

[0065]

[0066] In the formula, The loss is calculated by comparing the image from the source viewpoint with the image after depth correction, where λ is the loss weight, for example, λ = 0.8. I represents the rendered image from the source viewpoint and the image after depth correction, respectively. D represents the rendered image depth from the source view and the image depth after depth correction, respectively. SSIM is the SSIM loss function, used for calculating the structural similarity index.

[0067] S3: Project the image from the source viewpoint to obtain the 3D coordinates from the source viewpoint; perform adaptive depth deformation based on the 3D coordinates from the source viewpoint to obtain the 3D coordinates from the pseudo viewpoint.

[0068] In the third stage, considering that training with only four images would lead to overfitting of 3DGS and degrade rendering results at new viewpoints, a depth warp technique was used to generate pseudo-viewpoints near the training viewpoint. Specifically:

[0069] In the spatial coordinate system, if the change in the z-axis is 0, then the z-axis value of that point is set as the minimum depth value of the depth image under the source view. Based on the z-axis value of that point, the camera intrinsic parameters, and the preset deformation amount, the displacement deformation of the x-axis and y-axis is controlled.

[0070] If the change in the z-axis is not 0, the change in the z-axis of that point is set to a preset value. Based on the change in the z-axis of that point, the three-dimensional coordinates under the source viewpoint, the camera intrinsic parameters, and the preset deformation, the displacement and deformation of the x-axis and y-axis are controlled.

[0071] Based on the three-dimensional coordinates from the source viewpoint and the corresponding displacement and deformation on the x and y axes, the three-dimensional coordinates from the pseudo viewpoint are obtained.

[0072] The 3D coordinates under the pseudo-viewpoint are projected onto the camera plane to generate an image under the pseudo-viewpoint.

[0073] Preferably, in one scenario, the displacement deformation along the x and y axes is controlled based on the z-axis value at that point, camera intrinsic parameters, and a preset deformation amount.

[0074] According to the formula: accomplish;

[0075] In the formula, ε is the preset deformation amount, and f x f y Δt is the focal length in the camera's intrinsic parameters. x , Δt y , respectively, represent the displacement deformation along the x-axis and y-axis, and z is the z-axis value of that point from the source viewpoint.

[0076] One scenario involves controlling the displacement and deformation along the x and y axes based on the change in the z-axis at that point, the 3D coordinates from the source viewpoint, camera intrinsic parameters, and preset deformation values. This is done according to the formula: accomplish;

[0077] In the formula, ε is the preset deformation amount, and f x f y c is the focal length in the camera's intrinsic parameters. x c y The coordinates of the principal point in the camera's intrinsic parameters, Δt x , Δt y , Δt z These represent the displacement deformation along the x-axis, y-axis, and z-axis, respectively, where x and y are the x-axis and y-axis values ​​of that point from the source viewpoint.

[0078] The detailed derivation of the above formula is as follows:

[0079] (1) Project the 3D point p = (x, y, z) rendered from the source view V onto a new view V', with the formula: p' = KR'R -1 K -1 p+K(T'-R'R -1 T);

[0080] Where p is the point cloud obtained from the source viewpoint V through depth map projection, and p', R', and T' are the 3D point, rotation matrix, and translation vector of the new viewpoint V', respectively. K is the camera intrinsic parameter, T is the translation vector of the source viewpoint V, and R is the rotation matrix of the source viewpoint V.

[0081] (2) The 3D points under the new perspective V' can be projected onto the camera plane using the projection formula to generate the image under the new perspective V'. The formula is: w norm The coefficients are those of the normalized homogeneous coordinate system.

[0082] However, directly applying fixed displacement and rotation for depth deformation usually leads to two errors: 1) excessive displacement results in over-deformation, amplifying geometric errors and creating a distorted illusion; 2) insufficient displacement leads to insufficient deformation, limiting the change in viewpoint. To balance this trade-off, adaptive depth deformation is used, which is an adaptive depth deformation strategy constrained by pixel-level spatial consistency.

[0083] (3) For a three-dimensional point observed in the source viewpoint V, let This represents the pixel displacement in the new perspective V'. The formula in (2) uses... Replace with The following displacement relationship was obtained:

[0084]

[0085] in, f represents the displacement distance difference between two viewpoints. x ,f y ,c x ,c y It's the camera's internal parameters.

[0086] (4) Given an ideal deformation ε, ensure that the displacement of all pixels does not exceed ε, so as to achieve the effect of controlling the deformation:

[0087] (5) Combining formulas (3) and (4), we obtain information about ε and Δt. x , Δt y , Δt z Inequalities for x, y, and z:

[0088]

[0089] (6) Solving this inequality requires considering two cases:

[0090] First, Δt z =0, the formula can be simplified to:

[0091]

[0092] Given f x and f y The relative transformation Δt is controlled by setting z to the minimum value in the depth map, which is fixed, predefined, and z set to the minimum value in the depth map. x and Δt y To set an appropriate depth deformation. In actual processing, after controlling the z-axis value, camera intrinsic parameters, and preset deformation amount ε, the displacement deformation amount of the x-axis and y-axis will have a range of values, and usually the maximum value of this range is taken.

[0093] First, Δt z ≠0, which simplifies the formula to:

[0094]

[0095] Since only one change can be controlled in each inequality, Δt can be... z Set to a fixed value to control Δt x and Δty To set the appropriate depth of warp.

[0096] Through the above operations, a suitable displacement T' is found, and a new perspective V' is generated, providing dense training perspectives for 3DGS.

[0097] S4: The image under the pseudo-view is sequentially subjected to forward noise addition and reverse noise reduction to obtain an optimized pseudo-view; the optimized pseudo-view is used as the image under the source view, and S3 is executed until the preset number of executions is reached.

[0098] Considering that occlusion can create many holes during pseudo-view generation, which can affect the acquisition of a wider range of pseudo-views, the fourth stage uses forward noise addition and reverse denoising to fill these holes. To ensure consistency in the content between holes, a cascading method is also used for filling.

[0099] Preferably, the image under the pseudo-viewpoint is subjected to forward noise addition and reverse noise reduction sequentially to obtain an optimized pseudo-viewpoint, as follows:

[0100] S41: Based on the image under the pseudo-viewpoint, we obtain the image and the mask representing the hole. We can use a pre-trained variational autoencoder (VAE) to map the image and the mask to the latent space to obtain the latent space image.

[0101] S42: After adding noise to the latent space image using random noise image, in each time step, the non-masked region and the masked region in the noise image are fused to obtain a new noise image;

[0102] For example, a fixed noise is applied to the latent space image F to obtain a noisy image M1. For the regions in the mask, at each time step, the corresponding noisy image features M2 are generated based on the time step. In each time step, the two noisy images are fused using the mask: M = M1 * mask + (1.0 - mask) * M2. That is, the non-masked regions in the original image and the masked regions in the noise image are fused to obtain a new noise image M. This ensures that only the masked regions are reconstructed, while other regions retain their original information.

[0103] S43: Based on the time step, denoise the new noise map until the denoising is completed, and obtain the optimized pseudo-view.

[0104] S5: Calculate the loss between the image at the source view and the image after depth correction, as well as the loss between the optimized pseudo-view and its corresponding rendered image. The optimized pseudo-view that meets the preset conditions is then processed by 3D Gaussian splashing and used as the vehicle end view.

[0105] While forward noise addition and inverse denoising can effectively complete virtual images, the random generation process may lead to semantic inconsistencies with the actual scene. Therefore, this application employs a confidence-guided optimization method, utilizing multi-view consensus to weight the supervision signal. An L2 norm loss function is used to detect pixel alignment, and SSIM loss is employed to evaluate perceptual similarity at the structural level.

[0106] Preferably, the loss between the optimized pseudo-view and its corresponding rendered image is calculated according to the formula:

[0107]

[0108] accomplish;

[0109] In the formula, W is the confidence weight. To optimize the loss of the pseudo-view and its corresponding rendered image, Let λ be the mathematical expectation, λ1 be the weight of the loss function (which can be set to 0.8), L2 be the L2 norm loss function, and I' be the optimized pseudo-view. The rendered image is obtained using 3D Gaussian splashing with the same camera extrinsing as the pseudoview.

[0110] Therefore, in the process of obtaining the vehicle-side view during 3DGS training, the total loss function is:

[0111] This invention uses a small number of real roadside views to generate pseudo-views through radar depth completion, adaptive depth deformation, and hole completion. The optimized pseudo-views are then used as the source view images to continue generating pseudo-views and completing hole completion, thereby obtaining a large number of vehicle-side views with dense viewpoints and high realism. This improves the efficiency of data collection and the reliability of the synthesized dataset, meeting the data requirements of autonomous driving.

[0112] The present invention also provides a roadside view to vehicle-side view conversion system based on three-dimensional Gaussian splashing, comprising:

[0113] Depth Completion Module: Used to acquire roadside views, obtain monocular disparity maps using monocular depth estimation, calculate monocular depth values, and complete radar depth based on monocular depth values ​​and optimizable parameters to obtain images with corrected depth.

[0114] Pre-training module: Used to process the depth-corrected image using 3D Gaussian splashing to obtain the image from the source viewpoint;

[0115] Adaptive Depth Deformation Module: Projecting the image from the source viewpoint yields the 3D coordinates from that viewpoint; then performing adaptive depth deformation based on these 3D coordinates to obtain the 3D coordinates from the pseudo-viewpoint. The operation is as follows:

[0116] In the spatial coordinate system, if the change in the z-axis is 0, then the z-axis value of that point is set as the minimum depth value of the depth image under the source view. Based on the z-axis value of that point, the camera intrinsic parameters, and the preset deformation amount, the displacement deformation of the x-axis and y-axis is controlled.

[0117] If the change in the z-axis is not 0, the change in the z-axis of that point is set to a preset value. Based on the change in the z-axis of that point, the three-dimensional coordinates under the source viewpoint, the camera intrinsic parameters, and the preset deformation, the displacement and deformation of the x-axis and y-axis are controlled.

[0118] Based on the three-dimensional coordinates from the source viewpoint and the corresponding displacement and deformation on the x and y axes, the three-dimensional coordinates from the pseudo viewpoint are obtained.

[0119] The 3D coordinates under the pseudo-viewpoint are projected onto the camera plane to generate an image under the pseudo-viewpoint.

[0120] Hole completion module: The image under the pseudo view is sequentially subjected to forward noise addition and inverse noise reduction to obtain an optimized pseudo view; the optimized pseudo view is used as the image under the source view and enters the adaptive depth deformation module until the preset number of executions is reached;

[0121] Optimization module: used to calculate the loss between the image under the source view and the image after depth correction, as well as the loss between the optimized pseudo-view and its corresponding rendered image. The optimized pseudo-view that meets the preset conditions is processed by 3D Gaussian splashing and used as the vehicle end view.

Claims

1. A method for converting roadside views to vehicle-side views based on three-dimensional Gaussian splashing, characterized in that, Includes the following steps: S1: Acquire roadside views, obtain monocular disparity maps using monocular depth estimation, calculate monocular depth values, and complete radar depth based on monocular depth values ​​and optimizable parameters to obtain images with corrected depth. The process of completing the radar depth based on the monocular depth value and optimizable parameters to obtain a depth-corrected image is as follows: According to the formula: Complete the radar depth; where, For the completed radar depth value, This is a monocular disparity map. , , These are optimizable parameters; Calculate the consistency between the monocular depth value and the completed radar depth value. If the consistency is not greater than a preset value, optimize the optimizable parameters using a nonlinear least squares method with Huber loss. Use the optimized optimizable parameters to complete the radar depth and obtain the image with corrected depth. S2: The depth-corrected image is processed by 3D Gaussian splashing to obtain the image from the source viewpoint; S3: Project the image from the source viewpoint to obtain the 3D coordinates from the source viewpoint; perform adaptive depth deformation based on the 3D coordinates from the source viewpoint to obtain the 3D coordinates from the pseudo viewpoint, as follows: In the spatial coordinate system, if the change in the z-axis of a point in the three-dimensional coordinates under the source view is 0, then the z-axis value of that point is set as the minimum depth value of the depth image under the source view. Based on the z-axis value of that point, the camera intrinsic parameters, and the preset deformation amount, the displacement deformation of the x-axis and y-axis is controlled. The step of controlling the displacement deformation of the x-axis and y-axis based on the z-axis value of the point, camera intrinsic parameters, and preset deformation amount is based on the formula: ,accomplish; In the formula, To preset the deformation amount, , The focal length is a parameter in the camera's internal settings. , These represent the displacement and deformation along the x-axis and y-axis, respectively. This is the z-axis value of the point from the source viewpoint; If the change in the z-axis of a point in the three-dimensional coordinates under the source view is not 0, then the change in the z-axis of that point is set to a preset value. Based on the change in the z-axis of that point, the three-dimensional coordinates under the source view, the camera intrinsic parameters, and the preset deformation, the displacement and deformation of the x-axis and y-axis are controlled. The process of controlling the displacement and deformation of the x and y axes based on the change in the z-axis at that point, the three-dimensional coordinates from the source viewpoint, camera intrinsic parameters, and preset deformation is based on the formula: , to achieve; where, To preset the deformation amount, , The focal length is a parameter in the camera's internal settings. , The coordinates of the principal point in the camera's intrinsic parameters. , , These represent the displacement deformation along the x-axis, y-axis, and z-axis, respectively. , These are the points as seen from the source viewpoint. , Axis value; Based on the three-dimensional coordinates from the source viewpoint and the corresponding displacement and deformation on the x and y axes, the three-dimensional coordinates from the pseudo viewpoint are obtained. The 3D coordinates under the pseudo-viewpoint are projected onto the camera plane to generate an image under the pseudo-viewpoint. S4: The image under the pseudo-view is subjected to forward noise addition and reverse noise reduction in sequence to obtain an optimized pseudo-view; the optimized pseudo-view is used as the image under the source view, and S3 is executed until the preset number of executions is reached; S5: Calculate the loss between the image at the source view and the image after depth correction, as well as the loss between the optimized pseudo-view and its corresponding rendered image. The optimized pseudo-view that meets the preset conditions is then processed by 3D Gaussian splashing and used as the vehicle end view.

2. The method for converting roadside view to vehicle-side view based on three-dimensional Gaussian splashing according to claim 1, characterized in that, The optimization of the optimizable parameters is performed using a nonlinear least squares method with Huber loss, based on the formula: , ,accomplish; In the formula, Here is the Huber loss function. For radar depth, For radar depth and monocular parallax map data pairs, (i, j) are pixel coordinates.

3. The method for converting roadside view to vehicle-side view based on three-dimensional Gaussian splashing according to claim 1, characterized in that, S5 calculates the loss between the image at the source viewpoint and the image after depth correction, based on the formula: To achieve; In the formula, The loss between the image at the source viewpoint and the image after depth correction. To lose weight, , These are the rendered image from the original viewpoint and the image after depth correction, respectively. , , respectively, represent the rendered image depth from the source viewpoint and the image depth after depth correction, and SSIM(·) is the SSIM loss function.

4. The method for converting roadside view to vehicle-side view based on three-dimensional Gaussian splashing according to claim 1, characterized in that, In step S4, the image under the pseudo-viewpoint is sequentially subjected to forward noise addition and inverse noise reduction to obtain an optimized pseudo-viewpoint, as follows: S41: Based on the image under the pseudo-view, obtain the image and the mask representing the hole, and map the image and the mask to the latent space to obtain the latent space image; S42: After adding noise to the latent space image using random noise image, in each time step, the non-masked region and the masked region in the noise image are fused to obtain a new noise image; S43: Based on the time step, denoise the new noise map until the denoising is completed, and obtain the optimized pseudo-view.

5. The method for converting roadside view to vehicle-side view based on three-dimensional Gaussian splashing according to claim 1, characterized in that, S1 involves obtaining a monocular disparity map using monocular depth estimation and calculating the monocular depth value, specifically as follows: According to the formula: ,accomplish; In the formula, This is the monocular depth value. This is a monocular disparity map. , These are the baseline and focal length of the monocular depth estimation model, respectively.

6. A roadside view to vehicle-side view conversion system based on three-dimensional Gaussian splashing, characterized in that, include: Depth Completion Module: Used to acquire roadside views, obtain monocular disparity maps using monocular depth estimation, calculate monocular depth values, and complete radar depth based on monocular depth values ​​and optimizable parameters to obtain images with corrected depth. The process of completing the radar depth based on the monocular depth value and optimizable parameters to obtain a depth-corrected image is as follows: According to the formula: Complete the radar depth; where, To complete the radar depth value, This is a monocular disparity map. , , These are optimizable parameters; Calculate the consistency between the monocular depth value and the completed radar depth value. If the consistency is not greater than a preset value, optimize the optimizable parameters using a nonlinear least squares method with Huber loss. Use the optimized optimizable parameters to complete the radar depth and obtain the image with corrected depth. Pre-training module: Used to process the depth-corrected image using 3D Gaussian splashing to obtain the image from the source viewpoint; Adaptive Depth Deformation Module: Projecting the image from the source viewpoint yields the 3D coordinates from that viewpoint; then performing adaptive depth deformation based on these 3D coordinates to obtain the 3D coordinates from the pseudo-viewpoint. The operation is as follows: In the spatial coordinate system, if the change in the z-axis of a point in the three-dimensional coordinates under the source view is 0, then the z-axis value of that point is set as the minimum depth value of the depth image under the source view. Based on the z-axis value of that point, the camera intrinsic parameters, and the preset deformation amount, the displacement deformation of the x-axis and y-axis is controlled. The step of controlling the displacement deformation of the x-axis and y-axis based on the z-axis value of the point, camera intrinsic parameters, and preset deformation amount is based on the formula: ,accomplish; In the formula, To preset the deformation amount, , The focal length is a parameter in the camera's internal settings. , These represent the displacement and deformation along the x-axis and y-axis, respectively. This is the z-axis value of the point from the source viewpoint; If the change in the z-axis of a point in the three-dimensional coordinates under the source view is not 0, then the change in the z-axis of that point is set to a preset value. Based on the change in the z-axis of that point, the three-dimensional coordinates under the source view, the camera intrinsic parameters, and the preset deformation, the displacement and deformation of the x-axis and y-axis are controlled. The process of controlling the displacement and deformation of the x and y axes based on the change in the z-axis at that point, the three-dimensional coordinates from the source viewpoint, camera intrinsic parameters, and preset deformation is based on the formula: , to achieve; where, To preset the deformation amount, , The focal length is a parameter in the camera's internal settings. , The coordinates of the principal point in the camera's intrinsic parameters. , , These represent the displacement deformation along the x-axis, y-axis, and z-axis, respectively. , These are the points as seen from the source viewpoint. , Axis value; Based on the three-dimensional coordinates from the source viewpoint and the corresponding displacement and deformation on the x and y axes, the three-dimensional coordinates from the pseudo viewpoint are obtained. The 3D coordinates under the pseudo-viewpoint are projected onto the camera plane to generate an image under the pseudo-viewpoint. Hole completion module: The image under the pseudo view is sequentially subjected to forward noise addition and reverse noise reduction to obtain an optimized pseudo view; The optimized pseudo-view is used as the image from the source viewpoint and entered into the adaptive depth deformation module until the preset number of executions is reached; Optimization module: used to calculate the loss between the image under the source view and the image after depth correction, as well as the loss between the optimized pseudo-view and its corresponding rendered image. The optimized pseudo-view that meets the preset conditions is processed by 3D Gaussian splashing and used as the vehicle end view.

Images