Vehicle-mounted fisheye imaging view angle reconstruction method, system and electronic device

Abstract

The application provides a vehicle-mounted fisheye imaging view angle reconstruction method and system and an electronic device, relates to the technical field of artificial intelligence and automatic driving, and comprises the following steps: acquiring an original fisheye image of a vehicle-mounted fisheye camera and corresponding camera parameters; based on the camera parameters, equivalently splitting the original fisheye image into a plurality of virtual pinhole camera images; and performing Gaussian splash three-dimensional scene reconstruction on the plurality of virtual pinhole camera images to generate a composite image of a target view angle. The application solves the technical problem of mismatch between fisheye distortion and Gaussian splash, and significantly improves the view angle reconstruction quality and application flexibility of the vehicle-mounted fisheye image.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and autonomous driving technology, and in particular to a method, system and electronic device for reconstructing the field of view of an in-vehicle fisheye imaging system. Background Technology

[0002] Automotive fisheye cameras are widely used in surround-view systems and parking assistance for intelligent driving due to their large field of view and low cost. However, fisheye lenses suffer from significant nonlinear distortion, resulting in a large deviation between the image geometry and the real scene, which poses difficulties for subsequent perception and 3D reconstruction. Gaussian splashing, as an emerging 3D scene representation method, performs well in synthesizing new perspectives, but its current implementation is based on a conventional pinhole imaging model, which cannot directly adapt to the extreme distortion characteristics of fisheye images, making it difficult to recover accurate geometric information and generate high-quality new perspectives from automotive fisheye images. Therefore, how to effectively integrate fisheye imaging models and Gaussian splashing technology to achieve original perspective reconstruction and arbitrary new perspective synthesis from fisheye images is an urgent problem to be solved. Summary of the Invention

[0003] The purpose of this application is to provide a method, system, and electronic device for reconstructing the field of view of a vehicle-mounted fisheye imaging system, so as to alleviate the above-mentioned technical problems existing in the prior art.

[0004] In a first aspect, the present invention provides a method for reconstructing the field of view of a vehicle-mounted fisheye imaging system, comprising: Acquire the raw fisheye image and corresponding camera parameters from the vehicle-mounted fisheye camera; Based on the camera parameters, the original fisheye image is equivalently split into multiple virtual pinhole camera images; Gaussian splashing 3D scene reconstruction is performed on the multiple virtual pinhole camera images to generate a synthetic image from the target perspective.

[0005] In an optional implementation, acquiring the raw fisheye image captured by the vehicle-mounted fisheye camera and the corresponding camera parameters includes: The vehicle-mounted fisheye camera is calibrated relative to the vehicle body extrinsic parameters and the radar camera extrinsic parameters; Acquire point cloud data of the vehicle body collected by the radar camera; Align the vehicle body point cloud data with the original fisheye image, and register the vehicle body point cloud data to the image coordinate system of the original fisheye image; Motion distortion elimination processing is performed on the vehicle body point cloud data using vehicle pose information.

[0006] In an optional implementation, the camera parameters include the extrinsic parameter matrix of the vehicle-mounted fisheye camera; based on the camera parameters, the original fisheye image is equivalently split into multiple virtual pinhole camera images, including: Based on the position of the vehicle-mounted fisheye camera, multiple virtual pinhole cameras facing different directions are constructed according to a preset combination of yaw and pitch angles. Based on the size of the original fisheye image and the extrinsic parameter matrix of the vehicle-mounted fisheye camera, calculate the intrinsic parameter matrix and extrinsic parameter matrix of each virtual pinhole camera; The pixels of the original fisheye image are resampled to the viewpoints of each virtual pinhole camera by polar coordinate mapping to generate virtual pinhole camera images.

[0007] In an optional implementation, based on the size of the original fisheye image and the extrinsic parameter matrix of the vehicle-mounted fisheye camera, the intrinsic and extrinsic parameter matrices of each virtual pinhole camera are calculated, including: The intrinsic parameter matrix of the virtual pinhole camera is calculated based on the preset field of view and the size of the original fisheye image. Calculate the rotation offset matrix of the virtual pinhole camera relative to the fisheye camera based on the preset yaw and pitch angles. Based on the rotation offset matrix and the extrinsic parameter matrix of the vehicle-mounted fisheye camera, the extrinsic parameter matrix of the virtual pinhole camera is calculated.

[0008] In an optional implementation, Gaussian splash 3D scene reconstruction is performed on the multiple virtual pinhole camera images to generate a synthetic image from the target perspective, including: Gaussian splash 3D scene reconstruction is performed based on the multiple virtual pinhole camera images to obtain the Gaussian primitive parameters in the 3D scene. Rendering is performed based on the Gaussian meta-parameters to generate a synthetic image from the target viewpoint.

[0009] In an optional implementation, Gaussian splash 3D scene reconstruction is performed based on the multiple virtual pinhole camera images to obtain Gaussian meta-parameters of the 3D scene, including: An initial 3D scene is generated based on the multiple virtual pinhole camera images and 3D point cloud data; the initial 3D scene includes multiple 3D sample points. Three-dimensional Gaussian elements are constructed based on the three-dimensional sample points, wherein the initial parameters corresponding to the three-dimensional Gaussian elements include the spatial position of each Gaussian element, the anisotropic covariance parameter, and the appearance attribute parameter. Re-rendered images from the perspective of each virtual pinhole camera are generated based on the initial parameters; The initial parameters are updated based on the pixel error between the re-rendered image and the corresponding virtual pinhole camera image to obtain the various Gaussian primitive parameters of the 3D scene.

[0010] In an optional implementation, rendering is performed based on the Gaussian meta-parameters to generate a synthetic image from the target viewpoint, including: Obtain the camera pose from the target's perspective; Based on the camera pose of the target viewpoint, each Gaussian element in the 3D scene is projected onto the target imaging plane to obtain the projection area of ​​each Gaussian element on the target imaging plane and its corresponding color and transparency contribution. The projection regions of each 3D Gaussian element are sorted according to their depth values. Based on the sorting results, the color and transparency contributions of each Gaussian unit are accumulated and fused pixel by pixel to generate a synthetic image from the target viewpoint.

[0011] In an optional implementation, when the target viewpoint is the original fisheye camera viewpoint, the step of reconstructing a 3D scene from the plurality of virtual pinhole camera images using Gaussian splashing to generate a synthetic image from the target viewpoint includes: Calculate the ray direction vector of each pixel in the original fisheye camera view; The ray direction vector is transformed to the coordinate system of each virtual pinhole camera and projected onto the corresponding virtual pinhole camera imaging plane to obtain the sampling coordinates in the rendered image of each virtual pinhole camera; The rendered images of each virtual pinhole camera are resampled according to the sampling coordinates, and the pixel values ​​obtained by resampling are weighted and fused to obtain a synthetic image from the original fisheye camera view.

[0012] Secondly, the present invention provides a system for reconstructing the field of view of a vehicle-mounted fisheye imaging system, comprising: The image acquisition module is used to acquire the raw fisheye image and camera parameters of the vehicle-mounted fisheye camera; The splitting module is used to split the original fisheye image into multiple virtual pinhole camera images based on the camera parameters. The perspective reconstruction module is used to reconstruct a three-dimensional scene from the multiple virtual pinhole camera images using Gaussian splashing, generating a synthetic image from the target perspective.

[0013] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method described in any of the foregoing embodiments.

[0014] The method, system, and electronic device for reconstructing the viewpoint of vehicle-mounted fisheye imaging provided in this application acquire fisheye images and their camera parameters, providing an accurate distortion model and geometric benchmark for subsequent processing. Secondly, the original fisheye image is equivalently decomposed into multiple virtual pinhole camera images, effectively overcoming the inherent nonlinear distortion of fisheye lenses and converting extremely distorted fisheye imaging into multiple locally linearized pinhole images, allowing existing Gaussian splashing methods to be directly adapted. Finally, Gaussian splashing 3D reconstruction is performed on multiple virtual pinhole images to generate a composite image of the target viewpoint, restoring the geometric consistency of the original viewpoint and supporting real-time rendering of any new viewpoint. In summary, this application solves the technical problem of mismatch between fisheye distortion and Gaussian splashing, significantly improving the viewpoint reconstruction quality and application flexibility of vehicle-mounted fisheye images. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the specific embodiments of this application or the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating a method for reconstructing the field of view of a vehicle-mounted fisheye imaging system, as provided in an embodiment of this application; Figure 2 A structural diagram of a vehicle-mounted fisheye imaging view reconstruction system provided in an embodiment of this application; Figure 3 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0018] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0019] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0020] This application provides a method for reconstructing the field of view of a vehicle-mounted fisheye imaging system. See [link to relevant documentation]. Figure 1 As shown, the method mainly includes the following steps: S110 acquires the raw fisheye image and camera parameters from the vehicle-mounted fisheye camera.

[0021] Specifically, the vehicle-mounted fisheye camera is installed below the front and rear bumpers or exterior rearview mirrors of the vehicle, with a field of view of up to 180° or even larger. In this step, the original fisheye image captured by the fisheye camera, along with pre-calibrated camera parameters, are first acquired. The camera parameters include the intrinsic parameter matrix, distortion model, and extrinsic parameter matrix of the fisheye camera. The intrinsic parameter matrix characterizes the focal length and optical center, the distortion model describes the radial and tangential distortions unique to fisheye lenses, and the extrinsic parameter matrix characterizes the rotation and translation relationship of the fisheye camera relative to the vehicle coordinate system. It should be noted that the camera parameters can be obtained through offline calibration; this embodiment directly uses the calibration results, and the calibration process will not be described in detail here.

[0022] S120, based on the camera parameters, the original fisheye image is equivalently split into multiple virtual pinhole camera images.

[0023] Because fisheye images suffer from severe nonlinear distortion, directly using them for 3D reconstruction can lead to geometric inaccuracies. Therefore, this application constructs multiple virtual pinhole cameras with different orientations based on the intrinsic parameter matrix, distortion model, and extrinsic parameter matrix of the fisheye camera, using the fisheye camera's position as a reference and a preset combination of yaw and pitch angles. Each virtual pinhole camera shares the spatial position of the fisheye camera but has a different orientation, effectively segmenting the fisheye camera's field of view into several locally linearized sub-regions. Then, the pixels of the original fisheye image are resampled onto the imaging plane of each virtual pinhole camera through polar coordinate mapping, generating multiple virtual pinhole camera images with no distortion or very little distortion. In this way, a single fisheye image is transformed into images from multiple conventional pinhole camera perspectives, facilitating subsequent processing.

[0024] S130, Gaussian splash 3D scene reconstruction is performed on the multiple virtual pinhole camera images to generate a synthetic image from the target perspective.

[0025] Gaussian splashing is a scene representation method based on 3D Gaussian primitives. This step first uses multiple virtual pinhole camera images obtained in the previous step as input, and combines them with 3D point cloud data to generate 3D sample points. Based on these 3D sample points, several 3D Gaussian primitives are constructed. Each Gaussian primitive includes spatial location, appearance attributes, and anisotropic covariance parameters, which together constitute the Gaussian primitive parameters of the 3D scene. Then, using differentiable rendering technology, these Gaussian primitive parameters are rendered back to the viewpoint of each virtual pinhole camera to obtain a re-rendered image, and the pixel error between the re-rendered image and the original virtual pinhole camera image is calculated. The gradient is calculated based on this error, and the Gaussian primitive parameters are iteratively updated along the gradient descent direction until convergence, resulting in the optimized 3D scene Gaussian primitive parameters.

[0026] Building upon this foundation, when generating a synthetic image from a target perspective (such as a virtual bird's-eye view, a specific side view, or a restored fisheye camera view), the camera pose (including position and orientation) of that target perspective is first obtained. Then, each Gaussian primitive in the 3D scene is projected onto the target imaging plane, obtaining the projection area of ​​each Gaussian primitive and its color contribution. To correctly handle occlusion, the projection areas are sorted according to the depth value corresponding to each Gaussian primitive, with the color of Gaussian primitives with closer depths covering those with greater depths. Finally, the color contributions of all Gaussian primitives are fused to generate a synthetic image from that target perspective. This synthetic image possesses high-fidelity geometric structure and texture details.

[0027] This application effectively overcomes the adverse effects of fisheye distortion on 3D reconstruction by splitting fisheye images into multiple pinhole views and then using Gaussian splashing for 3D reconstruction. It can reconstruct high-quality 3D scenes from single or multiple fisheye images and supports the synthesis of arbitrary new viewpoints. Compared to directly reconstructing 3D from fisheye images, this method significantly improves geometric consistency and texture realism, providing more reliable visual information for applications such as surround-view perception and parking assistance in autonomous driving.

[0028] For ease of understanding, the method provided in this application will be described in detail below.

[0029] In acquiring the raw fisheye image and camera parameters, this embodiment also introduces multi-sensor data preprocessing. Specifically, in addition to acquiring the fisheye image, it also acquires the vehicle body point cloud data collected by the radar camera, and calibrates the extrinsic parameters of the vehicle-mounted fisheye camera relative to the vehicle body and the extrinsic parameters of the radar camera relative to the vehicle body.

[0030] The calibration process can be performed offline: the vehicle is placed in the calibration area, and using a calibration board of known size or natural feature points, multiple sets of fisheye images and LiDAR point clouds are collected to establish the transformation relationship between the fisheye camera coordinate system, the radar coordinate system, and the vehicle body coordinate system. To improve calibration accuracy, this embodiment introduces a joint optimization strategy, which constructs a joint cost function from the reprojection error of the fisheye camera and the registration error of the radar point cloud. The nonlinear least squares method (such as the Levenberg-Marquardt algorithm) is used to simultaneously optimize the camera intrinsic parameters, distortion coefficients, and extrinsic parameters among all sensors, thereby obtaining denoised vehicle body point cloud data (i.e., point cloud with outliers and motion distortion removed).

[0031] Because the sampling frequencies of fisheye cameras and LiDAR are different and there is a time difference, it is necessary to perform motion compensation alignment between the image frames corresponding to the original fisheye image and the point cloud frames corresponding to the LiDAR point cloud. One feasible alignment method is to perform linear interpolation compensation on the point cloud coordinates based on the vehicle speed and the time difference between the image and the point cloud, so that the point cloud and the image are strictly corresponding in time.

[0032] Furthermore, inconsistent sampling frequencies of different sensors can lead to inaccurate vehicle pose. Therefore, interpolation calculations are needed to obtain the vehicle pose at the moment the fisheye image was acquired. Based on this vehicle pose, the aligned vehicle point cloud data is registered to the image coordinate system of the original fisheye image, which means transforming the point cloud data from the radar coordinate system to the image coordinate system.

[0033] To eliminate the impact of motion distortion on point clouds, this embodiment utilizes vehicle pose information to perform distortion correction processing on the vehicle body point cloud data. For LiDAR point clouds, during vehicle movement, points acquired at different times within a single frame of the point cloud will experience spatial misalignment due to vehicle motion. Therefore, based on the timestamp of each point and the vehicle pose trajectory, all points in the point cloud are compensated to the same reference time (e.g., the image acquisition time) to obtain a distortion-free point cloud. After the above processing, the point cloud and the image achieve high-precision alignment in both time and space.

[0034] The above approach significantly reduces extrinsic errors and motion distortions between sensors through joint calibration optimization and multi-sensor temporal-space alignment, providing high-precision and highly consistent input data for subsequent fisheye camera splitting and 3D reconstruction, thereby improving the reconstruction accuracy and robustness of the entire system.

[0035] In the process of equivalently decomposing the original fisheye image into multiple virtual pinhole camera images, the camera parameters specifically include the intrinsic parameter matrix, distortion model, and extrinsic parameter matrix of the fisheye camera. Let the intrinsic parameter matrix of the fisheye camera be... ,in , These are the focal lengths of the image in the x and y directions, respectively (in pixels). The coordinates are those of the optical center. The distortion model uses the radial distortion coefficient. and tangential distortion coefficient , represented as The extrinsic parameter matrix is ,in It is a 3×3 rotation matrix. It is a 3×1 translation vector.

[0036] Based on the position of the fisheye camera, according to the preset yaw angle and pitch angle Multiple virtual pinhole cameras facing different directions can be constructed by combining them. The number of virtual pinhole cameras can be determined in the following way.

[0037] In one implementation, the number of virtual pinhole cameras can be adaptively selected based on the distortion distribution of the fisheye image. Specifically, the degree of distortion of a fisheye lens varies significantly across different regions of the image: the distortion is small in the central region and large in the peripheral regions. Accordingly, the number of virtual pinhole cameras can be adaptively determined based on the distortion rate of different regions in the fisheye image. Specifically, the distortion mapping gradient of each pixel position in the fisheye image is pre-calculated, and regions with distortion gradients exceeding a preset threshold (e.g., gradient magnitude greater than 0.1) are divided into several sub-regions, each corresponding to one virtual pinhole camera. When the vehicle is traveling on a curve or when it is necessary to focus on observing the lateral blind spot, the lateral distortion gradient is large, so the number of lateral virtual pinhole cameras is increased (e.g., the yaw angle range is further subdivided into 6 angles); when the vehicle is traveling in a straight line and mainly observing the front region, the front distortion gradient is small, so the number of front virtual pinhole cameras can be reduced (e.g., only one pinhole camera in the center is retained). In this way, computational resource consumption is dynamically adjusted while ensuring reconstruction accuracy.

[0038] In another implementation, the number of virtual pinhole cameras can be adaptively adjusted based on the scene's depth complexity. The accuracy of 3D reconstruction is affected by the degree of drastic changes in scene depth. When there are many close-range objects in the scene (such as pillars or pedestrians in a parking lot) or frequent depth jumps (such as roadside trees), more virtual pinhole cameras are needed to provide richer viewpoint information to constrain the optimization of Gaussian meta-parameters. Conversely, when the scene is an open road with gentle depth changes, fewer pinhole cameras are sufficient for reconstruction. Therefore, this scheme uses the depth standard deviation of the LiDAR point cloud as a criterion: calculating the standard deviation of the depth values ​​of each point in the current frame's point cloud. ,like If the distance exceeds a threshold (e.g., 2 meters), it is considered a high-complexity scene. The number of virtual pinhole cameras is increased to 6 or 8, and the angle intervals are refined in the yaw and pitch directions. If the number is less than the threshold, the number is maintained at 4 or reduced to 2. This method significantly reduces computational overhead while ensuring reconstruction quality.

[0039] In another alternative implementation, the selection can be based on the demand-driven nature of the target synthetic viewpoint. In practical applications, users or autonomous driving systems may only need to synthesize images from a specific viewpoint (e.g., only a bird's-eye view directly in front of the vehicle, or only the view from the left rearview mirror). In this case, the virtual pinhole cameras can no longer be uniformly distributed omnidirectionally, but rather determined inversely based on the field of view of the target viewpoint. Specifically, first, the camera pose and field of view of the target synthetic viewpoint are determined, and the fisheye image area covered by this field of view is calculated; then, virtual pinhole cameras are arranged within this area according to the principle of minimizing distortion, ensuring that their orientation covers the required area of ​​the target viewpoint, without needing to place cameras in other irrelevant areas. For example, if it is necessary to generate a synthetic image of the right blind spot when the vehicle turns right, the virtual pinhole cameras are concentrated in the positive yaw angle (e.g., φ=30°~60°) and the small pitch angle range, with the number set to 3. This method is highly targeted and has the highest computational efficiency.

[0040] In one specific example, each fisheye camera constructs four virtual pinhole cameras, with yaw angles... The values ​​are -47.5°, -47.5°, 47.5°, and 47.5°, respectively, for the pitch angle. The values ​​are -43.5°, 43.5°, -43.5°, and 43.5°. These angle combinations allow the fields of view of the four virtual pinhole cameras to effectively cover most of the original fisheye image, with overlapping areas between adjacent cameras.

[0041] When constructing each virtual pinhole camera, its intrinsic and extrinsic parameter matrices need to be calculated separately. Let the width of the original fisheye image be... Height is (Unit: pixels), the preset field of view of the virtual pinhole camera is... (For example, 70°). Then the focal length of the virtual pinhole camera is... The optical center is taken from the image center. , This yields the intrinsic parameter matrix of the virtual pinhole camera. , where the subscript i represents the i-th virtual pinhole camera.

[0042] The calculation of the extrinsic parameter matrix involves the yaw angle. and pitch angle First, according to and Calculate the rotation offset matrix ,in , Then, the extrinsic rotation matrix of the fisheye camera is... and Multiplying these matrices yields the extrinsic rotation matrix of the virtual pinhole camera. The translation vector is the same as that of the fisheye camera, i.e. Therefore, the extrinsic parameter matrix of the virtual pinhole camera is: .

[0043] In terms of pixel mapping, the pixels of the original fisheye image are resampled to the viewpoints of each virtual pinhole camera using polar coordinate mapping. For each pixel coordinate on the virtual pinhole camera... First, convert it to a normalized ray direction:

[0044] And normalized to a unit vector Then through the rotation matrix Transform the ray direction to the fisheye camera coordinate system to obtain... .remember .

[0045] Calculate the polar coordinate radius based on the isometric projection model of the fisheye camera. ,in The average focal length and angle of incidence of the fisheye camera. azimuth Then, the pixel coordinates in the fisheye image are obtained:

[0046] Finally, pixel values ​​are sampled from the original fisheye image using bilinear interpolation, and a mask is used. Remove invalid pixels (i.e., points that are outside the image boundary or whose ray direction is backward) to generate a virtual pinhole camera image.

[0047] The above method effectively overcomes the extreme distortion problem of fisheye images by splitting a single fisheye image into multiple virtual pinhole camera images with different orientations and local linearization. This allows subsequent 3D reconstruction to directly use the mature pinhole camera Gaussian splashing algorithm without making significant modifications to the existing algorithm, while retaining the large field of view advantage of fisheye cameras.

[0048] Furthermore, Gaussian splashing 3D scene reconstruction is performed on multiple virtual pinhole camera images to obtain the Gaussian primitive parameters in the 3D scene. In specific implementation, an initial 3D scene is first generated based on multiple virtual pinhole camera images and 3D point cloud data. This initial 3D field corresponds to multiple 3D sample points. In this embodiment, preferably, the 3D sample points are directly provided by LiDAR point cloud data. The point cloud data can be pre-processed with motion distortion compensation and spatiotemporal alignment to serve as the initial geometric representation of the 3D scene. For areas with missing point clouds, a multi-view stereo matching method can be used to estimate the depth. Through the above operations, multiple 3D sample points are obtained.

[0049] Three-dimensional Gaussian elements are constructed based on three-dimensional sample points. Each Gaussian element represents a local region in three-dimensional space and is described by a set of geometric and appearance attribute parameters. The initial parameters corresponding to the three-dimensional Gaussian elements include spatial position. (Mean vector), appearance attribute parameters (including opacity parameter) ,color (This embodiment uses RGB values) and anisotropic covariance matrix (Symmetric positive definite matrix). The initial covariance is set as an isotropic matrix. ,in Take a smaller value (e.g., 0.01 meters), and initialize the color to the color value of the corresponding pixel. These parameters together constitute the initial parameters corresponding to each Gaussian unit in the 3D scene.

[0050] Then, re-rendered images from each virtual pinhole camera's viewpoint are generated based on the initial parameters. Specifically, the initial Gaussian primitive parameters (i.e., the current Gaussian primitive parameters) are input into the differentiable renderer. For each virtual pinhole camera viewpoint, the rendering process is as follows: Each 3D Gaussian primitive... Projecting onto the image plane yields two-dimensional Gaussian elements. ,in These are the projected two-dimensional coordinates. Let be the projected covariance matrix. For each pixel, accumulate the color contributions of all 2D Gaussian elements covering that pixel, and perform alpha blending by depth sorting. The pixel color calculation formula is: ,in The opacity of the j-th Gaussian element (related to the value of the 2D Gaussian element at that pixel). The number of Gaussian pixels covering this pixel.

[0051] Calculate the pixel error between the re-rendered image and the original virtual pinhole camera image, for example, using L1 loss. The weighted sum with the SSIM loss. Then, the parameters of each Gaussian element are calculated by automatic differentiation. The Adam optimizer iteratively updates the parameters along the gradient descent direction relative to the error gradient. After each iteration, the image is re-rendered and the error is recalculated until the error converges or the preset number of iterations is reached.

[0052] The Gaussian meta-parameters of the 3D scene obtained through differentiable rendering and gradient descent iterative optimization can accurately represent the geometric structure and texture information of the scene. Furthermore, since the input is multiple distortion-free pinhole images, the optimization process avoids the geometric inconsistency problem caused by fisheye distortion, significantly improving the accuracy and convergence speed of 3D reconstruction.

[0053] Based on the Gaussian meta-parameters in the aforementioned 3D scene, a composite image from the target's perspective is generated through rendering. First, the camera pose from the target's perspective is obtained, including the camera's position in the world coordinate system. and orientation (rotation matrix) The pose can be a pre-set fixed viewpoint (such as a bird's-eye view from the roof or a 45° view from the left front), or it can be any viewpoint dynamically determined based on user input.

[0054] Then, each 3D Gaussian primitive in the 3D scene is projected onto the target imaging plane. The projection process consists of two steps: 1. Transform the Gaussian elements from the world coordinate system to the target camera coordinate system. Let the center of the Gaussian elements be... The covariance is After transformation, center covariance .

[0055] 2. Based on the pinhole camera intrinsic parameter matrix The Gaussian elements are projected onto the image plane. The center of the projected two-dimensional image is... Two-dimensional covariance matrix Where J is the projection function The Jacobian matrix at that location.

[0056] Each Gaussian unit contributes an elliptical region to the image, and its color contribution is determined by the color vector. Multiplied by two-dimensional Gaussian weights get.

[0057] To correctly handle occlusion relationships, the depth value of each 3D Gaussian element's center point in the camera coordinate system is used. Sort the images from smallest to largest, i.e., from farthest to nearest, and perform alpha blending. The blending formula is the same as the rendering formula in step three. Finally, after blending all projection areas, a composite image from the target's viewpoint is obtained.

[0058] The above method can generate high-fidelity synthetic images from any target viewpoint in real time. Since the 3D scene representation itself has continuity and integrity, there is no fisheye distortion in the synthetic image, which can be directly used for downstream tasks such as perception and planning in autonomous driving.

[0059] When the target viewpoint is the original fisheye camera viewpoint, the synthesized image needs to be converted back to a fisheye image to ensure compatibility with existing surround view display systems. The specific steps are as follows: First, calculate the ray direction vector for each pixel from the original fisheye camera's viewpoint. This is based on the fisheye camera's intrinsic parameter matrix. Backprojection is performed using an equidistant projection model. For each pixel coordinate in the original fisheye image... Normalization yields Calculate radial distance According to the isometric projection model, the incident angle (Note: Here) (This is the normalized radial distance, corresponding to the incident angle in radians), azimuth angle. The ray direction vector in the fisheye camera coordinate system is:

[0060] Next, the ray direction vector is transformed to the coordinate system of each virtual pinhole camera. For each virtual pinhole camera... (i=1~4), its extrinsic rotation matrix is The fisheye camera extrinsic rotation matrix is Then the rotation matrix The ray direction in the pinhole camera coordinate system is... Only when When the ray is directed toward the front of the camera, the pinhole camera is valid for that pixel.

[0061] Then, the effective ray direction is projected onto the corresponding virtual pinhole camera imaging plane. The intrinsic parameter matrix of this pinhole camera is... Calculate the sampling coordinates:

[0062] Check if the sampling coordinates meet the requirements. and (in (where the width and height are the dimensions of the pinhole camera image), and if satisfied, then from the rendered image of that pinhole camera... The pixel values ​​are obtained through bilinear interpolation sampling:

[0063] Since a single original fisheye pixel may correspond to multiple virtual pinhole cameras (due to overlapping camera fields of view), it is necessary to weight and fuse the multiple sampling results. The weights are set using angle weights: based on the angle between the ray direction and the optical axis of the pinhole camera. Assign weights, with larger angles resulting in larger weights. Specifically, the weights... ,in , This represents the optical axis direction of the pinhole camera. Finally, the sampled pixel values ​​from all valid pinhole cameras are weighted and summed:

[0064] Obtain the synthetic image from the original fisheye camera viewpoint.

[0065] This process achieves precise reverse mapping from pinhole rendering results to fisheye images. The generated fisheye images fully preserve the original fisheye camera's perspective and distortion characteristics, and can be directly used in existing vehicle surround view systems without modifying downstream display or sensing modules, demonstrating excellent compatibility and practicality.

[0066] In summary, this application solves the problem of mismatch between fisheye distortion and existing 3D reconstruction methods by splitting fisheye images into multiple pinhole views, enabling Gaussian splashing technology to be directly used for fisheye imaging. High-precision 3D Gaussian parameters are obtained through differentiable rendering and gradient iterative optimization, supporting real-time synthesis from any viewpoint. It can output distortion-free images for use by the perception module and also restore the original fisheye image for compatibility with existing surround-view systems. Combined with multi-sensor preprocessing, robustness in real-world road scenarios is ensured, significantly improving the overall utilization efficiency and 3D reconstruction quality of vehicle-mounted fisheye images.

[0067] Based on the above method embodiments, this application also provides a system for reconstructing the field of view of an in-vehicle fisheye imaging system, see [link to relevant documentation]. Figure 2 As shown, the system includes the following components: Image acquisition module 210 is used to acquire the original fisheye image and camera parameters of the vehicle-mounted fisheye camera; The splitting module 220 is used to split the original fisheye image into multiple virtual pinhole camera images based on the camera parameters. The perspective reconstruction module 230 is used to reconstruct a three-dimensional scene from the multiple virtual pinhole camera images using Gaussian splashing, and generate a synthetic image from the target perspective.

[0068] In one feasible implementation, the image acquisition module 210 is specifically used for: The vehicle-mounted fisheye camera is calibrated relative to the vehicle body extrinsic parameters and the radar camera extrinsic parameters to obtain denoised vehicle body point cloud data; Align the vehicle body point cloud data with the original fisheye image, and register the vehicle body point cloud data to the image coordinate system of the original fisheye image; Motion distortion elimination processing is performed on the vehicle body point cloud data using vehicle pose information.

[0069] In one feasible implementation, the camera parameters include the extrinsic parameter matrix of the vehicle-mounted fisheye camera; the aforementioned splitting module 220 is specifically used for: Based on the position of the fisheye camera, multiple virtual pinhole cameras facing different directions are constructed according to a preset combination of yaw and pitch angles; Based on the size of the original fisheye image and the extrinsic parameter matrix of the vehicle-mounted fisheye camera, calculate the intrinsic parameter matrix and extrinsic parameter matrix of each virtual pinhole camera; The pixels of the original fisheye image are resampled to the viewpoints of each virtual pinhole camera by polar coordinate mapping to generate virtual pinhole camera images.

[0070] In one feasible implementation, the aforementioned splitting module 220 is further configured to: The intrinsic parameter matrix of the virtual pinhole camera is calculated based on the preset field of view and the size of the original fisheye image. Calculate the rotation offset matrix of the virtual pinhole camera relative to the fisheye camera based on the preset yaw and pitch angles. Based on the rotation offset matrix and the extrinsic parameter matrix of the vehicle-mounted fisheye camera, the extrinsic parameter matrix of the virtual pinhole camera is calculated.

[0071] In one feasible implementation, the aforementioned perspective reconstruction module 230 is specifically used for: Gaussian splash 3D scene reconstruction is performed based on the multiple virtual pinhole camera images to obtain the Gaussian primitive parameters of the 3D scene; Rendering is performed based on the Gaussian meta-parameters to generate a synthetic image from the target viewpoint.

[0072] In one feasible implementation, the aforementioned perspective reconstruction module 230 is further used for: An initial 3D scene is generated based on the multiple virtual pinhole camera images and 3D point cloud data; the initial 3D scene includes multiple 3D sample points. Based on the three-dimensional sample points, a number of three-dimensional Gaussian elements are constructed, wherein the initial parameters corresponding to the three-dimensional Gaussian elements include the spatial position of each Gaussian element, the anisotropic covariance parameter, and the appearance attribute parameter. Re-rendered images from the perspective of each virtual pinhole camera are generated based on the initial parameters; The pixel error between the re-rendered image and the corresponding virtual pinhole camera image is calculated to update the initial parameters, thereby obtaining the Gaussian meta-parameters of the 3D scene.

[0073] In one feasible implementation, the aforementioned perspective reconstruction module 230 is further used for: Obtain the camera pose from the target's perspective; Based on the camera pose of the target viewpoint, each Gaussian element in the 3D scene is projected onto the target imaging plane to obtain the projection area of ​​each Gaussian element on the target imaging plane and its corresponding color and transparency contribution. The projection regions of each 3D Gaussian element are sorted according to their depth values. Based on the sorting results, the color and transparency contributions of each Gaussian unit are accumulated and fused pixel by pixel to generate a synthetic image from the target viewpoint.

[0074] In one feasible implementation, when the target viewpoint is the original fisheye camera viewpoint, the aforementioned viewpoint reconstruction module 230 is further used for: Calculate the ray direction vector of each pixel in the original fisheye camera view; The ray direction vector is transformed to the coordinate system of each virtual pinhole camera and projected onto the corresponding virtual pinhole camera imaging plane to obtain the sampling coordinates in the rendered image of each virtual pinhole camera; The rendered images of each virtual pinhole camera are resampled according to the sampling coordinates, and the pixel values ​​obtained by resampling are weighted and fused to obtain a synthetic image from the original fisheye camera view.

[0075] The system provided in this application embodiment has the same implementation principle and technical effects as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the system embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0076] This application also provides an electronic device, such as... Figure 3 The diagram shows the structure of the electronic device 100, which includes a processor 31 and a memory 30. The memory 30 stores computer-executable instructions that can be executed by the processor 31. The processor 31 executes the computer-executable instructions to implement any of the methods described above.

[0077] exist Figure 3 In the illustrated embodiment, the electronic device further includes a bus 32 and a communication interface 33, wherein the processor 31, the communication interface 33, and the memory 30 are connected via the bus 32.

[0078] The memory 30 may include high-speed random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. Communication between this system network element and at least one other network element is achieved through at least one communication interface 33 (which can be wired or wireless), such as the Internet, wide area network, local area network, metropolitan area network, etc. The bus 32 may be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 32 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0079] Processor 31 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of processor 31 or by software instructions. Processor 31 can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or implemented by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor 31 reads the information in the memory and, in conjunction with its hardware, completes the steps of the method in the aforementioned embodiment.

[0080] This application also provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions cause the processor to implement the above-described method. For specific implementation details, please refer to the foregoing method embodiments, which will not be repeated here.

[0081] The computer program product of the vehicle-mounted fisheye imaging perspective reconstruction method, system and electronic device provided in the embodiments of this application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the methods described in the preceding method embodiments. For specific implementation, please refer to the method embodiments, which will not be repeated here.

[0082] Unless otherwise specifically stated, the relative steps, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of this application.

[0083] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for reconstructing the field of view of a vehicle-mounted fisheye imaging system, characterized in that, include: Acquire the raw fisheye image and camera parameters from the vehicle-mounted fisheye camera; Based on the camera parameters, the original fisheye image is equivalently split into multiple virtual pinhole camera images; Gaussian splashing 3D scene reconstruction is performed on the multiple virtual pinhole camera images to generate a synthetic image from the target perspective.

2. The method according to claim 1, characterized in that, Acquire raw fisheye images and camera parameters captured by the vehicle-mounted fisheye camera, including: The vehicle-mounted fisheye camera is calibrated relative to the vehicle body extrinsic parameters and the radar camera extrinsic parameters; Acquire point cloud data of the vehicle body collected by the radar camera; Align the vehicle body point cloud data with the original fisheye image, and register the vehicle body point cloud data to the image coordinate system of the original fisheye image; Motion distortion elimination processing is performed on the vehicle body point cloud data using vehicle pose information.

3. The method according to claim 1, characterized in that, The camera parameters include the extrinsic parameter matrix of the vehicle-mounted fisheye camera; Based on the camera parameters, the original fisheye image is equivalently split into multiple virtual pinhole camera images, including: Based on the position of the vehicle-mounted fisheye camera, multiple virtual pinhole cameras facing different directions are constructed according to a preset combination of yaw and pitch angles. Based on the size of the original fisheye image and the extrinsic parameter matrix of the vehicle-mounted fisheye camera, calculate the intrinsic parameter matrix and extrinsic parameter matrix of each virtual pinhole camera; The pixels of the original fisheye image are resampled to the viewpoints of each virtual pinhole camera by polar coordinate mapping to generate virtual pinhole camera images.

4. The method according to claim 3, characterized in that, Based on the size of the original fisheye image and the extrinsic parameter matrix of the vehicle-mounted fisheye camera, calculate the intrinsic and extrinsic parameter matrices of each virtual pinhole camera, including: The intrinsic parameter matrix of the virtual pinhole camera is calculated based on the preset field of view and the size of the original fisheye image. Calculate the rotation offset matrix of the virtual pinhole camera relative to the fisheye camera based on the preset yaw and pitch angles. Based on the rotation offset matrix and the extrinsic parameter matrix of the vehicle-mounted fisheye camera, the extrinsic parameter matrix of the virtual pinhole camera is calculated.

5. The method according to any one of claims 1-4, characterized in that, Gaussian splash 3D scene reconstruction is performed on the multiple virtual pinhole camera images to generate a synthetic image from the target perspective, including: Gaussian splash 3D scene reconstruction is performed based on the multiple virtual pinhole camera images to obtain the Gaussian primitive parameters of the 3D scene; Rendering is performed based on the Gaussian meta-parameters to generate a synthetic image from the target viewpoint.

6. The method according to claim 5, characterized in that, Gaussian splash 3D scene reconstruction is performed based on the multiple virtual pinhole camera images to obtain Gaussian meta-parameters of the 3D scene, including: An initial 3D scene is generated based on the multiple virtual pinhole camera images and 3D point cloud data; the initial 3D scene includes multiple 3D sample points. Three-dimensional Gaussian elements are constructed based on the three-dimensional sample points, wherein the initial parameters corresponding to the three-dimensional Gaussian elements include the spatial position of each Gaussian element, the anisotropic covariance parameter, and the appearance attribute parameter. Re-rendered images from the perspective of each virtual pinhole camera are generated based on the initial parameters; The initial parameters are updated based on the pixel error between the re-rendered image and the corresponding virtual pinhole camera image to obtain the various Gaussian primitive parameters of the 3D scene.

7. The method according to claim 6, characterized in that, Rendering is performed based on the Gaussian meta-parameters to generate a synthetic image from the target viewpoint, including: Obtain the camera pose from the target's perspective; Based on the camera pose of the target viewpoint, each Gaussian element in the 3D scene is projected onto the target imaging plane to obtain the projection area of ​​each Gaussian element on the target imaging plane and its corresponding color and transparency contribution. The projection regions of each 3D Gaussian element are sorted according to their depth values. Based on the sorting results, the color and transparency contributions of each Gaussian unit are accumulated and fused pixel by pixel to generate a synthetic image from the target viewpoint.

8. The method according to any one of claims 1-4, characterized in that, When the target viewpoint is the original fisheye camera viewpoint, the step of reconstructing a 3D scene from the multiple virtual pinhole camera images using Gaussian splashing to generate a synthetic image from the target viewpoint includes: Calculate the ray direction vector of each pixel in the original fisheye camera view; The ray direction vector is transformed to the coordinate system of each virtual pinhole camera and projected onto the corresponding virtual pinhole camera imaging plane to obtain the sampling coordinates in the rendered image of each virtual pinhole camera; The rendered images of each virtual pinhole camera are resampled according to the sampling coordinates, and the pixel values ​​obtained by resampling are weighted and fused to obtain a synthetic image from the original fisheye camera view.

9. A system for reconstructing the field of view of a vehicle-mounted fisheye imaging system, characterized in that, include: The image acquisition module is used to acquire the raw fisheye image and camera parameters of the vehicle-mounted fisheye camera; The splitting module is used to split the original fisheye image into multiple virtual pinhole camera images based on the camera parameters. The perspective reconstruction module is used to reconstruct a three-dimensional scene from the multiple virtual pinhole camera images using Gaussian splashing, generating a synthetic image from the target perspective.

10. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing computer-executable instructions executable by the processor, the processor executing the computer-executable instructions to implement the method of any one of claims 1 to 8.

Images