An image reconstruction method, an electronic device, and a storage medium
By performing directional distortion processing on the input distorted image, the problem of consistent imaging parameters in the 3DGS reconstruction algorithm during distorted image processing is solved, achieving pixel-level alignment and efficient reconstruction, thus improving the image reconstruction accuracy of the autonomous driving system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CORECHENG (BEIJING) TECHNOLOGY CO LTD
- Filing Date
- 2026-01-15
- Publication Date
- 2026-06-09
AI Technical Summary
Existing 3DGS reconstruction algorithms struggle to achieve consistent imaging parameters when processing distorted images, resulting in reconstruction results that cannot accurately match the real distorted images, thus limiting their application in high-precision autonomous driving scenarios.
By performing directional distortion processing on the input image, including coordinate system transformation and directional distortion operations, the reconstructed image is ensured to have the same distortion parameters as the input image, achieving pixel-level alignment.
It improves the accuracy and processing efficiency of image reconstruction, ensures the consistency between the reconstruction results and the imaging parameters of the input image, and enhances the control precision of the autonomous driving system.
Smart Images
Figure CN122176153A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of computer processing technology, and more specifically, to an image reconstruction method, electronic device, and storage medium. Background Technology
[0002] In Driving Automation Systems (DAS), 3D scene reconstruction technology is crucial, relying on image data captured by onboard cameras. 3D Gaussian Splatting (3DGS), as an emerging reconstruction technique, has demonstrated advantages in high-fidelity rendering and efficient reconstruction. However, conventional 3DGS reconstruction algorithms typically assume that the input image is distortion-free or precisely corrected. In real-world autonomous driving scenarios, images acquired by onboard cameras (especially wide-angle and fisheye lenses) are generally distorted. Existing methods often employ strategies such as pre-distortion removal or replacement of the projection model when dealing with these distorted images, but these methods have significant limitations: the former leads to a direct and accurate match between the reconstructed (distortion-free) image and the real distorted image captured by the vehicle in real time, making it difficult to guarantee intrinsic parameter consistency; the latter is difficult to apply in practice due to model complexity, high computational cost, and incomplete support from existing toolchains. These limitations make it difficult for conventional processing methods to achieve pixel-level precise alignment between the original distortion input and the final rendered output while maintaining consistency between the camera's internal and external parameters, thus restricting the effective application of 3DGS in high-precision scenarios such as autonomous driving. Summary of the Invention
[0003] This disclosure provides a new technical solution for image reconstruction, which reduces distortion anomalies in the reconstructed image, thereby improving the accuracy and processing efficiency of image reconstruction.
[0004] According to a first aspect of this disclosure, an image reconstruction method is provided, comprising: The original primitive parameters of the three-dimensional geometric primitives in the scene captured by the input image are transformed into the first coordinate system to obtain the first primitive parameters of the three-dimensional geometric primitives. The input image is a camera image with distortion. The first coordinate system is transformed from the world coordinate system to the camera coordinate system. The primitive parameters of the three-dimensional geometric primitives are used to describe the spatial attributes of the corresponding three-dimensional geometric primitives in the scene. By applying directional distortion to the first primitive parameter of the three-dimensional geometric primitive, the second primitive parameter of the three-dimensional geometric primitive is obtained. The second coordinate system transformation is performed on the second primitive parameters of the 3D geometric primitive to obtain the target primitive parameters of the 3D geometric primitive; wherein, the second coordinate system transformation is a transformation operation from the camera coordinate system to the world coordinate system; Image rendering is performed based on the target primitive parameters of three-dimensional geometric primitives to obtain the target image of the scene; wherein, directional distortion makes the target image have the same distortion parameters as the input image.
[0005] Optionally, an orientation distortion is applied to the first primitive parameter of the three-dimensional geometric primitive to obtain the second primitive parameter of the three-dimensional geometric primitive, including: Using the camera intrinsic parameters of the fixed input image as constraints, the first primitive parameters of the three-dimensional geometric primitive are subjected to directional distortion processing to obtain the second primitive parameters of the three-dimensional geometric primitive.
[0006] Optionally, in this method, at least a portion of the target image is rendered from a different perspective than the input image is captured from a different perspective.
[0007] Optionally, before applying directional distortion to the first primitive parameter of the three-dimensional geometric primitive, the method further includes: Based on the image features of the input image, determine the image type of the input image; Based on the image type of the input image, match the distortion model to apply directional distortion.
[0008] Optionally, the input image type is a planar camera image; directional distortion is applied to the first primitive parameters of the 3D geometric primitive to obtain the second primitive parameters of the 3D geometric primitive, including: Based on the position of the first primitive of the three-dimensional geometric primitive, the normalized coordinates and radial distance of the projection of the three-dimensional geometric primitive into the pinhole camera model are determined; Based on the position of the first primitive of the 3D geometric primitive, its normalized coordinates, and radial distance, determine the position of the second primitive of the 3D geometric primitive; and / or, Based on the preset correction matrix corresponding to the planar camera image, the first primitive scale of the three-dimensional geometric primitive is corrected to obtain the second primitive scale of the three-dimensional geometric primitive.
[0009] Optionally, the input image type is a fisheye camera image, and the primitive parameters include primitive positions; directional distortion is applied to the first primitive parameters of the 3D geometric primitives to obtain the second primitive parameters of the 3D geometric primitives, including: The position of the first primitive of the three-dimensional geometric primitive is transformed into the third coordinate system to obtain the position of the third primitive of the three-dimensional geometric primitive; wherein, the third coordinate system is transformed from the camera coordinate system to the spherical coordinate system. The position of the fourth element of the three-dimensional geometric primitive is determined based on the polar angle of the position of the third element of the three-dimensional geometric primitive. The fourth coordinate system transformation is performed on the fourth element position of the three-dimensional geometric primitive to obtain the second element position of the three-dimensional geometric primitive; wherein, the fourth coordinate system is transformed from the spherical coordinate system to the camera coordinate system.
[0010] Optionally, the input image type is a fisheye camera image, and the primitive parameters include primitive scale and primitive rotation parameters; directional distortion is applied to the first primitive parameters of the 3D geometric primitive to obtain the second primitive parameters of the 3D geometric primitive, including: Based on the preset correction matrix corresponding to the fisheye camera image, the first primitive scale and first primitive rotation parameters of the three-dimensional geometric primitive are corrected to obtain the second primitive scale and second primitive rotation parameters of the three-dimensional geometric primitive.
[0011] Optionally, after obtaining the target image of the scene, the method further includes: Calculate the rendering error of the target image relative to the input image; The reconstruction data is updated based on the rendering error; the reconstruction data includes at least one of the original primitive parameters of the 3D geometric primitives and the parameters related to the applied orientation distortion. The input image is iteratively reconstructed based on the reconstructed data to obtain the updated target image.
[0012] According to a third aspect of this disclosure, a computing device is provided, including a memory for storing a computer program; and a processor configured to implement the method as described in any of the first aspects when executing the computer program stored in the memory.
[0013] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the first aspects.
[0014] According to embodiments of this disclosure, in the application of 3D image reconstruction based on distorted images, after performing a first coordinate system transformation, directional distortion, and a second coordinate system transformation on the original primitive parameters of the 3D geometric primitives in the scene captured by the input image, target primitive parameters of the 3D geometric primitives are obtained. Image rendering is then performed based on the target primitive parameters of the 3D geometric primitives to obtain the target image reconstructed from the input image. This disclosure, by employing directional distortion on the distorted image, ensures that the rendered target image maintains consistency with the input image in terms of distortion parameters, which facilitates pixel-level alignment, improves the accuracy and coupling of image reconstruction, and thus enhances the control precision of autonomous driving.
[0015] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description
[0016] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments of the present disclosure and, together with their description, serve to explain the principles of the present disclosure.
[0017] Figure 1 This is a schematic diagram of a data processing system to which the methods provided in the embodiments of this disclosure can be applied.
[0018] Figure 2 This is a schematic diagram of an image reconstruction method according to some embodiments.
[0019] Figure 3 This is a schematic diagram of an optimized method for image reconstruction according to some embodiments.
[0020] Figure 4 This is a schematic diagram of an electronic device according to some embodiments. Detailed Implementation
[0021] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.
[0022] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.
[0023] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.
[0024] In all the examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.
[0025] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.
[0026] This disclosure relates to an image reconstruction scheme based on distorted images. In the field of autonomous driving, image reconstruction technology is rapidly evolving from traditional static, offline processing modes towards dynamic, real-time scene understanding. Currently, methods represented by 3D Gaussian sputtering and its dynamic extensions (such as 4D Gaussian sputtering) have become mainstream. By explicitly representing the scene with 3D Gaussian point clouds and combining it with differentiable rendering technology, a significant balance has been achieved between rendering quality and efficiency, providing strong support for simulation testing, digital twins, and real-time environmental perception in autonomous driving. However, direct reconstruction from distorted raw images acquired from onboard fisheye cameras presents challenges. Most algorithms still rely on pre-distortion processing of the image and post-processing to add distortion after rendering. This can lead to deviations between the reconstructed results and the real perceived data in terms of imaging parameter consistency and pixel-level alignment, limiting its direct application in tasks such as path planning.
[0027] Therefore, this disclosure provides an image reconstruction method that can achieve consistent imaging parameters. This method can be used to generate simulation scenes offline or to perform real-time scene image reconstruction online.
[0028] The method of this disclosure embodiment can be used... Figure 1 The data processing system implementation is shown. For example... Figure 1 As shown, the data processing system includes a mobile device 101 and a computing device 102. The mobile device 101 is equipped with a data acquisition device 1014, used to acquire image data and / or point cloud data of a set scene at a set acquisition frequency during the movement of the mobile device 101. The set scene is, for example, a road scene. The computing device 102 is used to process the image data and / or point cloud data acquired by the mobile device 101 to perform target recognition in the set scene based on the image data.
[0029] The mobile device 101 can be a data collection vehicle specifically designed for data acquisition, or it can be a moving vehicle, robot, drone, etc., without limitation. Figure 1 As shown, in addition to the acquisition device 1014, the mobile device 101 may also include a processor 1011, a memory 1012, a communication device 1013, etc., so that the mobile device 101 has the ability to perform set tasks and communicate with the outside world.
[0030] Figure 1 The computing device 102 can be any physically existing device, machine, or system capable of performing predefined computing tasks through hardware components, namely, receiving input images, processing images, and outputting recognition results. The computing device 102 can be, for example, a server, computer, controller, or system-on-a-chip. The computing device 102 can be set up independently of the mobile device 101, or it can be at least partially set up within the mobile device 101; this is not limited here.
[0031] like Figure 1 As shown, the computing device 102 may include a processor 1021 and a memory 1022. The processor 1021 may include at least one of a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), a Field Programmable Gate Array (FPGA), a System-on-Chip (SOC), an Application Specific Integrated Circuit (ASIC), a Micro Controller Unit (MCU), or other processors. The memory 1022 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. In addition to storing instructions, the memory may also store input data, intermediate data, and result data, and the data stored in the memory can be retrieved and used by the processor.
[0032] In addition, the computing device 102 may also include a communication device 1023 and an interface device 1024, which can, for example, receive and process images acquired by the camera through the communication device 1023 or the interface device 1024.
[0033] The following is combined Figure 1 The systems shown illustrate various embodiments of this disclosure.
[0034] <First Embodiment> Figure 2 An image reconstruction method according to some embodiments is shown, which can be performed by... Figure 1 The computing device 102 is implemented in the middle. For example... Figure 2 As shown, the image reconstruction method of this embodiment includes the following steps S210 to S240: Step S210: Perform a first coordinate system transformation on the original primitive parameters of the three-dimensional geometric primitives in the scene captured by the input image to obtain the first primitive parameters of the three-dimensional geometric primitives.
[0035] In this embodiment, the input image is a distorted camera image, that is, the input image is the original image captured by the camera.
[0036] In this embodiment, the first coordinate system is transformed from the world coordinate system to the camera coordinate system. This first coordinate system transformation can be performed based on the actual camera extrinsic parameters (including position and orientation) of the vehicle-mounted camera corresponding to the input image.
[0037] In this embodiment, 3D geometric primitives are the basic units representing the geometric structure of a 3D scene. The primitive parameters of a 3D geometric primitive are used to describe the spatial properties of the corresponding 3D geometric primitive in the scene. The spatial properties of a 3D geometric primitive include, for example, positional properties and morphological properties.
[0038] In this embodiment, the image reconstruction algorithm model used for image reconstruction can be 3DGS, Neural Radiance Field (NeRF), 4DGS (4D Gaussian Splatting), etc. For different reconstruction algorithms, the three-dimensional geometric primitives can be of different types, and their spatial properties are characterized by different primitive parameters. For example, the three-dimensional geometric primitives can be Gaussian kernels, voxels, point cloud clusters, or meshes. The following explanation uses 3DGS as an example of the image reconstruction algorithm model to illustrate this embodiment. In 3DGS, the three-dimensional geometric primitive is a Gaussian kernel, typically an anisotropic Gaussian kernel, whose contour lines are ellipses, capable of capturing directional features in the image.
[0039] This embodiment does not require distortion correction preprocessing of the original distorted image; it can be directly input into the 3DGS model as the input image.
[0040] The original primitive parameters of the Gaussian kernel in the scene corresponding to the input image are determined by the point cloud data corresponding to the input image. The input image and its corresponding point cloud data are simultaneously input into the 3DGS model. The Gaussian kernel in the scene corresponding to the input image is initialized based on the point cloud data. Initialization of the Gaussian kernel actually initializes its primitive parameters, including the center position, covariance matrix, opacity, and color. The initialized parameter set is the original primitive parameter set. The scale and rotation parameters (rotation matrix) of the Gaussian kernel can be obtained from the covariance matrix. The scale is determined by the eigenvalues of the covariance matrix, and the rotation parameters are determined by the eigenvectors of the covariance matrix. That is, the spatial attributes of the Gaussian kernel in the scene of the input image are determined by its center position and the covariance matrix. Spatial attributes include primitive position and primitive shape. The primitive shape further includes scale and rotation parameters. The scale describes the spatial extent of the primitive, and the rotation parameter describes the orientation angle of the primitive.
[0041] In some examples, since the coordinate system of the 3D geometric primitives in the scene captured by the input image is the world coordinate system, and the projection based on the camera intrinsic parameters needs to be performed in the camera coordinate system, the coordinate system of the 3D geometric primitives is transformed to the camera coordinate system. This operation is mainly performed around the position reference point of the 3D geometric primitive, such as the center position of the Gaussian kernel. This transformation operation uses the rotation matrix and translation vector composed of the camera extrinsic parameters. First, the position reference point is translated according to the translation vector, and then rotated according to the rotation matrix to obtain the primitive parameters of the 3D geometric primitives in the camera coordinate system, that is, the first primitive parameters.
[0042] Step S220: Apply directional distortion to the first primitive parameter of the three-dimensional geometric primitive to obtain the second primitive parameter of the three-dimensional geometric primitive.
[0043] In this embodiment, by applying directional distortion to the first primitive parameter of the three-dimensional geometric primitive in the camera coordinate system, the target image output after subsequent processing and rendering, based on the second primitive parameter of the three-dimensional geometric primitive after directional distortion, has the same image distortion unique to camera lens imaging as the input image.
[0044] In some examples, before applying directional distortion to the first primitive parameter of the 3D geometric primitives, the process includes determining the image type of the input image based on its image features, and then matching a distortion model to be applied based on the image type. Distortion models differ for images captured by different types of cameras, which leads to different primitive parameters needing synchronous correction during directional distortion application, and different directional distortion models being used for correction. Therefore, when the computing device receives the input image, it needs to first determine the type of camera that captured and provided the input image based on its image features, especially its distortion features—that is, the image type of the input image. Then, based on the image type of the input image, a suitable directional distortion model is selected to ensure that the rendering effect is consistent with the intended purpose.
[0045] In some examples, where the input image received by the computing device is a planar camera image, step S220 includes determining the normalized coordinates and radial distance of the projection of the three-dimensional geometric primitive into the pinhole camera model based on the first primitive position of the three-dimensional geometric primitive; and determining the second primitive position of the three-dimensional geometric primitive based on the first primitive position, the normalized coordinates, and the radial distance of the three-dimensional geometric primitive.
[0046] Projecting the 3D geometric primitives in the camera coordinate system onto the image plane of the pinhole camera model is equivalent to removing the radial distortion caused by the lens of the planar camera, resulting in a distortion-free image corresponding to the input image. This means obtaining the true positional relationship of each entity from the perspective of the input image. Then, by applying the same radial distortion as the input image, the radial distortion that occurs when the planar camera that acquires the input image is imaging is imitated, so that the distortion mode of the subsequently rendered target image is consistent with that of the input image.
[0047] In this example, images captured by a planar camera generally exhibit radial distortion. For this type of input image, this embodiment first projects the 3D geometric primitives onto the image plane of the pinhole camera model during the orientation distortion stage, obtaining the normalized coordinates of the 3D geometric primitives projected onto the pinhole camera model. The radial distance is then calculated using these normalized coordinates, for example, using the following formula:
[0048] Among them, (X) C Y C Z C Let (u0, v0) be the position of the first primitive of the three-dimensional geometric primitive, (u0, v0) be the normalized coordinates of the projection of the three-dimensional geometric primitive, and r0 be the radial distance of the projection of the three-dimensional geometric primitive.
[0049] Based on the normalized coordinates and radial distance of the projection of the above-mentioned three-dimensional geometric primitives, an orientation distortion model of the first primitive position is established. In the model, the second primitive position of the three-dimensional geometric primitive after orientation distortion is (X... C ', Y C ', Z C The following equation must be satisfied:
[0050] Radial distortion of a planar camera image does not affect depth perception; therefore, the depth Z at the second primitive position is... C 'Depth Z relative to the position of the first primitive C They are the same, that is, Z C '=Z C Therefore, we can obtain...
[0051] Where r is the radial distance from the position of the first primitive of the three-dimensional geometric primitive.
[0052] In other examples, when the input image received by the computing device is a planar camera image, step S220 includes correcting the first primitive scale of the three-dimensional geometric primitive based on a preset correction matrix corresponding to the planar camera image to obtain the second primitive scale of the three-dimensional geometric primitive.
[0053] Applying the same radial distortion as the input image to the 3D geometric primitives projected onto the image plane of the pinhole camera model may cause the camera intrinsics of the final rendered target image to deviate from those of the input image. By synchronously correcting the primitive parameters based on the applied distortion, the camera intrinsics of the target image rendered based on the corrected primitive parameters are made consistent with those of the input image.
[0054] In this example, during directional distortion, the position of the first primitive of the 3D geometric primitive changes to the position of the second primitive, causing a mismatch between the scale of the 3D geometric primitive and the resulting distortion. The scale of the 3D geometric primitive is corrected using a pre-defined correction matrix corresponding to the planar camera image.
[0055] The preset correction matrix can be a Jacobian matrix, which is constructed using the directional distortion formula, as follows:
[0056]
[0057]
[0058] Where k1, k2, and k3 are camera distortion parameters, K = Given the directional distortion polynomial, combined with the expression for the radial distance r0 of the projection of the 3D geometric primitives, the distortion mapping of the directional distortion function in the camera coordinate system can be obtained. Jacobian matrix J f (3×3) is:
[0059] The Jacobian matrix above is used to correct the scale of the first primitive of the three-dimensional geometric primitive, so that the scale of the second primitive obtained after correction matches the radial distortion of the second primitive position, ensuring that the camera intrinsic parameters remain unchanged and avoiding blurred areas at the edges of the rendered image due to scale mismatch.
[0060] In some examples, where the input image received by the computing device is a fisheye camera image, the primitive parameters include primitive positions. Step S220 includes performing a third coordinate system transformation on the first primitive position of the three-dimensional geometric primitive to obtain the third primitive position of the three-dimensional geometric primitive; wherein, the third coordinate system is transformed from the camera coordinate system to the spherical coordinate system; based on the polar angle of the third primitive position of the three-dimensional geometric primitive, the fourth primitive position of the three-dimensional geometric primitive is determined; and a fourth coordinate system transformation is performed on the fourth primitive position of the three-dimensional geometric primitive to obtain the second primitive position of the three-dimensional geometric primitive; wherein, the fourth coordinate system is transformed from the spherical coordinate system to the camera coordinate system.
[0061] In this example, since fisheye camera distortion typically only considers the polar angle distortion θ, the Cartesian coordinates (X, X, ...) of the first primitive position in the camera coordinate system are... C Y C Z C The coordinates are converted to spherical coordinates (ρ, θ, φ) in a spherical coordinate system, which represents the position of the third primitive. Here, ρ is the radial distance, representing the distance from the center of the 3D geometric primitive to the camera origin; θ is the polar angle, representing the distance between the center of the 3D geometric primitive and the polar axis Z. C The angle between the two axes; φ is the azimuth angle, used to characterize the position of the center of the three-dimensional geometric primitive relative to the polar axis Z. C The projection onto the perpendicular plane and the orthogonal X C The included angle of the axes. By separating the sensitive dimension of polar angle θ in fisheye camera distortion, interference from radial distance ρ and azimuth angle φ is avoided. The transformation from Cartesian coordinates to spherical coordinates can be performed using the following formula: radial distance ; Polar angle ; Azimuth .
[0062] The fisheye camera-specific distortion model employs an 8th-order polynomial to fit a nonlinear mapping of the fisheye polar angle, and the constructed mapping function is as follows:
[0063] Where θ' is the polar angle after orientation distortion, and a0~a8 are the polynomial coefficients obtained by fitting the camera parameters.
[0064] Transform the spherical coordinates (ρ, θ', φ) of the fourth element position obtained after orientation distortion back to Cartesian coordinates (X) in the camera coordinate system. C ', Y C ', Z C The conversion formula is as follows:
[0065]
[0066]
[0067] After the above transformation and orientation distortion, the position of the second primitive of the three-dimensional geometric primitive is obtained.
[0068] In other examples, when the input image received by the computing device is a fisheye camera image, step S220 includes correcting the first primitive scale and the first primitive rotation parameter of the three-dimensional geometric primitive based on a preset correction matrix corresponding to the fisheye camera image, to obtain the second primitive scale and the second primitive rotation parameter of the three-dimensional geometric primitive.
[0069] In this example, the first primitive position of the 3D geometric primitive is transformed to the second primitive position through directional distortion. In a fisheye camera image, this results in the scale and rotation parameters of the 3D geometric primitive not matching the fisheye camera distortion. Similar to the processing method for planar camera images, a Jacobian matrix corresponding to the fisheye camera directional distortion function is constructed. This Jacobian matrix is derived based on the transformation derivative from the third primitive position to the second primitive position of the 3D geometric primitive, with its eigenvalues being scale correction coefficients and its eigenvectors representing the rotation parameter correction directions.
[0070] Based on the Jacobian matrix corresponding to the fisheye camera's directional distortion function, the first primitive scale is distorted according to the eigenvalues of the Jacobian matrix, and the first primitive rotation parameter is distorted according to the eigenvectors of the Jacobian matrix. The corrected scale and rotation axis are then Gram-Schmidt orthogonalized to ensure that the correction result meets the requirements of orthogonal axis, so as to avoid rendering artifacts after rendering.
[0071] Step S230: Perform a second coordinate system transformation on the second primitive parameters of the three-dimensional geometric primitive to obtain the target primitive parameters of the three-dimensional geometric primitive; wherein, the second coordinate system is transformed from the camera coordinate system to the world coordinate system.
[0072] In this embodiment, the second primitive parameter of the 3D geometric primitive is the primitive parameter that has undergone directional distortion and needs to be transformed from the camera coordinate system to the world coordinate system before subsequent rendering can be performed. As the inverse transformation of the first coordinate system transformation, the second coordinate system transformation can also be performed based on the actual camera extrinsic parameters of the vehicle-mounted camera corresponding to the input image.
[0073] In some examples, the input image is a planar camera image. Since the rotation parameters are not corrected during orientation distortion, the transformation back to the world coordinate system uses the transpose of the original rotation parameters.
[0074] In other examples, the input image is a fisheye camera image, and the rotation parameters are corrected based on the eigenvectors of the Jacobian matrix during the orientation distortion process. Therefore, the transpose of the corrected rotation parameters is used to transform back to the world coordinate system.
[0075] Step S240: Image rendering is performed based on the target primitive parameters of the three-dimensional geometric primitives to obtain the target image of the scene. The directional distortion makes the target image have the same distortion parameters as the input image.
[0076] In this embodiment, the target primitive parameters obtained by directional distortion of the three-dimensional geometric primitives are input into a pinhole raster renderer for rendering, and the reconstructed target image is output. By applying directional distortion to the first primitive parameters of the three-dimensional geometric primitives in step S220, the target image and the input image are kept consistent at least in terms of distortion parameters, which is beneficial for achieving pixel-level alignment.
[0077] In some examples, applying directional distortion to the first primitive parameter of the 3D geometric primitive is performed under the constraint of keeping the camera intrinsics of the input image constant. That is, after directional distortion of the coordinates of the reference point of the 3D geometric primitive, considering that the camera intrinsics will change due to distortion, at least one parameter among the scale and rotation parameters of the 3D geometric primitive can be simultaneously corrected to keep the camera intrinsics of the final rendered image unchanged. The camera intrinsics include focal length, principal point coordinates, and pixel aspect ratio. In this example, by applying directional distortion to the first primitive parameter of the 3D geometric primitive under the constraint of fixed camera intrinsics, the target image and the input image can be made consistent in terms of distortion parameters and camera intrinsics, thereby achieving pixel-level alignment between the target image and the input image.
[0078] In some examples, at least part of the target image is rendered from a different perspective than the input image was captured from.
[0079] Taking 3DGS as an example, during the process of image reconstruction of the scene corresponding to the input image, 3DGS establishes a three-dimensional scene model of the scene based on multiple input images and point cloud data from multiple shooting perspectives. Therefore, 3DGS can render target images from any shooting perspective that the user needs, which is different from the input image, according to the user's settings. It can also add composite information such as depth information while rendering the two-dimensional target image.
[0080] For example, if an input image from a certain perspective contains an obstruction, making it impossible to obtain sufficient useful information from the input image, then a target image from a perspective that does not contain the obstruction can be rendered based on the constructed 3D scene model. The obstruction can be dirt directly blocking the camera lens, or a target in the scene that creates a visual blind spot, preventing information from being obtained from that blind spot, such as a large truck.
[0081] For example, if there is image noise in the input image from a certain perspective that may be misidentified, such as raindrops, water splashes, or reflections, the target image that excludes these image noises can be rendered by constructing a 3D scene model, or a perspective with relatively less image noise can be selected for rendering.
[0082] In other examples, when rendering using a constructed 3D scene model, the pixel blurring of the input image can be compensated for. For pixel-blurred areas of the image, pixel-aligned replacements can be performed based on the 3D scene model to restore the image information that the area should have originally displayed.
[0083] This disclosure's embodiments transform the original primitive parameters of the 3D geometric primitives in the scene captured by the input image to the camera coordinate system, then perform directional distortion, and finally transform back to the world coordinate system for image rendering to obtain the reconstructed target image of the input image. The image reconstruction method of this disclosure achieves the effect of radial projection in the rendered reconstructed image by performing directional distortion on the distorted image during the image reconstruction process. Since image reconstruction is based on the original image without distortion correction, the loss of image information data is avoided. Furthermore, it achieves consistency of camera intrinsic parameters and pixel-level alignment before and after reconstruction, improving the accuracy and efficiency of image reconstruction.
[0084] <Second Embodiment> In this embodiment, after obtaining the target image reconstructed from the input image, as follows: Figure 3 As shown, the method further includes the following steps S310 to S330: Step S310: Calculate the rendering error of the target image relative to the input image.
[0085] Step S320: Update the reconstruction data based on the rendering error; wherein the reconstruction data includes at least one of the original primitive parameters of the three-dimensional geometric primitives and the parameters related to the applied orientation distortion.
[0086] Step S330: Perform iterative image reconstruction on the input image based on the reconstruction data to obtain the updated target image.
[0087] In this example, to improve the image reconstruction effect—that is, to reduce the rendering error between the reconstructed target image and the input image (ground image)—the rendering error between the target image and the input image can be calculated. This is then used to backpropagate and update at least one parameter from the original primitive parameters and the applied orientation distortion-related parameters of each 3D geometric primitive. These orientation distortion-related parameters can include coefficients from the orientation distortion formula and the orientation distortion function. Image reconstruction is then performed again based on the updated parameters. This process is iterated until a preset number of iterations is reached or the rendering error of the updated target image relative to the input image is less than a preset error threshold.
[0088] While iterating the target image online, the mapping model of the three-dimensional geometric primitives obtained from the input image and its point cloud data can also be iteratively updated online, improving the accuracy of the mapping model, reducing the number of iterations required for image reconstruction, and thus improving the efficiency of image reconstruction.
[0089] <Third Embodiment> This embodiment provides a computing device, such as Figure 4 As shown, the computing device 400 includes a memory 420 and a processor 410. The memory 420 is used to store a computer program executed by the processor 410. The processor 410 is configured to implement the method according to any embodiment of the present disclosure when executing the computer program stored in the memory 420.
[0090] The computing device can be set up independently of mobile devices such as vehicles, for example as a server, or it can be set up at least partially on mobile devices such as vehicles.
[0091] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0092] This disclosure may be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement any of the methods in the foregoing embodiments of this disclosure.
[0093] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media may include, for example, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), compact disc-read-only memory (CD-ROM), digital versatile disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any combination thereof. The computer-readable storage medium used herein is not to be interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0094] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include one or more of copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to computer-readable storage media in the respective computing / processing device.
[0095] The computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object programs written in any combination of one or more programming languages, including object-oriented programming languages (such as Smalltalk, C++, etc.) and conventional procedural programming languages (such as the "C" language or similar programming languages). The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network (e.g., a local area network or a wide area network), or it may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays, or programmable logic arrays, can execute computer-readable program instructions to implement various aspects of the embodiments of this disclosure by utilizing state information from the computer-readable program instructions.
[0096] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0097] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.
[0098] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It should be noted that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are all equivalent.
[0100] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of this disclosure is defined by the appended claims.
Claims
1. An image reconstruction method, characterized in that, include: The original primitive parameters of the three-dimensional geometric primitives in the scene captured by the input image are transformed into a first coordinate system to obtain the first primitive parameters of the three-dimensional geometric primitives; wherein, the input image is a camera image with distortion, the first coordinate system is transformed into a transformation operation from the world coordinate system to the camera coordinate system, and the primitive parameters of the three-dimensional geometric primitives are used to describe the spatial properties of the corresponding three-dimensional geometric primitives. A directional distortion is applied to the first primitive parameter of the three-dimensional geometric primitive to obtain the second primitive parameter of the three-dimensional geometric primitive. The second primitive parameters of the three-dimensional geometric primitive are transformed into a second coordinate system to obtain the target primitive parameters of the three-dimensional geometric primitive; wherein, the second coordinate system is transformed from the camera coordinate system to the world coordinate system. Image rendering is performed based on the target primitive parameters of the three-dimensional geometric primitives to obtain the target image of the scene; wherein, the directional distortion makes the target image have the same distortion parameters as the input image.
2. The method according to claim 1, characterized in that, The step of applying directional distortion to the first primitive parameter of the three-dimensional geometric primitive to obtain the second primitive parameter of the three-dimensional geometric primitive includes: Using the camera intrinsic parameters of the fixed input image as constraints, the first primitive parameters of the three-dimensional geometric primitive are subjected to directional distortion processing to obtain the second primitive parameters of the three-dimensional geometric primitive.
3. The method according to claim 1, characterized in that, At least a portion of the target image is rendered from a different perspective than the input image was captured from.
4. The method according to claim 1, characterized in that, Before applying directional distortion to the first primitive parameter of the three-dimensional geometric primitive, the method further includes: Based on the image features of the input image, determine the image type of the input image; Based on the image type of the input image, a distortion model for applying the directional distortion is matched.
5. The method according to claim 4, characterized in that, The input image is a planar camera image; the process of applying directional distortion to the first primitive parameters of the three-dimensional geometric primitive to obtain the second primitive parameters of the three-dimensional geometric primitive includes: Based on the position of the first primitive of the three-dimensional geometric primitive, the normalized coordinates and radial distance of the projection of the three-dimensional geometric primitive into the pinhole camera model are determined; Based on the first primitive position of the three-dimensional geometric primitive and the normalized coordinates and radial distance, determine the second primitive position of the three-dimensional geometric primitive; and / or, Based on a preset correction matrix corresponding to the planar camera image, the first primitive scale of the three-dimensional geometric primitive is corrected to obtain the second primitive scale of the three-dimensional geometric primitive.
6. The method according to claim 4, characterized in that, The input image is a fisheye camera image, and the primitive parameters include primitive positions; applying directional distortion to the first primitive parameters of the three-dimensional geometric primitive to obtain the second primitive parameters of the three-dimensional geometric primitive includes: The position of the first primitive of the three-dimensional geometric primitive is transformed into the third coordinate system to obtain the position of the third primitive of the three-dimensional geometric primitive; wherein, the third coordinate system is transformed from the camera coordinate system to the spherical coordinate system. The position of the fourth element of the three-dimensional geometric primitive is determined based on the polar angle of the position of the third element of the three-dimensional geometric primitive. The fourth coordinate system transformation is performed on the fourth primitive position of the three-dimensional geometric primitive to obtain the second primitive position of the three-dimensional geometric primitive; wherein, the fourth coordinate system is transformed from the spherical coordinate system to the camera coordinate system.
7. The method according to claim 4 or 6, characterized in that, The input image is a fisheye camera image, and the primitive parameters include primitive scale and primitive rotation parameters; applying directional distortion to the first primitive parameters of the three-dimensional geometric primitive to obtain the second primitive parameters of the three-dimensional geometric primitive includes: Based on the preset correction matrix corresponding to the fisheye camera image, the first primitive scale and the first primitive rotation parameter of the three-dimensional geometric primitive are corrected to obtain the second primitive scale and the second primitive rotation parameter of the three-dimensional geometric primitive.
8. The method according to any one of claims 1 to 4, characterized in that, After obtaining the target image of the scene, the method further includes: Calculate the rendering error of the target image relative to the input image; The reconstruction data is updated based on the rendering error; wherein the reconstruction data includes at least one of the original primitive parameters of the three-dimensional geometric primitives and the applied orientation distortion related parameters; Based on the reconstructed data, the input image is iteratively reconstructed to obtain the updated target image.
9. A computing device, characterized in that, include: Memory, used to store computer programs; as well as, The processor, the processing unit, is configured to implement the method of any one of claims 1 to 8 when executing a computer program stored in memory.
10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When a computer program is executed by a processor, it implements the method of any one of claims 1 to 8.