Image rendering method, medium and equipment

Abstract

The invention discloses an image rendering method, a medium and equipment, and the method comprises the steps: constructing a Gaussian ellipsoid obeying three-dimensional Gaussian distribution based on a three-dimensional space, and enabling the Gaussian ellipsoid to be formed by taking a central point with an initial attribute as a center; taking the camera pose of the fisheye camera model and the initial attribute of the Gaussian ellipsoid as the input of a micro renderer, and performing image rendering by the micro renderer based on the input camera pose and the initial attribute of the Gaussian ellipsoid to obtain a fisheye camera rendering image under the camera pose; the micro renderer is built based on a fisheye camera imaging model; the difference between the rendered image and a corresponding real image is calculated, the initial attribute of the Gaussian ellipsoid is optimized based on the difference, and the real image is an image in the real space under the same camera pose as the rendered image. Through the above mode, three-dimensional Gaussian splashing can directly use fisheye camera image data, and the method is more suitable for an automatic driving scene.

Description

Technical Field

[0001] The present application relates to the field of computer graphics, and in particular to an image rendering method, medium and device. Background Art

[0002] New perspective data generation is to reconstruct the entire scene by inputting images from certain camera perspectives to output images from new perspectives. High-quality new perspective data generation can be used in scenarios such as autonomous driving simulation and corner case data synthesis, thereby promoting the advancement of autonomous driving technology.

[0003] In the prior art, new perspectives can be synthesized through 3D Gaussian Splatting (3DGS). However, this method can only take pinhole camera images as input and output, and cannot process distorted fisheye camera image data that is more commonly used in autonomous driving scenarios. Summary of the invention

[0004] The present application mainly provides an image rendering method, medium and device to solve the problem that three-dimensional Gaussian splashing cannot process fisheye camera image data.

[0005] To solve the above technical problems, a technical solution adopted in the present application is: to provide an image rendering method based on a fisheye camera model, comprising: constructing a Gaussian ellipsoid obeying a three-dimensional Gaussian distribution based on a three-dimensional space, wherein the Gaussian ellipsoid is centered on a center point with initial attributes; taking the camera pose of the fisheye camera model and the initial attributes of the Gaussian ellipsoid as inputs of a differentiable renderer, and the differentiable renderer performs image rendering based on the input camera pose and the initial attributes of the Gaussian ellipsoid to obtain a fisheye camera rendered image under the camera pose; the differentiable renderer is built based on a fisheye camera imaging model; calculating the difference between the rendered image and the corresponding real image, and optimizing the initial attributes of the Gaussian ellipsoid based on the difference, wherein the real image is an image in real space under the same camera pose as the rendered image.

[0006] By converting points in real space to the image plane and rendering and optimizing them in the form of fisheye camera images, the three-dimensional Gaussian splashing method can process fisheye camera images, and optimize the rendering process through back propagation to render images that are closer to reality.

[0007] In some embodiments, the differentiable renderer performs image rendering based on the input camera pose and initial properties of the Gaussian ellipsoid to obtain a fisheye camera rendered image under the camera pose, including: calculating the mean vector of the center point of the Gaussian ellipsoid in the image coordinate system; calculating the covariance of the Gaussian ellipsoid in the image coordinate system; the differentiable renderer performs image rendering based on the mean vector and covariance to obtain a fisheye camera rendered image under the camera pose.

[0008] A two-dimensional Gaussian distribution is defined by a mean vector and a covariance matrix, where the mean vector describes the center position of the distribution and the covariance matrix describes the degree of dispersion of the data in each dimension and the correlation between dimensions.

[0009] In some embodiments, the calculating the mean vector of the center point of the Gaussian ellipsoid in the image coordinate system includes: projecting the center point of the Gaussian ellipsoid in the three-dimensional space to the camera coordinate system through a projection matrix to obtain the camera coordinates of the center point of the Gaussian ellipsoid in the camera coordinate system; obtaining camera distortion parameters involved in the camera, and converting the camera coordinates into the mean vector in the image coordinate system based on the camera intrinsic parameters and the camera distortion parameters.

[0010] In some embodiments, the formula for converting the camera coordinates into the mean vector in the image coordinate system is:

[0011]

[0012] Where u is the latitude coordinate of the mean vector, v is the longitude coordinate of the mean vector, [u, v] T represents the transpose of the matrix [u, v], C x is the latitude coordinate of the camera optical center of the fisheye camera model in the camera coordinate system, C y is the longitude coordinate of the camera optical center of the fisheye camera model in the camera coordinate system, f x is the focal length of the fisheye camera model in the latitude direction in the camera coordinate system, f y is the focal length of the fisheye camera model in the longitude direction of the camera coordinate system, h is the first function related to the camera distortion parameter, m is the first intermediate variable related to the camera coordinate, n is the second intermediate variable related to the camera coordinate, x is the latitude coordinate of the camera coordinate, and y is the longitude coordinate of the camera coordinate.

[0013] In some embodiments, calculating the covariance of the Gaussian ellipsoid in the image coordinate system includes: establishing a Jacobian matrix about the center point; obtaining the covariance of the Gaussian ellipsoid in the camera coordinate system, and calculating the covariance of the Gaussian ellipsoid in the image coordinate system through the covariance in the camera coordinate system and the Jacobian matrix.

[0014] In some embodiments, the establishing of a Jacobian matrix about the center point includes: calculating the derivatives of the center point in the image coordinate system with respect to the camera coordinate parameters; and constructing the Jacobian matrix using the derivatives as matrix parameters.

[0015] The Jacobian matrix is ​​a 3×3 positive definite matrix.

[0016] In some embodiments, the formula for calculating the derivative of the center point in the image coordinate system with respect to the camera coordinate parameter includes:

[0017]

[0018] or,

[0019] Wherein, u is the latitude coordinate of the mean vector, a is the latitude coordinate or longitude coordinate of the camera coordinate, z is the height coordinate of the camera coordinate, h is the first function related to the camera distortion parameter, m is the first intermediate variable related to the camera coordinate, and f x is the focal length of the fisheye camera model in the latitude direction in the camera coordinate system.

[0020] In some embodiments, the formula for calculating the derivative of the center point in the image coordinate system with respect to the camera coordinate parameter includes:

[0021]

[0022] or,

[0023] Wherein, v is the longitude coordinate of the mean vector, a is the latitude coordinate or longitude coordinate of the camera coordinate, z is the height coordinate of the camera coordinate, h is the first function related to the camera distortion parameter, n is the second intermediate variable related to the camera coordinate, f y is the focal length of the fisheye camera model in the longitude direction in the camera coordinate system.

[0024] In some embodiments, the calculating the difference between the rendered image and the corresponding real image and optimizing the initial properties of the Gaussian ellipsoid based on the difference includes: using the real image as a supervisory signal to calculate the difference between the rendered image and the real image; calculating the gradient of the loss function with respect to the initial parameters, and updating the initial parameters through a gradient descent algorithm until the loss function is less than a preset threshold.

[0025] The error of the rendered image is corrected through the real image, and the initial parameters of the Gaussian ellipsoid are updated to reconstruct the entire real space scene in the virtual space, so as to render an image that is closer to the real one and synthesize images from a new perspective.

[0026] In some embodiments, after optimizing the initial properties of the Gaussian ellipsoid based on the difference, it also includes: inputting the camera pose into the differentiable renderer, and the differentiable renderer performs image rendering based on the optimized Gaussian ellipsoid to obtain a fisheye camera composite image corresponding to the camera pose.

[0027] By using the optimized initial attributes for image rendering in a differentiable renderer, an image with a new perspective can be synthesized according to a given camera pose, and the output of the differentiable renderer is a fisheye camera image.

[0028] In order to solve the above technical problem, another technical solution adopted by the present application is: providing a storage medium on which program data is stored, and when the program data is executed by a processor, the steps of the image rendering method as described above are implemented.

[0029] The beneficial effects of the storage medium refer to the beneficial effects of the above-mentioned image rendering method, which will not be repeated here.

[0030] The present application also provides a computer device, including a processor and a memory connected to each other, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the image rendering method as described above are implemented.

[0031] The beneficial effects of the computer device refer to the beneficial effects of the above-mentioned image rendering method, which will not be repeated here.

[0032] The beneficial effects of the present application are as follows: Different from the prior art, the present application discloses an image rendering method, medium and device, which constructs a Gaussian ellipsoid that obeys a three-dimensional Gaussian distribution based on a three-dimensional space, and uses a central point with initial attributes as the center of the Gaussian ellipsoid to convert the points in the three-dimensional space into data that obeys a three-dimensional Gaussian distribution in the three-dimensional space, so as to directly process the fisheye camera image. The camera pose of the fisheye camera model and the initial attributes of the Gaussian ellipsoid are used as inputs of a differentiable renderer for image rendering to obtain a fisheye camera rendered image under the camera pose, wherein the differentiable renderer is built based on the fisheye camera imaging model, directly uses the camera pose as input, and renders based on the Gaussian ellipsoid data, so that the differentiable renderer directly processes the fisheye camera image and outputs a distorted fisheye camera rendered image, which is more suitable for autonomous driving scenarios. The difference between the rendered image and the corresponding real image is calculated, and the differentiable renderer is optimized based on the difference; wherein the real image is an image in the real space under the same camera pose as the rendered image. By optimizing the initial parameters of the Gaussian ellipsoid, the rendered image output by the differentiable renderer is closer to the real image. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0034] Figure 1 It is a flowchart of an embodiment of an image rendering method provided by the present application;

[0035] Figure 2 Yes Figure 1 The flowchart of step 20 of an embodiment of the image rendering method shown;

[0036] Figure 3 Yes Figure 2 The flowchart of step 21 of the image rendering method shown in FIG.

[0037] Figure 4 Yes Figure 2 The flowchart of step 22 of the image rendering method shown in the figure is an embodiment;

[0038] Figure 5 Yes Figure 4 The flowchart of step 221 of the image rendering method shown in FIG.

[0039] Figure 6 Yes Figure 1The flowchart of step 30 of an embodiment of the image rendering method shown;

[0040] Figure 7 It is a structural schematic diagram of an embodiment of a storage medium provided by the present application;

[0041] Figure 8 It is a structural diagram of an embodiment of a computer device provided by the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] The terms "first", "second", "third" in the embodiments of the present application are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first", "second", "third" can expressly or implicitly include at least one of the features. In the description of the present application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0044] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0045] See also Figure 1 , Figure 1 : is a flow chart of an embodiment of an image rendering method provided by the present application. The image rendering method based on a fisheye camera model includes:

[0046] Step 10: Construct a Gaussian ellipsoid that obeys the three-dimensional Gaussian distribution based on the three-dimensional space. The Gaussian ellipsoid is constructed with the center point with initial attributes as the center.

[0047] The points in the initial three-dimensional space are obtained through structure-from-motion (SFM) or lidar, and the corresponding learnable initial attributes such as size, color and transparency are given to these points. Each point in the three-dimensional space is regarded as the center of the Gaussian ellipsoid, so as to convert the point into a Gaussian ellipsoid in the three-dimensional space that obeys the three-dimensional Gaussian distribution.

[0048] The shape and orientation of a Gaussian distribution in three-dimensional space is determined by the covariance matrix.

[0049] Specifically, during the construction of the Gaussian ellipsoid, an initial covariance matrix is ​​first set to determine the shape of the Gaussian ellipsoid, and the size of the Gaussian ellipsoid in each direction is changed by adjusting the diagonal elements of the covariance matrix; the color of the Gaussian ellipsoid is represented by the spherical harmonic function, and other parameters contained in the Gaussian ellipsoid are initialized according to the application scenario and algorithm requirements.

[0050] Step 20: The camera pose of the fisheye camera model and the initial properties of the Gaussian ellipsoid are used as inputs of the differentiable renderer. The differentiable renderer performs image rendering based on the input camera pose and the initial properties of the Gaussian ellipsoid to obtain a fisheye camera rendered image under the camera pose.

[0051] The differentiable renderer determines the position and shape of the Gaussian ellipsoid in space based on its initial properties, and then projects the Gaussian ellipsoid onto the camera's imaging plane based on the camera pose of the input fisheye camera model. The color value of each pixel in the image is calculated by combining lighting conditions and material properties, and the final color value is synthesized to form a fisheye camera rendered image under the camera pose. The fisheye camera model is a special camera model that allows a larger field of view and is usually used to capture wide-angle or panoramic images. Compared with the ordinary pinhole camera model, the fisheye camera model produces significant distortion during the imaging process. This distortion is caused by the refraction of light when it passes through the fisheye lens.

[0052] Camera pose refers to the position and attitude of the camera in space, which is usually described by the camera's rotation matrix and translation vector. In a differentiable renderer, camera pose is one of the key inputs of the rendering process, which determines the camera's perspective and position in space, thus affecting the content of the rendered image.

[0053] A differentiable renderer is a computational tool based on the fisheye camera imaging model that can generate realistic images based on given camera pose, lighting conditions, object surface properties and other parameters. In a differentiable renderer, properties such as the shape, material and texture of an object can be accurately simulated and rendered.

[0054] Differentiable renderers can accurately project and render based on the initial properties of the Gaussian ellipsoid, further improving the accuracy of the rendering results. In real-time rendering and interactive applications, differentiable renderers usually use efficient rendering algorithms and hardware acceleration technologies to generate high-quality rendered images in a short time. Differentiable renderers can calculate the gradient of the image relative to the input parameters, which makes it possible to implement gradient-based optimization algorithms. Through the optimization algorithm, the difference between the rendered image and the real scene can be further reduced, and the realism of the rendering results can be improved.

[0055] The method of using the camera pose of the fisheye camera model as the input of the differentiable renderer and performing image rendering based on the initial properties of the Gaussian ellipsoid allows the rendering perspective to be flexibly changed by adjusting the camera pose, thereby generating fisheye camera rendered images under different perspectives. The adjustment of the camera pose can be accurate to tiny angles and position changes to ensure the accuracy of the rendering results. The fisheye camera model can accurately simulate the distortion effect of the lens, ensure the visual consistency of the rendered image with the real scene, and can be used to generate high-quality 3D model rendered images for 3D reconstruction.

[0056] Furthermore, step 20 also includes:

[0057] Step 21: Calculate the mean vector of the center point of the Gaussian ellipsoid in the image coordinate system.

[0058] In three-dimensional space, the center point of the Gaussian ellipsoid is represented by a vector, denoted as loc world =[x world ,y world , z world , 1]. This vector defines the spatial position of the center of the ellipsoid. The image coordinate system is usually a two-dimensional plane used to represent points on the image. In the standard case, the origin of the image coordinate system is located in the upper left corner of the image, with the x-axis pointing to the right and the y-axis pointing downward, denoted by loc p =[u,v].

[0059] The image coordinate system is a coordinate system in the image plane, which is used to describe the position of points on the image. The upper left corner can be used as the origin, the right is the positive direction of latitude, and the upward direction is the positive direction of longitude.

[0060] Since the center point of the Gaussian ellipsoid is located in three-dimensional space, a projection operation is required to transform it into a two-dimensional image coordinate system. The projection process involves the camera's internal parameters such as focal length, optical center, etc. and external parameters such as camera rotation, translation and distortion.

[0061] Furthermore, step 22 also includes:

[0062] Step 211: Project the center point of the Gaussian ellipsoid in the three-dimensional space into the camera coordinate system through the projection matrix to obtain the camera coordinates of the center point of the Gaussian ellipsoid in the camera coordinate system.

[0063] For the center point loc of the Gaussian ellipsoid in three-dimensional space world =[x world ,y world , z world , 1], through the projection matrix M world2camera Project it into the camera coordinate system and get the camera coordinate loc of the center point in the camera coordinate system camer =[x, y, z, 1] is calculated as:

[0064] loc camera =M world2camera *loc world .

[0065] Projection matrix M world 2camera It can be obtained through the calibration and positioning data of the virtual vehicle in the scene.

[0066] The camera coordinate system is the local coordinate system of the camera, which is used to describe the position and direction of the camera. The optical center of the camera can be used as the origin, the optical axis of the camera as the height direction, the horizontal direction of the camera as the latitude direction, and the vertical direction of the camera as the longitude direction.

[0067] Step 212: Obtain camera distortion parameters involved in the camera, and transform the camera coordinates into a mean vector in the image coordinate system based on the camera intrinsic parameters and the camera distortion parameters.

[0068] According to the setting parameters of the fisheye camera model, the camera distortion parameters of the camera are obtained, and the camera coordinates loc in the camera coordinate system are converted according to the camera intrinsic parameters and camera distortion parameters. camer = [x, y, z, 1] converted to the plane mean vector loc in the two-dimensional image coordinate system p =[u,v].

[0069] Specifically, the formula for converting the camera coordinates into the mean vector in the image coordinate system is:

[0070]

[0071] Where u is the latitude coordinate of the mean vector, v is the longitude coordinate of the mean vector, [u, v] T represents the transpose of the matrix [u, v], C x is the latitude coordinate of the camera optical center of the fisheye camera model in the camera coordinate system, C y is the longitude coordinate of the camera optical center of the fisheye camera model in the camera coordinate system, f xis the focal length of the fisheye camera model in the latitude direction of the camera coordinate system, f y is the focal length of the fisheye camera model in the longitude direction in the camera coordinate system, h is the first function related to the camera distortion parameter, m is the first intermediate variable related to the camera coordinate, n is the second intermediate variable related to the camera coordinate, x is the latitude coordinate of the camera coordinate, and y is the longitude coordinate of the camera coordinate.

[0072] Specifically, the formula for calculating the first intermediate variable m related to the camera coordinates is:

[0073]

[0074] The formula for calculating the second intermediate variable n related to the camera coordinates is:

[0075]

[0076] Furthermore, the formula for calculating the first function h related to the camera distortion parameter is:

[0077] h=θ+k1θ 3 +k2θ 5 +k3θ 7 +k4θ 9

[0078] The third intermediate variable related to the camera coordinates is θ=a tan 2 (r, z).

[0079] z is the height coordinate of the camera coordinates, and k1, k2, k3, and k4 are the distortion coefficients of the fisheye camera model.

[0080] For example, when the camera coordinate loc in the camera coordinate system camer =[2, 3, 4, 1], the first intermediate variable:

[0081]

[0082] The first function related to camera distortion parameters:

[0083]

[0084] Then h, C x , C y 、f x and f y Substituting into the formula:

[0085]

[0086] Calculate [u, v] T , and further solve the latitude coordinate u of the mean vector:

[0087]

[0088] And the longitude coordinate v of the mean vector:

[0089]

[0090] Step 22: Calculate the covariance of the Gaussian ellipsoid in the image coordinate system.

[0091] The mean vector describes the center position of a Gaussian distribution. In the two-dimensional image coordinate system, the mean vector is represented by loc p = [u, v], where u is the latitude coordinate of the mean vector and v is the longitude coordinate of the mean vector. The covariance matrix Σ describes the shape and direction of the Gaussian distribution. The covariance matrix of the Gaussian ellipsoid in three-dimensional space is Σ, and the covariance matrix of the image coordinate system in two-dimensional space is Σ', where Σ' is a 2x2 matrix.

[0092] Furthermore, step 22 also includes:

[0093] Step 221: Establish the Jacobian matrix about the center point.

[0094] Based on the camera coordinates loc camer = [x, y, z, 1], establish the Jacobian matrix J about the center point. The Jacobian matrix J describes the local linear approximation of a function mapped from three-dimensional space to two-dimensional space near a certain point. J can be expressed as:

[0095]

[0096] Among them, du / dx is the derivative of the latitude coordinate u of the mean vector with respect to the latitude coordinate x of the camera coordinate, du / dz is the derivative of the latitude coordinate u of the mean vector with respect to the altitude coordinate z of the camera coordinate, dv / dy is the derivative of the longitude coordinate v of the mean vector with respect to the longitude coordinate y of the camera coordinate, and so on. J is composed of the derivatives of the camera coordinate parameters.

[0097] Further, step 221 includes:

[0098] Step 2211: Calculate the derivative of the center point in the image coordinate system with respect to the camera coordinate parameters.

[0099] Calculate the derivatives of the center point in the Jacobian matrix J with respect to the camera coordinate parameters in the image coordinate system, such as du / dx, du / dz, and dv / dy.

[0100] Specifically, the formula for calculating the derivative of the center point in the image coordinate system with respect to the camera coordinate parameters includes:

[0101]

[0102] or,

[0103] Among them, u is the latitude coordinate of the mean vector, a is the latitude coordinate or longitude coordinate of the camera coordinate, z is the height coordinate of the camera coordinate, h is the first function related to the camera distortion parameter, m is the first intermediate variable related to the camera coordinate, and f x It is the focal length of the fisheye camera model in the latitude direction in the camera coordinate system.

[0104] For ease of calculation, the formula can be further derived as:

[0105]

[0106] or,

[0107] Among them, θ is the third intermediate variable related to the camera coordinates, and q is the fourth intermediate variable related to the camera coordinates.

[0108] For example, when a is the latitude coordinate x of the camera coordinates:

[0109]

[0110] Or, when a is the longitude coordinate y of the camera coordinates:

[0111]

[0112] The formula for calculating the derivative of the center point in the image coordinate system with respect to the camera coordinate parameters also includes:

[0113]

[0114] or,

[0115] Among them, v is the longitude coordinate of the mean vector, a is the latitude coordinate or longitude coordinate of the camera coordinate, z is the height coordinate of the camera coordinate, h is the first function related to the camera distortion parameter, n is the second intermediate variable related to the camera coordinate, f y It is the focal length of the fisheye camera model in the longitude direction in the camera coordinate system.

[0116] For example, when a is the latitude coordinate x of the camera coordinates:

[0117]

[0118] Or, when a is the longitude coordinate y of the camera coordinates:

[0119]

[0120] In the derivative formula of the longitude coordinate v of the mean vector, the part dh / da or dh / dz is also applicable to the above formula expansion calculation method.

[0121] Specifically, the first intermediate variable m related to the camera coordinates can be obtained by the formula in step 212:

[0122]

[0123] The second intermediate variable n related to the camera coordinates can be obtained by the formula in step 212:

[0124]

[0125] The first function h related to the camera distortion parameter and the third intermediate variable θ related to the camera coordinate are also the same as the algorithm in step 212 .

[0126] The derivative term of the fourth intermediate variable q about the coordinate parameter of the mean vector and The formula can be:

[0127]

[0128] And the formula:

[0129]

[0130] Obtain, where a is x or y correspondingly.

[0131] The derivatives du / dx, du / dy, du / dz, dv / dx, dv / dy and dv / dz for constructing the Jacobian matrix can be obtained in the above manner.

[0132] Step 2212: Construct the Jacobian matrix using the derivatives as matrix parameters.

[0133] The Jacobian matrix is ​​constructed based on the derivatives du / dx, du / dy, du / dz, dv / dx, dv / dy and dv / dz obtained in step 2211:

[0134]

[0135] To calculate the covariance matrix Σ' of the Gaussian ellipsoid with covariance matrix Σ in three-dimensional space in the image coordinate system in two-dimensional case.

[0136] Step 222: Obtain the covariance of the Gaussian ellipsoid in the camera coordinate system, and calculate the covariance of the Gaussian ellipsoid in the image coordinate system through the covariance in the camera coordinate system and the Jacobian matrix.

[0137] Get the covariance Σ of the Gaussian ellipsoid in real space and use the formula:

[0138]

[0139] Compute the covariance Σ' of the Gaussian ellipsoid in image coordinates.

[0140] Among them, Σ' is the covariance matrix of the Gaussian ellipsoid in the image coordinate system, Σ is the covariance matrix of the Gaussian ellipsoid in the camera coordinate system, J is the Jacobian matrix about the center point, M world 2camera is the projection matrix.

[0141] Step 24: The differentiable renderer performs image rendering based on the mean vector and the covariance to obtain a fisheye camera rendered image under the camera pose.

[0142] The Gaussian ellipsoid and camera pose are passed as input to the differentiable renderer, which obtains background information such as lighting conditions and material properties such as color and reflectivity for rendering to generate a two-dimensional image under the camera pose.

[0143] A differentiable renderer is a rendering process that can be differentiated. Its forward process is the same as traditional rendering, where the camera pose and Gaussian ellipsoid are input to obtain an image; the reverse process is to take the derivatives of pixels with respect to scene parameters, which enables the differentiable renderer to simultaneously obtain the rendering result and the derivative of the rendering result with respect to the input.

[0144] Step 30: Calculate the difference between the rendered image and the corresponding real image, and optimize the initial properties of the Gaussian ellipsoid based on the difference.

[0145] The real image is obtained by shooting with a real scene camera, and the difference between the rendered image and the corresponding real image is calculated, such as mean square error, structural similarity or perceptual loss, to quantify the mismatch between the rendered image and the real image. The gradient of the loss function is back-propagated to the initial attributes of the Gaussian ellipsoid using the differential characteristics of the differentiable renderer to optimize the initial attributes.

[0146] The real image is an image in real space with the same camera pose as the rendered image. The real image and the rendered image should be compared under the same viewing angle and lighting conditions.

[0147] The error of the rendered image is corrected through the real image, and the initial parameters of the Gaussian ellipsoid are updated to reconstruct the entire real space scene in the virtual space, so as to render an image that is closer to the real one and synthesize images from a new perspective.

[0148] Furthermore, step 30 also includes:

[0149] Step 31: Using the real image as a supervisory signal, calculate the difference between the rendered image and the real image.

[0150] The difference between the rendered image and the corresponding real image is calculated, such as mean square error, structural similarity or perceptual loss, to quantify the degree of mismatch between the rendered image and the real image, and a loss is generated as a guide for the initial attribute optimization direction.

[0151] Step 32: Calculate the gradient of the loss function with respect to the initial parameters, and update the initial parameters using a gradient descent algorithm until the loss function is less than a preset threshold.

[0152] The gradient of the loss function with respect to the Gaussian ellipsoid is calculated using the differential properties of the differentiable renderer. The gradient of the loss function is propagated through the back-propagation algorithm from the rendered image to the initial properties of the Gaussian ellipsoid. The initial properties of the Gaussian ellipsoid are updated using the gradient descent algorithm to minimize the loss function. The initial properties of the Gaussian ellipsoid are adjusted iteratively, and the parameter values ​​are updated according to the calculated gradients at each iteration until the difference between the rendered image and the real image is small enough, that is, the loss function converges below the preset threshold, and the optimization process ends.

[0153] Optionally, after step 30, the method further includes: inputting the camera posture into a differentiable renderer, and the differentiable renderer performs image rendering based on the optimized Gaussian ellipsoid to obtain a fisheye camera composite image corresponding to the camera posture.

[0154] After the initial properties of the Gaussian ellipsoid are optimized to the best, the camera pose and the optimized pose are input into the differentiable renderer, which renders the image based on the three-dimensional space constructed by the optimized Gaussian ellipsoid to obtain the fisheye camera synthetic image under the camera pose. The three-dimensional space constructed by the optimized Gaussian ellipsoid can synthesize a fisheye camera synthetic image with a new perspective, which is more suitable for autonomous driving scenarios.

[0155] The camera pose may be a new perspective that has not been trained.

[0156] By using the optimized initial attributes for image rendering in a differentiable renderer, an image with a new perspective can be synthesized according to a given camera pose, and the output of the differentiable renderer is a fisheye camera image.

[0157] By converting points in real space to the image plane through a Gaussian ellipsoid that obeys a three-dimensional Gaussian distribution, rendering and optimizing them in the form of fisheye camera images, the three-dimensional Gaussian splashing method can process fisheye camera images, and optimize the rendering process through back propagation to render images that are closer to reality.

[0158] See also Figure 7 , Figure 7 It is a structural diagram of an embodiment of the storage medium provided by the present application.

[0159] The storage medium 300 stores program data 301. When the program data 301 is executed by the processor, the following is achieved: Figure 1 The steps of the image rendering method are described.

[0160] The program data 301 is stored in a storage medium 300, and includes a number of instructions for enabling a network device (such as a router, a personal computer, a server, or other network device) or a processor to execute all or part of the steps of the method described in each embodiment of the present application.

[0161] Optionally, the storage medium 300 may be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, or other medium that can store program data.

[0162] See also Figure 8 , Figure 8 It is a structural schematic diagram of an embodiment of a computer device provided by the present application.

[0163] The computer device 400 includes a processor 420 and a memory 410 which are connected to each other. The memory 410 stores a computer program. When the processor 420 executes the computer program, the steps of the above-mentioned image rendering method are implemented.

[0164] Different from the prior art, the present application converts the points in the three-dimensional space into data that obeys the three-dimensional Gaussian distribution in the three-dimensional space, namely, a Gaussian ellipsoid, and projects the Gaussian ellipsoid in the three-dimensional space onto a two-dimensional plane for calculation through coordinate system conversion and calculation, so as to obtain a fisheye camera image rendered by a differentiable renderer under a specific camera posture. The three-dimensional space is optimized based on the real space to restore the picture of the real space, so that fisheye camera images of new perspectives can also be synthesized. The Gaussian ellipsoid in the three-dimensional space is projected onto a two-dimensional plane for calculation, so that the entire three-dimensional Gaussian splashing process can use the real image taken by the fisheye camera as the training input, and use the fisheye camera rendered image and the fisheye camera synthesized image as the output, which is more suitable for autonomous driving scenarios.

[0165] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the storage medium embodiment and the computer device embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0166] The present application can be used in many general or special computing system environments or configurations, such as personal computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, network PCs, minicomputers, and distributed computing environments including any of the above systems or devices.

[0167] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed.

[0168] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0169] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image rendering method based on a fisheye camera model, characterized in that: include: Constructing a Gaussian ellipsoid that obeys a three-dimensional Gaussian distribution based on a three-dimensional space, wherein the Gaussian ellipsoid is constructed with a center point having initial attributes as the center; The camera pose of the fisheye camera model and the initial properties of the Gaussian ellipsoid are used as inputs of a differentiable renderer, and the differentiable renderer performs image rendering based on the input camera pose and the initial properties of the Gaussian ellipsoid to obtain a fisheye camera rendered image under the camera pose; the differentiable renderer is built based on the fisheye camera imaging model; The difference between the rendered image and a corresponding real image is calculated, and initial properties of the Gaussian ellipsoid are optimized based on the difference, wherein the real image is taken by a real camera in real space at the same camera pose as the rendered image.

2. The image rendering method according to claim 1, characterized in that: The differentiable renderer performs image rendering based on the input camera pose and initial properties of the Gaussian ellipsoid to obtain a fisheye camera rendered image under the camera pose, including: Calculate the mean vector of the center point of the Gaussian ellipsoid in the image coordinate system; Calculating the covariance of the Gaussian ellipsoid in the image coordinate system; The differentiable renderer performs image rendering based on the mean vector and the covariance to obtain a fisheye camera rendered image under the camera posture.

3. The image rendering method according to claim 2, characterized in that: The step of calculating the mean vector of the center point of the Gaussian ellipsoid in the image coordinate system includes: Projecting the center point of the Gaussian ellipsoid in the three-dimensional space into a camera coordinate system through a projection matrix to obtain the camera coordinates of the center point of the Gaussian ellipsoid in the camera coordinate system; Obtain camera distortion parameters involved in the camera, and transform the camera coordinates into a mean vector in an image coordinate system based on the camera intrinsic parameters and the camera distortion parameters.

4. The image rendering method according to claim 3, characterized in that: The formula for converting the camera coordinates into the mean vector in the image coordinate system is: Where u is the latitude coordinate of the mean vector, v is the longitude coordinate of the mean vector, [u, v] T represents the transpose of the matrix [u, v], C x is the latitude coordinate of the camera optical center of the fisheye camera model in the camera coordinate system, C y is the longitude coordinate of the camera optical center of the fisheye camera model in the camera coordinate system, x is the focal length of the fisheye camera model in the latitude direction in the camera coordinate system, f y is the focal length of the fisheye camera model in the longitude direction of the camera coordinate system, h is the first function related to the camera distortion parameter, m is the first intermediate variable related to the camera coordinate, n is the second intermediate variable related to the camera coordinate, x is the latitude coordinate of the camera coordinate, and y is the longitude coordinate of the camera coordinate.

5. The image rendering method according to claim 3, characterized in that: The calculating the covariance of the Gaussian ellipsoid in the image coordinate system comprises: Establishing a Jacobian matrix about the center point; The covariance of the Gaussian ellipsoid in the camera coordinate system is obtained, and the covariance of the Gaussian ellipsoid in the image coordinate system is calculated by using the covariance in the camera coordinate system and the Jacobian matrix.

6. The image rendering method according to claim 5, characterized in that: The method is based on establishing a Jacobian matrix about the center point, comprising: Calculating the derivative of the center point in the image coordinate system with respect to the camera coordinate parameter; The Jacobian matrix is ​​constructed using the derivatives as matrix parameters.

7. The image rendering method according to claim 6, characterized in that: The formula for calculating the derivative of the center point in the image coordinate system with respect to the camera coordinate parameter includes: or, Wherein, u is the latitude coordinate of the mean vector, a is the latitude coordinate or longitude coordinate of the camera coordinate, z is the height coordinate of the camera coordinate, h is the first function related to the camera distortion parameter, m is the first intermediate variable related to the camera coordinate, and f x is the focal length of the fisheye camera model in the latitude direction in the camera coordinate system.

8. The image rendering method according to claim 6, characterized in that: The formula for calculating the derivative of the center point in the image coordinate system with respect to the camera coordinate parameter also includes: or, Wherein, v is the longitude coordinate of the mean vector, a is the latitude coordinate or longitude coordinate of the camera coordinate, z is the height coordinate of the camera coordinate, h is the first function related to the camera distortion parameter, n is the second intermediate variable related to the camera coordinate, f y is the focal length of the fisheye camera model in the longitude direction in the camera coordinate system.

9. The image rendering method according to claim 1, characterized in that: The calculating the difference between the rendered image and the corresponding real image, and optimizing the initial properties of the Gaussian ellipsoid based on the difference, comprises: Using the real image as a supervisory signal, calculating the difference between the rendered image and the real image; The gradient of the loss function with respect to the initial parameters is calculated, and the initial parameters are updated by a gradient descent algorithm until the loss function is less than a preset threshold.

10. The image rendering method according to claim 1, characterized in that: After optimizing the initial properties of the Gaussian ellipsoid based on the difference, the method further includes: The camera pose is input into the differentiable renderer, and the differentiable renderer performs image rendering based on the optimized Gaussian ellipsoid to obtain a fisheye camera composite image corresponding to the camera pose.

11. A storage medium having program data stored thereon, characterized in that: When the program data is executed by a processor, the steps of the image rendering method according to any one of claims 1 to 10 are implemented.

12. A computer device, characterized in that: Includes interconnected processors and storage The memory stores a computer program, and when the processor executes the computer program, Implement the steps of the image rendering method as described in any one of claims 1-10.

Images