Single-view-based multi-view high-fidelity image generation method, system and device for vehicle and storage medium

Abstract

The present application relates to the technical field of automatic driving, and relates to a single-view-based vehicle multi-view high-fidelity image generation method, which comprises the following steps: acquiring a vehicle image and camera parameter information corresponding to a camera for shooting the vehicle image; constructing a network structure of an implicit expression model; extracting features of the vehicle image to obtain shape features and texture features; constructing a normalized coordinate system with a geometric center of the vehicle as an origin, and determining normalized parameter information of the camera in the normalized coordinate system; determining network parameters of the implicit expression model network according to the shape features, the texture features and the normalized parameter information, to obtain a three-dimensional implicit expression model of the vehicle; and obtaining a view image of the vehicle at a given view angle according to the three-dimensional implicit expression model. The present application further provides a single-view-based vehicle multi-view high-fidelity image generation system, a device and a storage medium. The present application can reduce the possibility that the reconstruction of a three-dimensional implicit expression model of a vehicle is affected due to the limitation of a vehicle image acquisition angle.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of automatic driving technology, in particular to a single-view-based vehicle multi-view high-fidelity image generation method, system, device and storage medium. BACKGROUND

[0002] In the field of autonomous driving, vehicle reconstruction is a key technical link. Vehicle reconstruction methods obtain perception data through various sensors such as radar, LiDAR, cameras, etc., and use computer vision, machine learning and deep learning technologies to extract, understand and reconstruct the information of surrounding vehicles, so that the autonomous driving system can accurately perceive and predict the behavior of other vehicles.

[0003] Currently, the NeRF technology generally processes driving scene images and camera parameters to predict the three-dimensional model of the driving scene; the full name of NeRF in English is Neural Radiance Fields, which means neural radiation field in Chinese; NeRF is an advanced computer graphics technology that reconstructs complex vehicle three-dimensional scenes from a set of partial two-dimensional images through deep learning technology. The core of NeRF is to learn the geometry, objects and angles of a specific scene through training a neural network, so as to be able to render a 3D view of the vehicle in the scene from a new perspective; however, the existing vehicle reconstruction method based on NeRF mainly reconstructs the vehicle through multi-view or few-view, but due to the limitation of the angle of view of the autonomous driving dataset, it is usually impossible to obtain multi-angle views of each vehicle in the scene in the real scene, so that the existing technology has many limitations in predicting vehicle multi-view images. SUMMARY

[0004] In order to reduce the possibility of affecting the reconstruction of the vehicle three-dimensional implicit representation model due to the limitation of the vehicle image collection angle, and improve the convenience of predicting vehicle multi-view images, a single-view-based vehicle multi-view high-fidelity image generation method, system, device and storage medium are provided.

[0005] In order to achieve the above-mentioned purpose of the present application, the present application provides a single-view-based vehicle multi-view high-fidelity image generation method.

[0006] The single-view-based vehicle multi-view high-fidelity image generation method provided by the present application comprises:

[0007] Obtaining vehicle image and camera parameter information corresponding to the camera shooting the vehicle image;

[0008] Constructing a network structure of an implicit representation model;

[0009] Extracting features of the vehicle image to obtain shape features and texture features;

[0010] The vehicle geometric center is determined according to the shape feature, a normalized coordinate system is constructed with the vehicle geometric center as the origin, and the normalized parameter information of the camera in the normalized coordinate system is determined according to the camera parameter information;

[0011] The network parameters of the implicit expression model network are determined according to the shape feature, the texture feature and the normalized parameter information, and a three-dimensional implicit expression model of the vehicle is obtained.

[0012] The view image of the vehicle at a given view angle is obtained according to the three-dimensional implicit expression model.

[0013] Optionally, the implicit expression model network comprises a shape component and a texture component, the shape component is used to represent the body density information of the vehicle, and the texture component is used to represent the color information of the vehicle; the network parameters of the implicit expression model network comprise shape component weight parameters and texture component weight parameters.

[0014] The network parameters of the implicit expression model network are determined according to the shape feature, the texture feature and the normalized parameter information, and a three-dimensional implicit expression model of the vehicle is obtained.

[0015] The vehicle three-dimensional scene is sampled through the camera rays;

[0016] The sampling points used for training the implicit expression model network are randomly selected from the camera rays;

[0017] The three-dimensional coordinates and the visual directions of the sampling points are determined according to the normalized parameter information;

[0018] The three-dimensional coordinates and the view directions are respectively processed in a format, and the three-dimensional coordinates and the view directions processed in the format are input into the implicit expression model network, and the implicit expression model network outputs the vehicle three-dimensional scene body density prediction result and the vehicle three-dimensional scene projection color prediction result;

[0019] The shape component weight parameters and the texture component weight parameters of the three-dimensional implicit expression model are optimized according to the vehicle three-dimensional scene body density prediction result and the vehicle three-dimensional scene color prediction result, and the final three-dimensional implicit expression model is obtained.

[0020] Optionally, the method further comprises:

[0021] The camera light is modeled as a cone;

[0022] The three-dimensional coordinates of the sampling points are integrated position coded to obtain a position integrated position coding result, and the shape component obtains the vehicle three-dimensional scene body density prediction result according to the position integrated position coding result and the shape feature;

[0023] The visual direction of the sampling point is integrated position coded to obtain a direction integrated position coding result, and the texture component obtains a vehicle three-dimensional scene projection color prediction result according to the direction integrated position coding result, the vehicle three-dimensional scene body density prediction result and a texture feature.

[0024] Optionally, the position of the camera light cone in the direction d is represented as a multivariate Gaussian distribution with a mean μ (o, d) and a variance ∑ (d), o is the cone vertex, and d represents the light direction;

[0025] The output feature expression of the shape component is

[0026] f out ,σ=f θ,shape (γ pos (μ(o,d),∑(d))+z shape )

[0027] Wherein, σ represents the density, f θ,shape is a shape component weight parameter, γ pos (.) represents position integrated position coding, and z shape represents a shape feature;

[0028] The output feature expression of the texture component is

[0029] f out ,c=f θ,texture (f out ,σ+γ dir (d)+z texture )

[0030] Wherein, c represents the color, f θ,texture is a texture component weight parameter, γ dir (d) represents direction integrated position coding, and z texture represents a texture feature.

[0031] Optionally, the camera parameter information includes camera pose information and lens orientation information;

[0032] The vehicle geometric center is determined according to the shape feature, a normalized coordinate system is constructed with the vehicle geometric center as the origin, and the normalized parameter information of the camera in the normalized coordinate system is determined according to the camera parameter information, comprising:

[0033] The geometric center of the vehicle and the current vehicle size information are determined according to the shape feature;

[0034] A normalized coordinate system is constructed with the vehicle geometric center as the origin, the normalized parameter information of the camera in the normalized vehicle coordinate is determined based on the camera pose information and the lens orientation information, and the normalized camera pose is

[0035]

[0036] wherein, represents the camera coordinates in the three-dimensional coordinate system of the car, l represents the length of the vehicle body, h represents the height of the vehicle, and w represents the width of the vehicle body, represents the Hadamard product, I 4×4 represents the unit matrix.

[0037] Optionally, after determining the geometric center of the vehicle and the current vehicle size information according to the shape features, before establishing the normalized coordinate system, the method further comprises:

[0038] adjusting the vehicle size with the camera geometric center as the origin, and adjusting the vehicle size to a preset spatial range; the camera pose relative to the normalized coordinate system after size adjustment is

[0039]

[0040] is the camera pose relative to the normalized coordinate system before size adjustment, is the pose of the vehicle relative to the camera, is a scaling matrix for scaling the vehicle size with the camera geometric center as the origin.

[0041] Optionally, the three-dimensional implicit expression model is used to obtain a view image of the vehicle at a given view angle, comprising:

[0042] sampling light rays emitted from the given view angle, and using the three-dimensional implicit expression model to predict the color and volume density of each sampling point;

[0043] combining the color and volume density of the sampling points on the light rays emitted from the given view angle using volume rendering technology to generate a final image.

[0044] In order to solve the above problems, the application further provides a single-view-based vehicle multi-view high-fidelity image generation system, which comprises:

[0045] an acquisition module, configured to acquire a vehicle image and camera parameter information corresponding to a camera for shooting the vehicle image;

[0046] a construction module, configured to construct a network structure of an implicit expression model;

[0047] a feature extraction module, configured to extract features of the vehicle image to obtain shape features and texture features;

[0048] The camera parameter normalization module is configured to determine a vehicle geometric center according to the shape feature, construct a normalized coordinate system with the vehicle geometric center as the origin, and determine normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information.

[0049] The model generation module is configured to determine network parameters of the implicit expression model network according to the shape feature, the texture feature and the normalized parameter information, and obtain a three-dimensional implicit expression model of the vehicle.

[0050] The view image generation module is configured to obtain a view image of the vehicle at a given view angle according to the three-dimensional implicit expression model.

[0051] To solve the above problems, the present application further provides an electronic device, which comprises:

[0052] at least one processor; and

[0053] a memory connected in communication with the at least one processor; wherein

[0054] the memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the single-view-based vehicle multi-view high-fidelity image generation method described above.

[0055] To solve the above problems, the present application further provides a computer readable storage medium, which stores at least one computer program, and the at least one computer program is executed by a processor in an electronic device to implement the single-view-based vehicle multi-view high-fidelity image generation method described above.

[0056] In summary, the present application has the following beneficial technical effects:

[0057] To learn the 3D shape of the vehicle from a single vehicle image supervised implicit expression model network, the present application constructs a normalized coordinate system according to the shape feature, and converts the camera coordinate system into a normalized coordinate system with the vehicle geometric center as the origin and aligned along the main axis, so that the implicit expression model network can efficiently learn and generate a three-dimensional implicit expression model of the vehicle at a 3D view according to the shape feature, the texture feature and the normalized parameter information, and reduce the possibility of affecting the reconstruction of the three-dimensional implicit expression model of the vehicle due to the limitation of the vehicle image acquisition angle; the body rendering method is used to obtain multiple view images based on the three-dimensional implicit expression model of the vehicle, and the convenience of vehicle view image prediction is improved. BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 A flowchart of the single-view-based vehicle multi-view high-fidelity image generation method according to an embodiment of the present application is shown.

[0059] Figure 2 A vehicle multi-view high-fidelity image generation model architecture diagram provided by an embodiment of the present application;

[0060] Figure 3 A schematic diagram of the relationship between the camera and the normalized vehicle coordinate system provided by an embodiment of the present application;

[0061] Figure 4 A schematic diagram of the acquired vehicle image provided by an embodiment of the present application;

[0062] Figure 5 A new view image obtained after the vehicle spatial position is transformed based on the vehicle three-dimensional implicit expression model provided by an embodiment of the present application;

[0063] Figure 6 A structural schematic diagram of an electronic device for implementing the vehicle multi-view high-fidelity image generation method based on a single view provided by an embodiment of the present application.

[0064] The reference signs: 10, processor; 11, memory; 12, communication bus; 13, communication interface.

[0065] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0066] The embodiments of the present application will be described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference signs represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application.

[0067] In the description of the present application, it should be understood that the terms "longitudinal", "transverse", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and simplify the description, and therefore cannot be understood as indicating or implying that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation of the present application.

[0068] In the description of the present application, unless otherwise specified and limited, it should be noted that the terms "mounting", "connecting", "connecting" should be understood in a broad sense, for example, it can be mechanical connection or electrical connection, or the communication between two elements, or direct connection, or indirect connection through an intermediate medium, and the specific meaning of the above terms can be understood by those skilled in the art according to the specific circumstances.

[0069] Referring to Figure 1 Fig. 1 is a flowchart of a single-view-based vehicle multi-view high-fidelity image generation method according to an embodiment of the present application. In this embodiment, the single-view-based vehicle multi-view high-fidelity image generation method comprises the following steps.

[0070] S1, obtaining camera parameter information corresponding to a vehicle image and a camera for shooting the vehicle image.

[0071] The camera parameter information includes camera pose information, lens orientation information, focal length, and intrinsic matrix, etc. The camera parameter information can be obtained by a camera calibration tool (such as COLMAP), or other tools can be used to determine the camera parameter information, which is not limited in the present application.

[0072] S2, constructing a network structure of an implicit representation model.

[0073] The network structure of the implicit representation model is a neural network (such as a multi-layer perceptron MLP), which is trained to learn a continuous function mapping of data to obtain a three-dimensional implicit representation model of the vehicle. Taking an image as an example, each pixel point can be represented by its position and color value. The neural network (such as a multi-layer perceptron MLP) is trained to predict the color value according to the pixel position. The trained implicit representation model can not only encode the data seen in the training process, but also predict the pixel color value at any position, including those positions that do not directly appear in the training data, thereby realizing the subsequent prediction of the vehicle multi-view image.

[0074] S3, extracting features of the vehicle image to obtain shape features and texture features.

[0075] Referring to Figure 1 and Figure 2 , the shape features are used to guide the implicit representation network to obtain the volume density occupation of the 3D point, and the appearance features are used to guide the implicit representation network to obtain the RGB color of the 3D point under a given observation direction.

[0076] ResNet, VGG16, etc. feature extraction network can be used for feature extraction of vehicle image, the full name of ResNet in English is Residual Networks, and the Chinese interpretation is "residual network". ResNet is a deep convolutional neural network architecture proposed by Microsoft Research, and the core feature of ResNet is to introduce residual connection, which solves the gradient disappearance and degradation problem in deep network, so that the network can be trained deeper; the full name of VGG16 in English is Visual Geometry Group16layers, and the Chinese interpretation is "visual geometry group 16 layers"; the feature of VGG16 is to use a large number of small size 3x3 convolution kernel and 2x2 maximum pooling layer, which reduces the number of parameters and improves the calculation efficiency. In this embodiment, ResNet-34 feature extraction network is used to extract shape features and texture features from vehicle image.

[0077] S4, determining the vehicle geometric center according to the shape feature, constructing a normalized coordinate system with the vehicle geometric center as the origin, and determining the normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information.

[0078] Reference Figure 3 , according to the shape feature, the vehicle geometric center is determined, a normalized coordinate system is constructed with the vehicle geometric center as the origin, and the normalized parameter information of the camera in the normalized coordinate system is determined according to the camera parameter information, comprising:

[0079] S41, determining the geometric center of the vehicle and the current vehicle size information according to the shape feature.

[0080] S42, constructing a normalized coordinate system with the vehicle geometric center as the origin, and determining the normalized parameter information of the camera in the normalized vehicle coordinate based on the camera pose information and the lens orientation information.

[0081] The normalized camera coordinate which can be represented by the following formula:

[0082]

[0083] wherein, represents the camera coordinate in the automobile three-dimensional coordinate system, l represents the length of the vehicle body, h represents the height of the vehicle, and w represents the width of the vehicle body, is Hadamard product, Hadamard product is a kind of operation mode of matrix, and the specific operation of is to multiply each item; I 4×4 represents unit matrix, and the size (length l, height h, width w) of the automobile is obtained from the result of 3D detection module.

[0084] Preferably, after determining the geometric center of the vehicle and the current vehicle size information according to the shape features, before establishing the normalized coordinate system, the single-view-based multi-view high-fidelity image generation method of the vehicle further comprises:

[0085] Adjusting the vehicle size with the camera geometric center as the origin, adjusting the vehicle size to a preset spatial range; the camera pose relative to the normalized coordinate system after size adjustment For

[0086]

[0087] For the camera pose relative to the normalized coordinate system before size adjustment, For the pose of the vehicle relative to the camera, For the scaling ratio matrix for scaling the vehicle size with the camera geometric center as the origin.

[0088] Referring to Figure 3 , in a popular way, the camera and the vehicle have a pose in the world coordinate system, the coordinate system constructed with the camera lens focal length as the origin is the camera coordinate system, and the coordinate system with the vehicle geometric center as the origin is the vehicle coordinate system. The vehicle coordinate system takes the vehicle geometric center as the origin, the x-axis is the vehicle head direction, the y-axis is upward, and the z-axis is the right side of the vehicle body. Since a single view is used for model training, a normalized coordinate is constructed with the vehicle geometric center as the origin, and a standardized processing of the camera parameter information is performed by using a normalized operation. Since the vehicle sizes are inconsistent, a size transformation is needed. Based on the vehicle size, the transformation is performed to a square space of -1 to 1. In this embodiment, indicates a scaling matrix for mapping the normalized coordinate system to the camera center scaling coordinate system, The diagonal values of are (1, -1, -1, 1), and scaling is performed to adapt to the size of the vehicle.

[0089] S5, determining the network parameters of the implicit expression model network according to the shape features, the texture features and the normalized parameter information, and obtaining a three-dimensional implicit expression model of the vehicle.

[0090] Referring to Figure 2 , the implicit expression model network includes a shape component and a texture component. The shape component is used to represent the body density information of the vehicle, and the texture component is used to represent the color information of the vehicle. The network parameters of the implicit expression model network include shape component weight parameters and texture component weight parameters.

[0091] In order to improve the network representation ability, the hypernetwork is introduced into the implicit representation model, the hypernetwork can generate the network parameters of the implicit representation model network according to the features of the input image, so that the implicit representation model can adaptively generate a suitable view synthesis model for different scenes, and the implicit representation model network converts the feature vector obtained by the resnet34 into the network parameters of the implicit representation model network. Mathematically, the hypernetwork can be expressed as: Ψ:R k →R l , where R k represents the dimension of the latent code k, and R l represents the dimension of the network parameter space of the implicit representation model network l, and '→' represents the conversion operation.

[0092]

[0093] z represents the feature vector, Ψ(z) represents the network parameters of the implicit representation model network, θ shape represents the shape component weight parameter, θ texture represents the texture component weight parameter. represents that the feature vector is input into the hypernetwork to obtain the shape component weight parameter and the texture component weight parameter.

[0094] Specifically, the network parameters of the implicit representation model network are determined according to the shape feature, the texture feature and the normalized parameter information, including:

[0095] S51, sampling the vehicle three-dimensional scene through the camera ray.

[0096] Referring to Figure 3 , Figure 3 the blue dashed line in the figure is a schematic sampling light obtained by sampling the vehicle three-dimensional scene through the camera ray, the light is emitted from the transformed coordinate system to the vehicle, and the image corresponding to the view angle of the camera is generated after selecting the sampling point.

[0097] S52, randomly selecting a sampling point for training the implicit representation model network from the camera ray.

[0098] S53, determining the three-dimensional coordinates and the visual direction of the sampling point according to the normalized parameter information.

[0099] S54, respectively processing the three-dimensional coordinates and the visual direction, inputting the processed three-dimensional coordinates and the visual direction into the implicit representation model network, and the implicit representation model network outputs the vehicle three-dimensional scene body density prediction result and the vehicle three-dimensional scene projection color prediction result.

[0100] S55, according to the vehicle three-dimensional scene body density prediction result and the vehicle three-dimensional scene color prediction result, optimizing the shape component weight parameter and the texture component weight parameter of the three-dimensional implicit expression model, and obtaining a final three-dimensional implicit expression model.

[0101] In the model architecture, due to the large variation in the size of the input image patches, the area represented by the pixels also varies significantly. Therefore, instead of ray tracing, conic tracing is used to achieve continuous scaling, and integrated positional encoding (IPE) is used for anti-aliasing. Anti-aliasing is a technique that softens the jagged edges of triangles in a rendered image, making it smoother and more realistic. When anti-aliasing is enabled, the edges of the image appear smoother and closer to the real object, resulting in better image quality.

[0102] Reference Figure 2 In the preferred embodiment of the present embodiment, the camera rays are modeled as cones; through conic tracing, the size of the cone can be adjusted at different levels (i.e., scaling the cone), thereby controlling the range and accuracy of sampling. In the initial stage, a larger cone is used for rough sampling to quickly obtain a rough representation of the scene. Then, the size of the cone is gradually reduced for more detailed sampling to improve the quality and details of the rendering.

[0103] Conic tracing can improve the robustness and efficiency of rendering, especially when dealing with reflections, refractions, or dynamic scenes; by sampling in a wider area, the details of the scene can be better captured, while reducing rendering artifacts caused by individual sampling point errors.

[0104] The goal of IPE (Integrated Positional Encoding) is to use a multivariate Gaussian distribution to represent the continuous distribution of rays or cones in an image and encode this distribution. IPE uses this distribution to replace the point positions in traditional positional encoding, improving the anti-aliasing ability and multi-scale processing ability of objects.

[0105] Suppose we perform a perfect sampling of a cone with vertex o and direction d, where d is the direction of the line containing the height of the cone in the normalized coordinate system. The position of the cone with vertex o and direction d in direction d can be represented as a multivariate Gaussian distribution with mean μ(o, d) and variance Σ(d). At this point, the formula for integrated positional encoding is as follows:

[0106]

[0107] Where:

[0108] μ(o, d) represents the mean of the cone in direction d (similar to position), o is the vertex of the cone, and d is the direction of the line containing the height of the cone in the normalized coordinate system.

[0109] ∑(d) represents the variance in the direction, ∑(d) describes the expansion of the cone;

[0110] k is a set of basis vectors, usually used for Fourier transform (also can be sine cosine transform), k T is the transpose of k;

[0111] represents the influence of variance on encoding, used to control the position encoding at different scales;

[0112] sin(k T μ(o,d)) represents the sine transform of the mean, cos(k T μ(o,d)) represents the cosine transform of the mean.

[0113] In the NeRF model, not only the position is usually encoded, but also the direction of the light is encoded. Therefore, the IPE can be divided into two parts: the position integrated position encoding result and the direction integrated position encoding result.

[0114] The three-dimensional coordinates of the sampling points are integrated position encoded to obtain the position integrated position encoding result, and the position integrated position encoding is the encoding of the mean μ(o,d) and the variance ∑(d). Specifically, the position integrated position encoding result corresponding to the cone with vertex o and direction d is denoted as γ pos (o,d), γ pos (o,d) = IPE(μ(o,d),∑(d)), IPE(.) is the integrated position encoding; the shape component obtains the vehicle three-dimensional scene body density prediction result according to the integrated position encoding result and the shape feature;

[0115] The visual direction of the sampling points is integrated position encoded to obtain the direction integrated position encoding result, and the integrated position encoding of the direction is the Fourier encoding of the direction. Specifically, the direction integrated position encoding result corresponding to the cone with vertex o and direction d is denoted as γ dir (d), γ dir (d) = IPE(d).

[0116] The texture component obtains the vehicle three-dimensional scene projection color prediction result according to the direction integrated position encoding result, the vehicle three-dimensional scene body density prediction result and the texture feature.

[0117] The position of the cone corresponding to the camera light in the direction d is represented as a multivariate Gaussian distribution of the mean μ(o,d) and the variance ∑(d), o is the vertex of the cone, and d represents the direction of the light;

[0118] The output feature expression of the shape component is

[0119] fout ,σ = f θ,shape (γpos(μ(o,d),∑(d))+z shape )

[0120] where σ denotes density, f θ,shape is a shape component weight parameter, γ pos (.) denotes position integrated position encoding, z shape denotes shape features;

[0121] The output feature expression of the texture component is

[0122] f out ,c = f θ,texture (fout,σ+γ dir (d)+z texture )

[0123] where c denotes color, f θ,texture is a texture component weight parameter, γ dir (d) denotes direction integrated position encoding, z texture denotes texture features.

[0124] S6, obtaining a perspective image of the vehicle at a given perspective according to the three-dimensional implicit representation model.

[0125] The three-dimensional implicit representation model has the ability to encode and decode instance categories, can encode input vehicle images at the instance level, generate shape codes and appearance codes of the corresponding vehicle, and finally input the queried 3D points and observation direction into the subsequent decoder to obtain the rendered vehicle image in the real scene. Considering the different sizes of actual vehicles, in order to meet the model requirements of the decoder training, facilitate 3D point generation and query, the implicit representation of the vehicle is normalized to a cube space of -1 to 1, so as to learn an implicit representation model covering various vehicle categories.

[0126] Specifically, step S6 includes:

[0127] S61, sampling light rays emitted from a given perspective, and predicting the color and volume density of each sampling point using the three-dimensional implicit representation model;

[0128] S62, combining the color and volume density of the sampling points on the light rays emitted from the given perspective using volume rendering technology to generate a final image.

[0129] The volume rendering sampling method also adopts the general sampling strategy as the implicit representation model. The volume rendering sampling step is divided into two steps: firstly, coarse sampling is performed, the coarse sampling uses equidistant sampling method, and then fine sampling is performed, the fine sampling is calculated through the following cumulative weight formula, and then the probability density function (PDF) is obtained through the weight distribution, and then converted into its cumulative distribution (CDF) function, the fine sampling points are obtained through the inverse transform sampling method, and finally the volume rendering is performed combined with the coarse sampling points.

[0130] The cumulative weight ω(r) from the near distance t n to the far distance t f on the cone axis is calculated by the following formula:

[0131]

[0132] ω(r) is the cumulative transmission weight, which is used to describe the color contribution or energy transfer of the light ray on the path r. It represents the contribution of the light ray passing through each point t when the light ray passes through the medium from the starting point to the ending point.

[0133] is the integral of the light ray on the path, which represents the contribution of the light ray passing through each point t when the light ray passes through the medium from the starting point t n to the ending point t f ; is the transmission term, which represents the penetration or transmittance of the light ray from the starting point t n to the current point t; σ(r(t)) is the volume density of the light ray at position t on the path r, which represents the local absorption or scattering intensity of the current point t to the light ray; r(s) represents the position s of the light ray on the path r, and r(t) represents the position t of the light ray on the path r.

[0134] In volume rendering, the light ray passes through different positions in the scene, and the color and density are weighted and averaged along the path of the light ray. The specific calculation formula of volume rendering is:

[0135]

[0136] wherein:

[0137] C(r) is the final rendered color value of the light ray with path r,

[0138] σ(r(t)) is the volume density at position r(t), which determines whether the position blocks the light ray;

[0139] c(r(t), d) is the color value at position r(t), which is determined by the radiation field of the point and the light ray direction d; d represents the light ray direction;

[0140] T(t) is the transmittance from the starting point t nThe cumulative transmittance to parameter t (that is, the probability that the light is not completely blocked when passing through position t).

[0141] Transmittance T(t) represents the transmission rate of light from the starting point t n The transmission probability to point t, which indicates that the light is not completely absorbed by the voxel on this path. The calculation formula of transmittance T(t) is:

[0142]

[0143] Where: σ(r(s)) is the volume density at position r(s), which represents the intensity of light blocked at that position; is the light from t n The transmittance to t.

[0144] This application uses volume rendering technology to calculate the estimated rendering color and the estimated cumulative weights along the cone axes At this pixel position, the true color is represented as RGB. The corresponding binary mask is obtained by the 2D segmenter, denoted as α, and the rendering loss Lr is defined as follows:

[0145]

[0146] in, is the estimated rendering color obtained from fine sampling, For the estimated rendering color obtained from the coarse sampling, the hyperparameter λ c Used to balance the losses of the two sampling stages; is the mean square error (MSE); in order to prevent the implicit expression model from learning the background color or shape, this application adds a binary mask α to the loss function. α is specifically a binary mask of the vehicle. Pixels without vehicles are set to 0, and pixels with vehicles are set to 1.

[0147] The segmentation loss L corresponding to the binary mask of the vehicle s The definition is as follows:

[0148]

[0149] represents the transmission weight of the fine network, represents the transmission weight of the coarse network;

[0150] In summary, the total loss function expression is:

[0151] L=L r +λ s L s

[0152] Among them, λ s is the split balance coefficient;

[0153] The single-view-based vehicle multi-view high-fidelity image generation method further comprises obtaining a road image. The road image is an image collected during driving of a vehicle on a road. The road image is processed by using neural networks such as instance segmentation and monocular depth estimation network to obtain class information, mask information, depth information and spatial 3D information of the vehicle in the road image, and camera intrinsic parameters of a camera for shooting are extracted from the road image.

[0154] The vehicle in the road image is segmented by using a Segment Anything segmentation model to obtain mask information of the vehicle. The Segment Anything Model (SAM) is developed by the MetaAI laboratory of Facebook Company and is a basic model for solving the image segmentation task. The SAM has learned general concepts about objects and can generate masks for any object in any image or video, even including objects and image types that have not been encountered during the training process. The depth information of the image is estimated by using an Omnidata monocular depth estimation network, and a real-world vehicle dataset containing rich vehicle types and vehicle spatial positions is constructed.

[0155] The vehicle dataset includes a plurality of vehicle three-dimensional models of the same type and different colors, and a plurality of vehicle three-dimensional models of different types. The single-view-based vehicle multi-view high-fidelity image generation method further comprises:

[0156] The shape features and texture features extracted from the vehicle image are compared with the vehicle three-dimensional models collected in the training database, so that the three-dimensional structure of the vehicle in the vehicle image can be quickly determined, and the view image of the vehicle at a given view angle can be quickly obtained.

[0157] Figure 4 And Figure 5 An effect diagram of view transformation of a target vehicle is shown, Figure 4 a schematic diagram of an obtained vehicle image, Figure 4 The red vehicle in the schematic diagram is a target vehicle, and a three-dimensional implicit representation model of the red vehicle in the schematic diagram is constructed by using the single-view-based vehicle multi-view high-fidelity image generation method of the present application. Figure 4 The three-dimensional implicit representation model of the red vehicle in the schematic diagram is constructed by using the single-view-based vehicle multi-view high-fidelity image generation method of the present application. Figure 5 That is, a new view image obtained after transforming the spatial position of the vehicle based on the vehicle three-dimensional implicit representation model; according to Figure 5 As shown in the schematic diagram, the technical solution of the present application can effectively learn and generate a 3D view automobile model from a single-view image, support high-quality controllable appearance editing and spatial operation, be suitable for automatic driving simulation and other 3D scene generation tasks, and reduce the possibility of affecting the reconstruction of the vehicle three-dimensional implicit representation model due to the limitation of the vehicle image collection angle.

[0158] Based on the same inventive concept, an embodiment of the present application provides a single-view-based vehicle multi-view high-fidelity image generation system.

[0159] The single-view-based vehicle multi-view high-fidelity image generation system can be loaded in an electronic device. According to the functions implemented, the single-view-based vehicle multi-view high-fidelity image generation system comprises:

[0160] An acquisition module is configured to acquire a vehicle image and camera parameter information corresponding to a camera for shooting the vehicle image;

[0161] A construction module is configured to construct a network structure of an implicit expression model;

[0162] A feature extraction module is configured to extract features of the vehicle image to obtain shape features and texture features;

[0163] A camera parameter normalization module is configured to determine a geometric center of the vehicle according to the shape features, construct a normalized coordinate system with the geometric center of the vehicle as the origin, and determine normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information;

[0164] A model generation module is configured to determine network parameters of the implicit expression model network according to the shape features, the texture features, and the normalized parameter information, and obtain a three-dimensional implicit expression model of the vehicle;

[0165] A view image generation module is configured to obtain a view image of the vehicle at a given view angle according to the three-dimensional implicit expression model.

[0166] The modules of the present application can also be referred to as units, which refer to a series of computer program segments that can be executed by an electronic device processor and can complete fixed functions, and are stored in the memory of the electronic device.

[0167] The various variations and specific examples of the single-view-based vehicle multi-view high-fidelity image generation method provided in the above embodiments are also applicable to the single-view-based vehicle multi-view high-fidelity image generation system of the present embodiment. Through the foregoing detailed description of the single-view-based vehicle multi-view high-fidelity image generation method, those skilled in the art can clearly understand the implementation method of the single-view-based vehicle multi-view high-fidelity image generation system in the present embodiment. In order to make the description brief, the implementation method of the single-view-based vehicle multi-view high-fidelity image generation system in the present embodiment will not be described in detail here.

[0168] The present application also discloses an electronic device, such as Figure 6As shown, it is a structural schematic diagram of an electronic device provided by an embodiment of the application, which is a single-view-based vehicle multi-view high-fidelity image generation method. The electronic device can include at least one processor 10, a memory 11 in communication with the at least one processor, a communication bus 12, and a communication interface 13, and can further include a computer program stored in the memory 11 and executable on the processor 10, such as a single-view-based vehicle multi-view high-fidelity image generation method program.

[0169] In some embodiments, the processor 10 can be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more combinations of central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The processor 10 is the control unit of the electronic device, which connects all components of the electronic device through various interfaces and lines, executes programs or modules stored in the memory 11 (such as the single-view-based vehicle multi-view high-fidelity image generation method), and calls data stored in the memory 11 to perform various functions and process data of the electronic device.

[0170] The memory 11 includes at least one type of readable storage medium, including flash memory, mobile hard disk, multimedia card, card-type memory (such as SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 can also be an external storage device of the electronic device, such as a plug-in mobile hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 11 can include both the internal storage unit and the external storage device of the electronic device. The memory 11 can be used not only to store application software and various data installed in the electronic device, such as the code of the single-view-based vehicle multi-view high-fidelity image generation method program, but also to temporarily store data that has been or will be output.

[0171] The communication bus 12 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. The bus is configured to enable connection and communication between the memory 11, the at least one processor 10, etc.

[0172] The communication interface 13 is configured to enable communication between the electronic device and other devices, including a network interface and a user interface. Optionally, the network interface can include a wired interface and / or a wireless interface (e.g., a WI-FI interface, a Bluetooth interface, etc.), and is typically configured to establish a communication connection between the electronic device and other electronic devices. The user interface can be a display, an input unit (e.g., a keyboard), and optionally, the user interface can also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch screen, etc. The display can also be referred to as a display screen or a display unit, and is configured to display information processed in the electronic device and to display a visualized user interface.

[0173] Figure 6 Only the electronic device with components is shown, and those skilled in the art can understand that, Figure 6 The structure shown does not limit the electronic device, and the electronic device can include fewer or more components than shown, or combine certain components, or arrange different components. For example, although not shown, the electronic device can also include a power supply (e.g., a battery) for powering the components, and preferably, the power supply can be logically connected to the at least one processor 10 through a power management device, so that the power management device can implement functions such as charge management, discharge management, and power consumption management. The power supply can also include one or more direct current or alternating current power supplies, recharging devices, power supply fault detection circuits, power supply converters or inverters, power supply status indicators, etc. The electronic device can also include various sensors, Bluetooth modules, Wi-Fi modules, etc., which are not described here.

[0174] It should be understood that the embodiments are only for illustration, and the scope of the patent application is not limited by the structure.

[0175] Furthermore, if the module / unit integrated into the electronic device is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0176] The present application provides a computer-readable storage medium, including, for example, any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM). The computer-readable storage medium stores a computer program capable of being loaded by a processor and executing the method for generating a high-fidelity multi-view image of a vehicle based on a single view according to the above embodiment.

[0177] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "example," "specific example," "one implementation," "a preferred implementation," or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0178] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A method for generating high-fidelity vehicle multi-view images based on a single view, characterized by: Obtaining a vehicle image and camera parameter information corresponding to a camera that photographed the vehicle image; Construct the network structure of the implicit expression model; Extract features of vehicle images to obtain shape features and texture features; Determine the vehicle's geometric center based on the shape features, construct a normalized coordinate system with the vehicle's geometric center as the origin, and determine the normalized parameter information of the camera in the normalized coordinate system based on the camera parameter information; Determine the network parameters of the implicit expression model network based on the shape features, texture features and normalized parameter information to obtain a three-dimensional implicit expression model of the vehicle; Obtaining a perspective image of the vehicle at a given perspective based on a three-dimensional implicit expression model; The implicit expression model network includes a shape component and a texture component. The shape component is used to represent the volume density information of the vehicle, and the texture component is used to represent the color information of the vehicle. The network parameters of the implicit expression model network include a shape component weight parameter and a texture component weight parameter. The network parameters of the implicit expression model network are determined according to the shape features, texture features and normalized parameter information, including: Sampling the vehicle's 3D scene via camera rays; Randomly select sampling points from the camera ray for implicit expression model network training; Determine the three-dimensional coordinates and visual direction of the sampling point according to the normalized parameter information; Formatting the three-dimensional coordinates and viewing direction respectively, inputting the formatted three-dimensional coordinates and viewing direction into the implicit expression model network, and the implicit expression model network outputs the vehicle three-dimensional scene volume density prediction result and the vehicle three-dimensional scene projection color prediction result; The shape component weight parameters and texture component weight parameters of the three-dimensional implicit expression model are optimized according to the vehicle three-dimensional scene volume density prediction results and the vehicle three-dimensional scene color prediction results to obtain the final three-dimensional implicit expression model.

2. The method for generating high-fidelity vehicle multi-view images based on a single view according to claim 1, wherein: The method further comprises: Model camera rays as cones; The three-dimensional coordinates of the sampling points are integrated position-encoded to obtain the position integrated position coding result. The shape component obtains the vehicle three-dimensional scene volume density prediction result based on the position integrated position coding result and shape features; The visual direction of the sampling point is integrated and position-coded to obtain the direction integrated position coding result. The texture component obtains the vehicle 3D scene projection color prediction result based on the direction integrated position coding result, the vehicle 3D scene volume density prediction result and the texture feature.

3. The method for generating a high-fidelity vehicle multi-view image based on a single view according to claim 2, wherein: The position of the cone corresponding to the camera light in direction d is represented by a multivariate Gaussian distribution with mean μ(o, d) and variance ∑(d), where o is the vertex of the cone and d is the direction of the light. The output feature expression of the shape component is: f out ,σ=f θ,shape ·(c pos (μ(o,d),∑(d))+z shape ) Where σ represents density, f θ,shape is the shape component weight parameter, γ pos (,) represents position integrated position coding, z shape Represents shape features; The output feature expression of the texture component is f out ,c=f θ,texture (f out ,σ+γ dir (d)+z texture ) Among them, c represents color, f θ,texture is the texture component weight parameter, γ dir (d) represents the direction integrated position encoding, z texture Represents texture features.

4. The method for generating a multi-view high-fidelity image of a vehicle based on a single view according to any one of claims 1 to 3, wherein: Camera parameter information includes camera pose information and lens orientation information; The determining of the vehicle geometric center according to the shape feature, constructing a normalized coordinate system with the vehicle geometric center as the origin, and determining normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information includes: Determine the geometric center of the vehicle and the current vehicle size information based on the shape features; A normalized coordinate system is constructed with the geometric center of the car as the origin. The normalized parameter information of the camera in the normalized vehicle coordinates is determined based on the camera pose information and lens orientation information. The normalized camera pose is in, Indicates the camera coordinates in the three-dimensional coordinate system of the vehicle, l represents the length of the vehicle, h represents the height of the vehicle, and w represents the width of the vehicle. represents the Hadamard product, I 4×4 Represents the identity matrix.

5. The method for generating a high-fidelity vehicle multi-view image based on a single view according to claim 4, wherein: After determining the geometric center of the vehicle and the current vehicle size information according to the shape features and before establishing the normalized coordinate system, the method further includes: Adjust the vehicle size with the camera's geometric center as the origin, and adjust the vehicle size to the preset spatial range; after the size adjustment, the camera's posture relative to the normalized coordinate system for: is the camera pose relative to the normalized coordinate system before resizing, is the vehicle’s posture relative to the camera, A scaling matrix that will scale the vehicle's dimensions relative to the camera's geometric center.

6. The method for generating a high-fidelity vehicle multi-view image based on a single view according to claim 1, 2, 3 or 5, wherein: The method of obtaining a perspective image of the vehicle at a given perspective based on the three-dimensional implicit expression model includes: Sample the light emitted from a given viewpoint and use a 3D implicit representation model to predict the color and volume density of each sampling point; Volume rendering techniques combine the color and volume density of sampled points along the ray emanating from a given viewpoint to produce the final image.

7. A system for generating high-fidelity images of vehicles from multiple perspectives based on a single view, for implementing the method for generating high-fidelity images of vehicles from multiple perspectives based on a single view according to any one of claims 1 to 6, characterized in that: include: An acquisition module, configured to acquire a vehicle image and camera parameter information corresponding to a camera that photographed the vehicle image; Building module, used to construct the network structure of implicit expression model; Feature extraction module, used to extract features of vehicle images to obtain shape features and texture features; A camera parameter normalization module is used to determine the vehicle's geometric center based on shape features, construct a normalized coordinate system with the vehicle's geometric center as the origin, and determine the normalized parameter information of the camera in the normalized coordinate system based on the camera parameter information; A model generation module is used to determine the network parameters of the implicit expression model network based on the shape features, texture features and normalized parameter information to obtain a three-dimensional implicit expression model of the vehicle; The perspective image generation module is used to obtain the perspective image of the vehicle at a given perspective based on the three-dimensional implicit expression model.

8. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively coupled to the at least one processor (10); The memory (11) stores a computer program that can be executed by the at least one processor (10), and the computer program is executed by the at least one processor (10) so that the at least one processor (10) can execute the method for generating high-fidelity multi-view images of a vehicle based on a single view as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program; when the computer program is executed by a processor, the method for generating a multi-view high-fidelity image of a vehicle based on a single view as described in any one of claims 1 to 6 is implemented.

Images