Vehicle multi-view-angle high-fidelity image generation method, system and device based on single view and storage medium

Abstract

The invention relates to the technical field of automatic driving, and relates to a vehicle multi-view high-fidelity image generation method based on a single view. The method comprises the following steps: acquiring a vehicle image and camera parameter information corresponding to a camera shooting the vehicle image; constructing a network structure of the implicit expression model; extracting features of the vehicle image to obtain shape features and texture features; constructing a standardized coordinate system by taking the geometric center of the vehicle as an original point, and determining standardized parameter information of the camera in the standardized coordinate system; determining network parameters of an implicit expression model network according to the shape features, the texture features and the normalized parameter information to obtain a three-dimensional implicit expression model of the vehicle; and obtaining a visual angle image of the vehicle at a given visual angle according to the three-dimensional implicit expression model. The invention further provides a single-view-based vehicle multi-view-angle high-fidelity image generation system and device and a storage medium. According to the method, the possibility that the reconstruction of the vehicle three-dimensional implicit expression model is influenced due to the limitation of the vehicle image acquisition angle can be reduced.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, system, device and storage medium for generating multi-view high-fidelity images of a vehicle based on a single view. Background Art

[0002] In the field of autonomous driving, vehicle reconstruction is a key technical link. Vehicle reconstruction methods obtain perception data through multiple sensors (such as radar, LiDAR, cameras, etc.), and use computer vision, machine learning, and deep learning technologies to extract, understand, and reconstruct information about surrounding vehicles, so that the autonomous driving system can accurately perceive and predict the behavior of other vehicles.

[0003] At present, NeRF technology is usually used to process driving scene images and camera parameters to predict the three-dimensional model of the driving scene; NeRF's full name is Neural Radiance Fields, which means neural radiation field in Chinese; NeRF is an advanced computer graphics technology that reconstructs complex vehicle three-dimensional scenes from a partial two-dimensional image set through deep learning technology. The core of NeRF is to learn the geometry, objects and angles of a specific scene by training a neural network, so that a 3D view of the vehicle in the scene can be rendered from a new perspective; however, the existing NeRF-based vehicle reconstruction method mainly reconstructs the vehicle through multiple views or fewer views, but due to the perspective limitations of the autonomous driving data set, it is usually impossible to obtain multi-angle views of each vehicle in the scene in real scenes, which makes the existing technology have many limitations on the prediction of multi-view images of vehicles. Summary of the invention

[0004] In order to reduce the possibility of affecting the reconstruction of the vehicle's three-dimensional implicit expression model due to the limitation of the vehicle image acquisition angle and improve the convenience of vehicle multi-view image prediction, a method, system, device and storage medium for generating high-fidelity multi-view images of a vehicle based on a single view are provided.

[0005] In order to achieve the above-mentioned objective of the present invention, the present invention provides a method for generating multi-view high-fidelity images of a vehicle based on a single view.

[0006] The present invention provides a method for generating a vehicle multi-view high-fidelity image based on a single view, comprising:

[0007] Acquire a vehicle image and camera parameter information corresponding to a camera that photographed the vehicle image;

[0008] Construct the network structure of implicit expression model;

[0009] Extract the features of the vehicle image to obtain shape features and texture features;

[0010] Determine the geometric center of the vehicle according to the shape feature, construct a normalized coordinate system with the geometric center of the vehicle as the origin, and determine the normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information;

[0011] Determine the network parameters of the implicit expression model network according to the shape features, texture features and normalized parameter information, and obtain a three-dimensional implicit expression model of the vehicle;

[0012] The perspective image of the vehicle at a given perspective is obtained according to the three-dimensional implicit expression model.

[0013] Optionally, the implicit expression model network includes a shape component and a texture component, the shape component is used to represent the volume density information of the vehicle, and the texture component is used to represent the color information of the vehicle; the network parameters of the implicit expression model network include a shape component weight parameter and a texture component weight parameter;

[0014] The network parameters of the implicit expression model network are determined according to the shape features, texture features and normalized parameter information, including:

[0015] Sampling the vehicle's three-dimensional scene through camera rays;

[0016] Randomly select sampling points from the camera ray for implicit expression model network training;

[0017] Determine the three-dimensional coordinates and visual direction of the sampling point according to the normalized parameter information;

[0018] The three-dimensional coordinates and the viewing direction are formatted respectively, and the formatted three-dimensional coordinates and the viewing direction are input into the implicit expression model network, and the implicit expression model network outputs the vehicle three-dimensional scene volume density prediction result and the vehicle three-dimensional scene projection color prediction result;

[0019] According to the vehicle three-dimensional scene volume density prediction results and the vehicle three-dimensional scene color prediction results, the shape component weight parameters and texture component weight parameters of the three-dimensional implicit expression model are optimized to obtain the final three-dimensional implicit expression model.

[0020] Optionally, the method further comprises:

[0021] Model the camera ray as a cone;

[0022] Perform integrated position coding on the three-dimensional coordinates of the sampling points to obtain the position integrated position coding result, and the shape component obtains the vehicle three-dimensional scene volume density prediction result according to the position integrated position coding result and the shape feature;

[0023] The visual direction of the sampling point is integrated and positionally encoded to obtain the direction integrated and positionally encoded result. The texture component obtains the vehicle 3D scene projection color prediction result based on the direction integrated and positionally encoded result, the vehicle 3D scene volume density prediction result and the texture feature.

[0024] Optionally, the position of the cone corresponding to the camera light in the direction d is represented as a multivariate Gaussian distribution with a mean of μ(o, d) and a variance of ∑(d), where o is the vertex of the cone and d is the direction of the light;

[0025] The output feature expression of the shape component is

[0026] f out ,σ=f θ,shape (γ pos (μ(o,d),∑(d))+z shape )

[0027] Among them, σ represents density, f θ,shape is the shape component weight parameter, γ pos (.) indicates positional integrated position coding, z shape Represents shape features;

[0028] The output feature expression of the texture component is

[0029] f out ,c=f θ,texture (f out ,σ+γ dir (d)+z texture )

[0030] Among them, c represents the color, f θ,texture is the texture component weight parameter, γ dir (d) represents the direction integrated position encoding, z texture Represents texture features.

[0031] Optionally, the camera parameter information includes camera pose information and lens orientation information;

[0032] The step of determining the geometric center of the vehicle according to the shape feature, constructing a normalized coordinate system with the geometric center of the vehicle as the origin, and determining normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information includes:

[0033] Determine the geometric center of the vehicle and current vehicle size information based on shape features;

[0034] A normalized coordinate system is constructed with the vehicle geometric center as the origin. The normalized parameter information of the camera in the normalized vehicle coordinates is determined based on the camera pose information and lens orientation information. The normalized camera pose is

[0035]

[0036] in, represents the camera coordinates in the three-dimensional coordinate system of the vehicle, l represents the length of the vehicle, h represents the height of the vehicle, and w represents the width of the vehicle. represents the Hadamard product, I 4×4 Represents the identity matrix.

[0037] Optionally, after determining the geometric center of the vehicle and the current vehicle size information according to the shape features and before establishing the normalized coordinate system, the method further includes:

[0038] The vehicle size is adjusted with the camera geometric center as the origin, and the vehicle size is adjusted to the preset spatial range; after the size adjustment, the camera posture relative to the normalized coordinate system for

[0039]

[0040] is the camera pose relative to the normalized coordinate system before resizing, is the vehicle’s attitude relative to the camera, is a scaling matrix that will scale the vehicle's dimensions around the camera's geometric center.

[0041] Optionally, obtaining a perspective image of the vehicle at a given perspective according to the three-dimensional implicit expression model includes:

[0042] Sample the light emitted from a given viewpoint and use a 3D implicit representation model to predict the color and volume density of each sampling point;

[0043] Volume rendering techniques combine the color and volume density of sampled points on the light emitted from a given viewpoint to produce the final image.

[0044] In order to solve the above problems, the present invention also provides a vehicle multi-view high-fidelity image generation system based on a single view, the system comprising:

[0045] An acquisition module, used to acquire a vehicle image and camera parameter information corresponding to a camera that photographs the vehicle image;

[0046] A building module is used to construct the network structure of the implicit expression model;

[0047] A feature extraction module is used to extract features of vehicle images to obtain shape features and texture features;

[0048] A camera parameter normalization module is used to determine the geometric center of the vehicle according to the shape features, construct a normalized coordinate system with the geometric center of the vehicle as the origin, and determine the normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information;

[0049] A model generation module is used to determine the network parameters of the implicit expression model network according to the shape features, texture features and normalized parameter information to obtain a three-dimensional implicit expression model of the vehicle;

[0050] The perspective image generation module is used to obtain the perspective image of the vehicle at a given perspective based on the three-dimensional implicit expression model.

[0051] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:

[0052] at least one processor; and,

[0053] a memory communicatively connected to the at least one processor; wherein,

[0054] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the above-mentioned method for generating multi-view high-fidelity images of a vehicle based on a single view.

[0055] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is executed by a processor in an electronic device to implement the above-mentioned method for generating multi-view high-fidelity images of a vehicle based on a single view.

[0056] In summary, this application includes the following beneficial technical effects:

[0057] In order to supervise the implicit expression model network to learn the 3D shape of a vehicle from a single vehicle image, the present application constructs a normalized coordinate system based on shape features, and converts the camera coordinate system into a normalized coordinate system with the geometric center of the vehicle as the origin and aligned along the main axis, so that the implicit expression model network can efficiently learn and generate a 3D perspective three-dimensional implicit expression model of the vehicle based on shape features, texture features and normalized parameter information, thereby reducing the possibility of affecting the reconstruction of the vehicle's three-dimensional implicit expression model due to the limitation of the vehicle image acquisition angle; the volume rendering method is used to obtain multiple perspective images based on the vehicle's three-dimensional implicit expression model, thereby improving the convenience of vehicle perspective image prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 A schematic diagram of a flow chart of a method for generating a multi-view high-fidelity image of a vehicle based on a single view provided by an embodiment of the present invention;

[0059] Figure 2 A diagram of the architecture of a vehicle multi-view high-fidelity image generation model provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of the relationship between a camera and a normalized vehicle coordinate system is provided for one embodiment of the present invention;

[0061] Figure 4 A schematic diagram of a vehicle image obtained according to an embodiment of the present invention;

[0062] Figure 5 A new perspective image obtained by transforming the spatial position of a vehicle based on a three-dimensional implicit expression model of the vehicle provided in an embodiment of the present invention;

[0063] Figure 6 A schematic diagram of the structure of an electronic device for implementing the method for generating multi-view high-fidelity images of a vehicle based on a single view provided in one embodiment of the present invention.

[0064] Reference numerals: 10, processor; 11, memory; 12, communication bus; 13, communication interface.

[0065] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0066] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0067] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0068] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0069] Reference Figure 1 FIG. 1 is a flow chart of a method for generating a high-fidelity image of a vehicle from multiple perspectives based on a single view according to an embodiment of the present invention. In this embodiment, the method for generating a high-fidelity image of a vehicle from multiple perspectives based on a single view includes:

[0070] S1. Obtain a vehicle image and camera parameter information corresponding to a camera that photographs the vehicle image.

[0071] The camera parameter information includes camera pose information, lens orientation information, focal length, and intrinsic parameter matrix, etc. The camera parameter information can be obtained through a camera calibration tool (such as COLMAP), or other tools can be used to determine the camera parameter information, which is not limited in this application.

[0072] S2. Construct the network structure of implicit expression model.

[0073] The network structure of the implicit expression model is a neural network (such as a multi-layer perceptron MLP). The neural network is trained to learn a continuous function mapping of the data to obtain a three-dimensional implicit expression model of the vehicle. Taking an image as an example, each pixel can be represented by its position and color value. The neural network (such as a multi-layer perceptron MLP) is trained to predict the color value based on the pixel position. The trained implicit expression model can not only encode the data seen during the training process, but also predict the color value of pixels at any position, including those positions that have not directly appeared in the training data, thereby realizing the subsequent prediction of multi-view images of the vehicle.

[0074] S3. Extract the features of the vehicle image to obtain shape features and texture features;

[0075] Reference Figure 1 and Figure 2 The shape feature is used to guide the implicit expression network to obtain the volume density occupancy of the 3D point, and the appearance feature is used to guide the implicit expression network to obtain the RGB color of the 3D point under a given observation direction.

[0076] Feature extraction networks such as ResNet and VGG16 can be used to extract features from vehicle images. The full name of ResNet in English is Residual Networks, and its Chinese meaning is "residual network". ResNet is a deep convolutional neural network architecture proposed by Microsoft Research. The core feature of ResNet is the introduction of residual connections, which solves the gradient vanishing and degradation problems in deep networks and allows the network to be trained deeper. The full name of VGG16 in English is Visual Geometry Group 16 layers, and its Chinese meaning is "visual geometry group 16 layers". VGG16 is characterized by the use of a large number of small-size 3x3 convolution kernels and 2x2 maximum pooling layers. This design reduces the number of parameters and improves computational efficiency. In this embodiment, the ResNet-34 feature extraction network is used to extract shape features and texture features from vehicle images.

[0077] S4. Determine the geometric center of the vehicle according to the shape features, construct a normalized coordinate system with the geometric center of the vehicle as the origin, and determine normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information.

[0078] Reference Figure 3 , determine the vehicle geometric center according to the shape features, construct a normalized coordinate system with the vehicle geometric center as the origin, and determine the normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information, including:

[0079] S41. Determine the geometric center of the vehicle and current vehicle size information according to the shape features.

[0080] S42, constructing a normalized coordinate system with the geometric center of the vehicle as the origin, and determining normalized parameter information of the camera in the normalized vehicle coordinates based on the camera pose information and the lens orientation information.

[0081] Normalized camera coordinates It can be expressed by the following formula:

[0082]

[0083] in, represents the camera coordinates in the three-dimensional coordinate system of the vehicle, l represents the length of the vehicle, h represents the height of the vehicle, and w represents the width of the vehicle. It is the Hadamard product. The Hadamard product is a matrix operation method. The specific operation is to multiply item by item; 4×4 represents the unit matrix, and the size of the car (length l, height h, width w) is obtained from the results of the 3D detection module.

[0084] Preferably, after determining the geometric center of the vehicle and the current vehicle size information according to the shape features and before establishing the normalized coordinate system, the method for generating a multi-view high-fidelity image of a vehicle based on a single view further includes:

[0085] The vehicle size is adjusted with the camera geometric center as the origin, and the vehicle size is adjusted to the preset spatial range; after the size adjustment, the camera posture relative to the normalized coordinate system for

[0086]

[0087] is the camera pose relative to the normalized coordinate system before resizing, is the vehicle’s attitude relative to the camera, is a scaling matrix that will scale the vehicle's dimensions around the camera's geometric center.

[0088] Reference Figure 3 In layman's terms, both the camera and the vehicle have a pose in the world coordinate system. The coordinate system constructed with the focal length of the camera lens as the origin is the camera coordinate system, and the coordinate system with the geometric center of the vehicle as the origin is the vehicle coordinate system. The vehicle coordinate system takes the geometric center of the vehicle as the origin, the x-axis is the direction of the front of the vehicle, the y-axis is upward, and the z-axis is toward the right side of the vehicle body. Since a single view is used to train the model, the normalized coordinates are constructed with the geometric center of the vehicle as the origin, and a normalized operation is used to standardize the camera parameter information: At the same time, due to the inconsistency of vehicle sizes, a size transformation is required. Based on the vehicle size, it is transformed to a square space from -1 to 1. In this embodiment, represents the scaling matrix that maps the canonical coordinate system to the camera-centered scaled coordinate system, The diagonal values ​​are (1,-1,-1,1) and are scaled to fit the size of the vehicle.

[0089] S5. Determine the network parameters of the implicit expression model network according to the shape features, texture features and normalized parameter information to obtain a three-dimensional implicit expression model of the vehicle.

[0090] Reference Figure 2 , the implicit expression model network includes a shape component and a texture component, the shape component is used to represent the volume density information of the vehicle, and the texture component is used to represent the color information of the vehicle; the network parameters of the implicit expression model network include a shape component weight parameter and a texture component weight parameter;

[0091] In order to improve the network representation ability, the implicit expression model introduces a hyperparameter network HyperNetwork, which can generate the network parameters of the implicit expression model network according to the characteristics of the input image, so that the implicit expression model can adaptively generate a suitable view synthesis model for different scenes. The feature vector obtained by the implicit expression model network through resnet34 is converted into the network parameters of the implicit expression model network. Mathematically, the hypernetwork can be expressed as: Ψ:R k →R l , where R k represents the dimension of the latent code is k, and R l The dimension of the network parameter space representing the implicit expression model network is l, and “→” represents a conversion operation.

[0092]

[0093] z represents the feature vector, Ψ(z) represents the network parameters of the implicit expression model network, θ shape represents the shape component weight parameter, θ texture Represents the texture component weight parameter; It means that the feature vector is input into the hyperparameter network to obtain the shape component weight parameter and the texture component weight parameter.

[0094] Specifically, the network parameters of the implicit expression model network are determined according to the shape features, texture features and normalized parameter information, including:

[0095] S51. Sample the vehicle three-dimensional scene through camera rays.

[0096] Reference Figure 3 , Figure 3 The blue dotted line in the middle is a schematic sampling ray obtained by sampling the vehicle's three-dimensional scene through the camera ray. The ray is emitted from the transformed coordinate system to pass through the vehicle, and the sampling point is selected to generate an image of the camera's corresponding perspective.

[0097] S52. Randomly select sampling points from the camera ray for implicit expression model network training.

[0098] S53. Determine the three-dimensional coordinates and visual direction of the sampling point according to the normalized parameter information.

[0099] S54, formatting the three-dimensional coordinates and viewing direction respectively, inputting the formatted three-dimensional coordinates and viewing direction into the implicit expression model network, and the implicit expression model network outputs the vehicle three-dimensional scene volume density prediction result and the vehicle three-dimensional scene projection color prediction result.

[0100] S55. Optimize the shape component weight parameters and texture component weight parameters of the three-dimensional implicit expression model according to the vehicle three-dimensional scene volume density prediction result and the vehicle three-dimensional scene color prediction result to obtain the final three-dimensional implicit expression model.

[0101] In the model architecture, since the size of the input image patch varies greatly and the area represented by the pixel also changes significantly, cone tracing is used instead of ray tracing to achieve continuous scaling, and anti-aliasing is performed through integrated position encoding (IPE). Anti-aliasing is the process of softening the triangular jagged edges of objects in the display screen, which can make the image smoother and clearer, closer to the real object, and improve the image quality. Anti-aliasing is a technology that can eliminate the jagged edges of the image in the image output by the display. After turning on anti-aliasing, the image edge will look smoother and closer to the real object.

[0102] Reference Figure 2 In the preferred implementation of this embodiment, the camera rays are modeled as cones; through cone tracing, the size of the cone can be adjusted at different levels (i.e., the cone is scaled) to control the range and accuracy of sampling. In the initial stage, a larger cone is used for coarse sampling to quickly obtain a rough scene representation. Then, the size of the cone is gradually reduced for finer sampling to improve the quality and details of the rendering.

[0103] Cone tracing can improve the robustness and efficiency of rendering, especially when dealing with reflections, refractions, or dynamic scenes; by sampling over a wider area, it can better capture the details of the scene while reducing rendering artifacts caused by errors in a single sampling point.

[0104] The goal of IPE (Integrated Position Encoding) is to use multivariate Gaussian distribution to represent the continuous distribution of light or cones in an image and encode this distribution. Integrated Position Encoding uses this distribution to replace the point position in traditional position encoding, improving the anti-aliasing ability and multi-scale processing ability of objects.

[0105] Suppose we carefully sample a cone with vertex o and direction d, where direction d is the direction of the line where the cone's height lies in the normalized coordinate system; the position of the cone with vertex o and direction d in direction d can be expressed as a multivariate Gaussian distribution with mean μ(o, d) and variance Σ(d). At this time, the formula for integrated position encoding is as follows:

[0106]

[0107] in:

[0108] μ(o,d) represents the mean value of the cone in direction d (similar to position), o is the vertex of the cone, and d is the direction of the line where the cone's height lies in the normalized coordinate system;

[0109] Σ(d) represents the variance in this direction, and Σ(d) describes the expansion of the cone;

[0110] k is a set of basis vectors, usually used for Fourier transform (or sine-cosine transform), k T is the transpose of k;

[0111] Indicates the effect of variance on encoding, which is used to control position encoding at different scales;

[0112] sin(k T μ(o,d)) represents the sine transform of the mean, cos(k T μ(o,d)) represents the cosine transform of the mean.

[0113] In the NeRF model, usually not only the position but also the direction of the light is encoded. Therefore, IPE can be divided into two parts: position-integrated position encoding results and direction-integrated position encoding results.

[0114] The three-dimensional coordinates of the sampling points are integrated and encoded to obtain the position integrated position encoding result. The position integrated position encoding is the encoding of the mean μ(o, d) and the variance ∑(d). Specifically, the position integrated position encoding result corresponding to the cone with vertex o and direction d is expressed as γ pos (o,d),γ pos (o,d)=IPE(μ(o,d),∑(d)), IPE(.) is the integrated position encoding; the shape component obtains the vehicle 3D scene volume density prediction result according to the integrated position encoding result and shape features;

[0115] The visual direction of the sampling point is integrated and positionally encoded to obtain the direction integrated position encoding result. The direction integrated position encoding is the Fourier encoding of the direction. Specifically, the direction integrated position encoding result corresponding to the cone with vertex o and direction d is expressed as γ dir (d), γ dir (d) = IPE(d).

[0116] The texture component obtains the vehicle 3D scene projection color prediction result based on the directional integrated position encoding result, the vehicle 3D scene volume density prediction result and the texture feature.

[0117] The position of the cone corresponding to the camera light in direction d is represented as a multivariate Gaussian distribution with mean μ(o, d) and variance ∑(d), where o is the vertex of the cone and d is the direction of the light.

[0118] The output feature expression of the shape component is

[0119] fout ,σ=f θ,shape (γpos(μ(o,d),∑(d))+z shape )

[0120] Among them, σ represents density, f θ,shape is the shape component weight parameter, γ pos (.) indicates positional integrated position coding, z shape Represents shape features;

[0121] The output feature expression of the texture component is

[0122] f out ,c=f θ,texture (fout,σ+γ dir (d)+z texture )

[0123] Among them, c represents color, f θ,texture is the texture component weight parameter, γ dir (d) represents the direction integrated position encoding, z texture Represents texture features.

[0124] S6. Obtain a perspective image of the vehicle at a given perspective according to the three-dimensional implicit expression model.

[0125] The three-dimensional implicit expression model has the ability to encode and decode instance categories. It can encode the input vehicle image at the instance level, generate the shape code and appearance code of the corresponding vehicle, and finally input it together with the queried 3D point and observation direction into the subsequent decoder to obtain the rendered vehicle image in the real scene. Taking into account the different sizes of actual vehicles, in order to meet the model requirements of decoder training and facilitate 3D point generation and query, the implicit expression of the vehicle is normalized to a cube space of -1 to 1, so that implicit expression models covering various vehicle categories can be learned.

[0126] Specifically, step S6 includes:

[0127] S61, sampling light rays emitted from a given viewing angle, and predicting the color and volume density of each sampling point using a three-dimensional implicit expression model;

[0128] S62. Use volume rendering technology to combine the color and volume density of the sampling points on the light emitted from a given perspective to generate a final image.

[0129] The volume rendering sampling method also adopts the usual sampling strategy used in the implicit expression model. The volume rendering sampling steps are divided into two steps: first, coarse sampling is performed, and coarse sampling uses equidistant sampling. Then, fine sampling is performed. Fine sampling is calculated using the following cumulative weight formula, and then the probability density function (PDF) is obtained through the weight distribution, which is then converted into its cumulative distribution (CDF) function. Fine sampling points are obtained through the inverse transform sampling method, and finally, volume rendering is performed in combination with the coarse sampling points.

[0130] Cone axis from close distance t n To long distance f The cumulative weight ∫ω is calculated by the following formula:

[0131]

[0132] ω(r) is the cumulative transmission weight, which is used to describe the color contribution or energy transfer of light on path r. It represents the contribution of each point t when the light passes through the medium from the starting point to the end point.

[0133] is the integral of the light on the path, indicating that the light starts from the starting point t n To the end point f The contribution at each point t when passing through the medium; is the transmission term, which means that the light is transmitted from the starting point t n The penetration or transmittance to the current point t; σ(r(t)) is the volume density of the light at position t on the path r, which represents the local absorption or scattering intensity of the light at the current point t; r(s) represents the position s of the light on the path r, and r(t) represents the position t of the light on the path r.

[0134] In volume rendering, light passes through different locations in the scene, and the color and density are weighted averaged along the path of the light. The specific calculation formula for volume rendering is:

[0135]

[0136] in:

[0137] C(r) is the final color value rendered by the light with path r.

[0138] σ(r(t)) is the volume density at position r(t), which determines whether the position blocks light;

[0139] c(r(t), d) is the color value at position r(t), which is determined by the radiation field at that point and the direction of the light d, where d represents the direction of the light;

[0140] T(t) is the distance from the starting point of the ray t nThe cumulative transmittance to parameter t (i.e., the probability that the light is not completely blocked when passing through position t).

[0141] The transmittance T(t) represents the transmission rate of the light from the starting point t n The transmission probability to point t indicates that the light is not completely absorbed by the voxel on this path. The calculation formula of transmittance T(t) is:

[0142]

[0143] Where: σ(r(s)) is the volume density at position r(s), which indicates the intensity of light blocked at that position; is the light from t n The transmittance to t.

[0144] This application uses volume rendering techniques to calculate the estimated rendering color and the estimated cumulative weights along the cone axes At this pixel position, the true color is represented as RGB. The corresponding binary mask is obtained by the 2D segmenter, denoted as α, and the rendering loss Lr is defined as follows:

[0145]

[0146] in, is the estimated rendering color obtained from fine sampling, For the estimated rendering color obtained from the coarse sampling, the hyperparameter λ c Used to balance the losses of the two sampling stages; is the MSE mean square error; in order to prevent the implicit expression model from learning the background color or shape, the present application adds a binary mask α to the loss function, where α is specifically a binary mask of the vehicle, where pixels without vehicles are set to 0 and pixels with vehicles are set to 1.

[0147] The segmentation loss L corresponding to the binary mask of the vehicle s The definition is as follows:

[0148]

[0149] represents the transmission weight of the fine network, represents the transmission weight of the coarse network;

[0150] In summary, the total loss function expression is:

[0151] L=L r +λ s L s

[0152] Among them, λ s is the split balance coefficient;

[0153] The method for generating high-fidelity images of vehicles from multiple perspectives based on a single view also includes obtaining road acquisition images. Road acquisition images are images collected while the vehicle is driving on the road, and the road acquisition images are processed using neural networks such as instance segmentation and monocular depth estimation networks to obtain category information, mask information, depth information, and spatial 3D information of the vehicle in the road acquisition images, and the camera intrinsic parameters of the shooting camera are extracted from the road acquisition images;

[0154] The Segment Anything segmentation model is used to segment the vehicles in the road collection images to obtain the vehicle mask information. The Segment Anything Model (SAM) is a basic model developed by Facebook's MetaAI laboratory to solve image segmentation tasks. SAM has learned the general concept of objects and can generate masks for any object in any image or video, even including objects and image types that have not been encountered during training; the Omnidata monocular depth estimation network is used to estimate the depth information of the image and construct a real-world vehicle dataset containing rich vehicle types and vehicle spatial locations.

[0155] The vehicle dataset includes a plurality of vehicle 3D models of the same model and / or different colors, and a plurality of vehicle 3D models of different models. The vehicle multi-view high-fidelity image generation method based on a single view also includes:

[0156] The shape features and texture features extracted from the vehicle image are compared with the vehicle 3D model collected in the training database, so as to quickly determine the 3D structure of the vehicle in the vehicle image, and then quickly obtain the perspective image of the vehicle at a given perspective.

[0157] Figure 4 and Figure 5 The effect of changing the perspective of the target vehicle is shown. Figure 4 Schematic diagram of the vehicle image obtained. Figure 4 The red vehicle in the figure is the target vehicle, and the vehicle multi-view high-fidelity image generation method based on a single view of the present application is used to construct Figure 4 The three-dimensional implicit expression model of the red vehicle in the middle; Figure 5 That is, the new perspective image is obtained after transforming the vehicle's spatial position based on the vehicle's three-dimensional implicit expression model; Figure 5 As shown, the technical solution of the present application can effectively learn and generate a 3D perspective car model from a single-view image, and supports high-quality controllable appearance editing and spatial operations. It is suitable for autonomous driving simulation and other 3D scene generation tasks, and reduces the possibility of affecting the reconstruction of the vehicle's three-dimensional implicit expression model due to the limitation of the vehicle image acquisition angle.

[0158] Based on the same inventive concept, an embodiment of the present invention provides a vehicle multi-view high-fidelity image generation system based on a single view.

[0159] The vehicle multi-view high-fidelity image generation system based on a single view of the present invention can be loaded in an electronic device. According to the functions implemented, the vehicle multi-view high-fidelity image generation system based on a single view of the present invention includes:

[0160] An acquisition module, used to acquire a vehicle image and camera parameter information corresponding to a camera that photographs the vehicle image;

[0161] A building module is used to construct the network structure of the implicit expression model;

[0162] A feature extraction module is used to extract features of vehicle images to obtain shape features and texture features;

[0163] A camera parameter normalization module is used to determine the geometric center of the vehicle according to the shape features, construct a normalized coordinate system with the geometric center of the vehicle as the origin, and determine the normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information;

[0164] A model generation module is used to determine the network parameters of the implicit expression model network according to the shape features, texture features and normalized parameter information to obtain a three-dimensional implicit expression model of the vehicle;

[0165] The perspective image generation module is used to obtain the perspective image of the vehicle at a given perspective based on the three-dimensional implicit expression model.

[0166] The module described in the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in a memory of the electronic device.

[0167] The various variations and specific examples of the vehicle multi-perspective high-fidelity image generation method based on a single view provided in the above-mentioned embodiment are also applicable to the vehicle multi-perspective high-fidelity image generation system based on a single view in the present embodiment. Through the above-mentioned detailed description of the vehicle multi-perspective high-fidelity image generation method based on a single view, those skilled in the art can clearly understand the implementation method of the vehicle multi-perspective high-fidelity image generation system based on a single view in the present embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0168] The present application also discloses an electronic device, such as Figure 6, which is a schematic diagram of the structure of an electronic device for a method for generating high-fidelity images of vehicles from multiple perspectives based on a single view according to an embodiment of the present invention. The electronic device may include at least one processor 10, a memory 11 connected to the at least one processor for communication, a communication bus 12, and a communication interface 13, and may also include a computer program stored in the memory 11 and executable on the processor 10, such as a program for a method for generating high-fidelity images of vehicles from multiple perspectives based on a single view.

[0169] Among them, the processor 10 may be composed of an integrated circuit in some embodiments, for example, it may be composed of a single packaged integrated circuit, or it may be composed of multiple integrated circuits packaged with the same function or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (for example, executing a vehicle multi-view high-fidelity image generation method based on a single view, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device and process data.

[0170] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the memory 11 may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device. Further, the memory 11 may also include both an internal storage unit of the electronic device and an external storage device. The memory 11 can not only be used to store application software and various types of data installed in the electronic device, such as the code of the method program for generating high-fidelity images of multiple perspectives of a vehicle based on a single view, but can also be used to temporarily store data that has been output or is to be output.

[0171] The communication bus 12 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0172] The communication interface 13 is used for communication between the above-mentioned electronic device and other devices, including a network interface and a user interface. Optionally, the network interface may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device and other electronic devices. The user interface may be a display (Display), an input unit (such as a keyboard (Keyboard)), and optionally, the user interface may also be a standard wired interface, a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode, organic light-emitting diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device and to display a visual user interface.

[0173] Figure 6 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 6 The structure shown does not constitute a limitation on the electronic device, and may include fewer or more components than shown, or combine certain components, or arrange the components differently. For example, although not shown, the electronic device may also include a power supply (such as a battery) for supplying power to each component. Preferably, the power supply may be logically connected to at least one processor 10 through a power management device, so that functions such as charging management, discharging management, and power consumption management are implemented through the power management device. The power supply may also include any components such as one or more DC or AC power supplies, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0174] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited by this structure.

[0175] Furthermore, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable storage medium can be volatile or non-volatile.

[0176] The embodiment of the present application provides a computer-readable storage medium, for example, including: any entity or device, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM) that can carry the computer program code. The computer-readable storage medium stores a computer program that can be loaded by a processor and execute the method for generating high-fidelity images of vehicles from multiple perspectives based on a single view in the above embodiment.

[0177] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", "an implementation", "a preferred implementation" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0178] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A vehicle multi-view high-fidelity image generation method based on a single view, characterized in that: Acquire a vehicle image and camera parameter information corresponding to a camera that photographed the vehicle image; Construct the network structure of implicit expression model; Extract the features of the vehicle image to obtain shape features and texture features; Determine the geometric center of the vehicle according to the shape feature, construct a normalized coordinate system with the geometric center of the vehicle as the origin, and determine the normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information; Determine the network parameters of the implicit expression model network according to the shape features, texture features and normalized parameter information, and obtain a three-dimensional implicit expression model of the vehicle; The perspective image of the vehicle at a given perspective is obtained according to the three-dimensional implicit expression model.

2. The method for generating a high-fidelity image of a vehicle from multiple perspectives based on a single view as claimed in claim 1, characterized in that: The implicit expression model network includes a shape component and a texture component, the shape component is used to represent the volume density information of the vehicle, and the texture component is used to represent the color information of the vehicle; the network parameters of the implicit expression model network include a shape component weight parameter and a texture component weight parameter; The network parameters of the implicit expression model network are determined according to the shape features, texture features and normalized parameter information, including: Sampling the vehicle's three-dimensional scene through camera rays; Randomly select sampling points from the camera ray for implicit expression model network training; Determine the three-dimensional coordinates and visual direction of the sampling point according to the normalized parameter information; The three-dimensional coordinates and the viewing direction are formatted respectively, and the formatted three-dimensional coordinates and the viewing direction are input into the implicit expression model network, and the implicit expression model network outputs the vehicle three-dimensional scene volume density prediction result and the vehicle three-dimensional scene projection color prediction result; According to the vehicle three-dimensional scene volume density prediction results and the vehicle three-dimensional scene color prediction results, the shape component weight parameters and texture component weight parameters of the three-dimensional implicit expression model are optimized to obtain the final three-dimensional implicit expression model.

3. The method for generating a high-fidelity image of a vehicle from multiple perspectives based on a single view as claimed in claim 2, characterized in that: The method further comprises: Model the camera ray as a cone; Perform integrated position coding on the three-dimensional coordinates of the sampling points to obtain the position integrated position coding result, and the shape component obtains the vehicle three-dimensional scene volume density prediction result according to the position integrated position coding result and the shape feature; The visual direction of the sampling point is integrated and positionally encoded to obtain the direction integrated and positionally encoded result. The texture component obtains the vehicle 3D scene projection color prediction result based on the direction integrated and positionally encoded result, the vehicle 3D scene volume density prediction result and the texture feature.

4. The method for generating a high-fidelity image of a vehicle from multiple perspectives based on a single view as claimed in claim 3, characterized in that: The position of the cone corresponding to the camera light in direction d is represented as a multivariate Gaussian distribution with mean μ(o, d) and variance ∑(d), where o is the vertex of the cone and d is the direction of the light. The output feature expression of the shape component is: f out ,σ=f θ,shape ·(c pos (μ(o,d),∑(d))+z shape ) Among them, σ represents density, f θ,shape is the shape component weight parameter, γ pos (,) represents position integrated position encoding, z shape Represents shape features; The output feature expression of the texture component is f out ,c=f θ,texture (f out ,σ+γ dir (d)+z texture ) Among them, c represents color, f θ,texture is the texture component weight parameter, γ dir (d) represents the direction integrated position encoding, z texture Represents texture features.

5. The method for generating a high-fidelity image of a vehicle from multiple perspectives based on a single view according to any one of claims 1 to 4, characterized in that: The camera parameter information includes the camera pose information and the lens orientation information; The step of determining the geometric center of the vehicle according to the shape feature, constructing a normalized coordinate system with the geometric center of the vehicle as the origin, and determining normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information includes: Determine the geometric center of the vehicle and current vehicle size information based on shape features; A normalized coordinate system is constructed with the geometric center of the car as the origin. The normalized parameter information of the camera in the normalized vehicle coordinates is determined based on the camera pose information and lens orientation information. The normalized camera pose is in, represents the camera coordinates in the three-dimensional coordinate system of the vehicle, l represents the length of the vehicle, h represents the height of the vehicle, and w represents the width of the vehicle. represents the Hadamard product, I 4×4 Represents the identity matrix.

6. The method for generating a high-fidelity image of a vehicle from multiple perspectives based on a single view as claimed in claim 5, characterized in that: After determining the geometric center of the vehicle and the current vehicle size information according to the shape features and before establishing the normalized coordinate system, the method further includes: The vehicle size is adjusted with the camera geometric center as the origin, and the vehicle size is adjusted to the preset spatial range; after the size adjustment, the camera posture relative to the normalized coordinate system for: is the camera pose relative to the normalized coordinate system before resizing, is the vehicle’s attitude relative to the camera, is a scaling matrix that will scale the vehicle's dimensions around the camera's geometric center.

7. The method for generating a high-fidelity image of a vehicle from multiple perspectives based on a single view according to claim 1, 2, 3, 4 or 6, characterized in that: The step of obtaining a perspective image of the vehicle at a given perspective according to the three-dimensional implicit expression model includes: Sample the light emitted from a given viewpoint and use a 3D implicit representation model to predict the color and volume density of each sampling point; Volume rendering techniques combine the color and volume density of sampled points on the light emitted from a given viewpoint to produce the final image.

8. A vehicle multi-view high-fidelity image generation system based on a single view, used to implement the vehicle multi-view high-fidelity image generation method based on a single view as claimed in any one of claims 1 to 7, characterized in that: include: An acquisition module, used to acquire a vehicle image and camera parameter information corresponding to a camera that photographs the vehicle image; A building module is used to construct the network structure of the implicit expression model; A feature extraction module is used to extract features of vehicle images to obtain shape features and texture features; A camera parameter normalization module is used to determine the geometric center of the vehicle according to the shape features, construct a normalized coordinate system with the geometric center of the vehicle as the origin, and determine the normalized parameter information of the camera in the normalized coordinate system according to the camera parameter information; A model generation module is used to determine the network parameters of the implicit expression model network according to the shape features, texture features and normalized parameter information to obtain a three-dimensional implicit expression model of the vehicle; The perspective image generation module is used to obtain the perspective image of the vehicle at a given perspective based on the three-dimensional implicit expression model.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor (10); and, a memory (11) communicatively connected to the at least one processor (10); The memory (11) stores a computer program executable by the at least one processor (10), and the computer program is executed by the at least one processor (10) so that the at least one processor (10) can execute the vehicle multi-view high-fidelity image generation method based on a single view as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer storage medium stores a computer program; when the computer program is executed by a processor, the method for generating a multi-view high-fidelity image of a vehicle based on a single view as described in any one of claims 1 to 7 is implemented.

Images