A vehicle explicit 3D reconstruction method with coarse camera pose annotation

Abstract

The application discloses a vehicle explicit three-dimensional reconstruction method under a coarse camera pose labeling condition. The method comprises the following steps: firstly, a plurality of vehicle original images of different angles of the same vehicle are preprocessed and foreground extraction is performed, so that corresponding vehicle foreground images are obtained; then, a camera coarse pose labeling module is used to obtain coarse camera pose information; finally, the plurality of vehicle foreground images of the current vehicle and the corresponding coarse camera pose information are input into a three-dimensional model reconstruction network, the coarse camera pose, basic spherical vertex coordinates, material and light parameters are modified, and after multiple iterations, the basic sphere is changed into a high-precision vehicle three-dimensional model, so that the explicit three-dimensional reconstruction of the vehicle is realized. The application uses a neural network to predict and enhance the camera pose, so that the problem that the three-dimensional reconstruction process is seriously dependent on fine camera pose labeling is solved, and the three-dimensional model of the vehicle is reconstructed explicitly, so that the shape, material and light of the three-dimensional model are effectively reconstructed.

Description

Technical Field

[0001] The present invention relates to a multi-view explicit reconstruction method in the field of three-dimensional reconstruction, and in particular to a vehicle explicit three-dimensional reconstruction method under the condition of rough camera pose annotation. Background Art

[0002] Multi-view explicit 3D reconstruction is the process of reconstructing 3D information from multiple images of the same object from different viewpoints. The reconstructed results conform to traditional 3D model definitions, such as meshes, point clouds, and voxels. This process primarily involves two stages: estimating the camera poses from different viewpoints, and then reconstructing them. As a key application branch of computer vision, 3D reconstruction has a wide range of applications and research implications, including computer-aided geometric design, computer animation, virtual reality, and augmented reality. Early 3D reconstruction methods primarily relied on pre-calibrated cameras or cameras moving along specific trajectories. This approach only allowed for 3D reconstruction in static and known environments, limiting its application. In the real world, readily available images cannot guarantee that the camera moves along a specific trajectory, and pre-calibrating camera parameters is also difficult. Without accurate camera pose calibration, current algorithms struggle to reconstruct high-precision, high-fidelity 3D models. The current mainstream method is to estimate the camera pose using limit constraints through multi-view feature point extraction and feature matching. However, the accuracy of the camera pose obtained in this way is limited by the accuracy of the feature extraction and feature matching algorithms. Even when the feature matching is highly accurate, there is still a certain gap between the estimated camera pose and the true pose, resulting in the reconstruction accuracy always being lower than that of the standard camera pose. Summary of the Invention

[0003] In order to overcome the problem of low three-dimensional reconstruction accuracy caused by the lack of accurate camera pose labeling in the real world, the present application provides a vehicle explicit three-dimensional reconstruction method under the condition of rough camera pose labeling. The rough camera pose that the reconstruction method can support refers to that compared with the true standard camera pose, the input camera pose parameters allow the translation parameter to deviate within 1m, and the rotation parameter to deviate within 10° in x, y and z directions. The present application obtains rough camera pose through a simple labeling module. Then, an image feature is extracted by using a camera pose neural network, and the rough camera pose is enhanced to obtain fine camera pose. At the same time, the image features are extracted by using shape, material and light neural networks and are encoded into the corresponding physical models. The fine camera pose, shape, material and light parameters are input into the differentiable renderer to obtain the corresponding two-dimensional image under the current view angle. The original image is used as the supervision information to repeatedly update for multiple times, and finally the accurate camera pose information and three-dimensional information such as shape, material and light can be obtained. Compared with the traditional method, the difficulty of obtaining camera parameters is greatly reduced, and the camera pose, shape, material and light neural network encoders all adopt light weight models, which can quickly and accurately perform three-dimensional reconstruction.

[0004] The technical solutions adopted by the present application are as follows:

[0005] 1) After preprocessing and foreground extraction are respectively performed on multiple vehicle original images of the same vehicle at different viewing angles, corresponding vehicle foreground images are obtained;

[0006] 2) The multiple vehicle foreground images of the current vehicle are respectively input into a camera rough pose labeling module, and the rough camera pose information corresponding to the multiple vehicle foreground images is output;

[0007] 3) A three-dimensional model reconstruction network is constructed, the multiple vehicle foreground images of the current vehicle and the corresponding rough camera pose information are input into the three-dimensional model reconstruction network for multiple iterations, and a trained three-dimensional model reconstruction network is obtained. The trained three-dimensional model reconstruction network outputs a high-precision vehicle three-dimensional model, and the explicit three-dimensional reconstruction of the vehicle is realized.

[0008] In the camera rough pose labeling module, first, the vehicle type classification is performed on the current vehicle foreground image to obtain the corresponding vehicle type. Then, the corresponding basic vehicle three-dimensional model is selected according to the vehicle type, the translation parameter and the rotation parameter of the basic vehicle three-dimensional model in the world coordinate system are adjusted, and then the rendering is performed to obtain the vehicle model rendering image. The vehicle in the rendering image is centered with the vehicle in the current foreground image, and the vehicle bottom surface is parallel. Finally, the current translation parameter and the rotation parameter of the basic vehicle three-dimensional model in the world coordinate system are taken as the rough camera pose information of the current vehicle foreground image.

[0009] The three-dimensional model reconstruction network includes a normalization layer, a matrix conversion layer, a shape neural network encoder, a material neural network encoder, a light neural network encoder, a camera neural network encoder, a basic sphere, a material model, a light model and a differentiable renderer; each vehicle foreground image is input into the normalization layer, the normalization layer is connected to the shape neural network encoder, the material neural network encoder, the light neural network encoder and the camera neural network encoder, the rough camera pose information corresponding to each vehicle foreground image is input into the matrix conversion layer, and the output of the camera neural network encoder modifies the output of the matrix conversion layer to obtain fine camera pose information. And input it into the differentiable renderer, after the output of the shape neural network encoder modifies the current spherical model, the modified spherical model is obtained and the spherical model is updated, and the updated spherical model is input into the differentiable renderer, after the output of the material neural network encoder modifies the current material model, the modified material model is obtained and the material model is updated, and the updated material model is input into the differentiable renderer, after the output of the color neural network encoder modifies the current light model, the modified light model is obtained and the light model is updated, and the updated light model is input into the differentiable renderer, the differentiable renderer renders and outputs a virtual image.

[0010] The camera neural network encoder includes a first feature extraction backbone network and two camera information encoding heads. The input of the camera neural network encoder serves as the input of the first feature extraction backbone network. The first feature extraction backbone network is connected to the two camera information encoding heads respectively. The two camera information encoding heads respectively output the offset of the translation parameter and the offset of the rotation parameter and are recorded as the camera pose compensation. The camera pose compensation serves as the output of the camera neural network encoder.

[0011] The shape neural network encoder includes a second feature extraction backbone network and two shape information encoding heads. The input of the shape neural network encoder serves as the input of the second feature extraction backbone network. The second feature extraction backbone network is connected to the two shape information encoding heads respectively. The two shape information encoding heads respectively output the effective distance field of each vertex and the corresponding vertex offset vector. The output of the two shape information encoding heads serves as the output of the shape neural network encoder.

[0012] The material neural network encoder and the light neural network encoder have the same structure, both consisting of 6 layers of fully connected layers connected in sequence.

[0013] The high-precision vehicle three-dimensional model is composed of a basic sphere, a material model and a lighting model in a trained three-dimensional model reconstruction network.

[0014] The output of the camera neural network encoder is the camera pose compensation, including the offset of the translation parameter and the offset of the rotation parameter. The translation parameter in the coarse camera pose information is added to the offset of the translation parameter to obtain the modified translation parameter; the offset of the rotation parameter is transformed to obtain the offset rotation matrix, and the offset rotation matrix is ​​added to the rotation parameter in the coarse camera pose information to obtain the modified rotation parameter. The fine camera pose information is composed of the modified translation parameter and the modified rotation parameter. The specific modification formula is as follows:

[0015] T′=(T x +ΔT x , T y +ΔT y , T z +ΔT z )

[0016] R′=R+ΔR

[0017]

[0018]

[0019] B1=N(A1)

[0020] B2=N(A2-(B1·A2)·B1)

[0021] B3=B1×B2

[0022]

[0023] Among them, T x , T y , T z are the components of the marked translation parameters in the x, y, and z directions, ΔT x , ΔT y , ΔT z are the offsets of the translation parameters in the x, y, and z axis directions, Rot out Represents the offset of the rotation parameter output by the camera pose encoder, A i The i-th row in the offset representing the rotation parameter, Rot out There are two rows, A1 and A2; Represents the 1st, 2nd, and 3rd values ​​of the i-th row of the offset of the rotation parameter, where i = 1, 2; B1, B2, and B3 are the 1st, 2nd, and 3rd columns of the offset rotation matrix of the rotation parameter, respectively. Trans represents the transpose operation, N(·) represents the operation of normalizing a vector to a unit vector, T′ and R′ are the modified translation parameter and rotation parameter, respectively. ΔR represents the offset rotation matrix of the rotation parameter, and R represents the rotation matrix form of the rotation parameter in the coarse camera pose information.

[0024] The output of the shape neural network encoder is an effective distance field and a corresponding vertex offset vector of each vertex, and the spherical model is modified according to the effective distance field and the corresponding vertex offset vector of each vertex, and the modification formula is as follows:

[0025] V k =(x+Δx k ,y+Δy k ,z+Δz k )

[0026]

[0027] Wherein, V k and V j represent the coordinates of two vertices of the same triangular plane, V k is the kth vertex, V j is the jth vertex, and xyz represent the original coordinate values of the vertex V k on the x-axis, y-axis and z-axis, respectively, and Δx k , Δy k , Δz k represent the offset of the coordinates of the vertex V k on the x-axis, y-axis and z-axis, respectively, V k and V j , S k and S j represent the effective distance field of the vertex V k and the vertex V j , and V kj represents the new vertex coordinate obtained by interpolation according to the effective distance field of the vertex V k and the vertex V j and the offset coordinate value.

[0028] The output of the material neural network encoder is material information, and the material information modifies the material model; the output of the light neural network encoder is light information, and the light information modifies the light model.

[0029] The beneficial effects of the present application mainly include:

[0030] Compared with the current mainstream multi-view explicit three-dimensional reconstruction, the present application designs a simple and fast camera coarse pose annotation information acquisition module for vehicles, uses a camera pose neural network to refine the rough camera pose, and simultaneously uses shape, material and light neural network encoders to learn the three-dimensional information corresponding to the image, iterates the basic model, and can effectively obtain a high-precision decoupled explicit three-dimensional reconstruction result.

[0031] This paper studies the influence of different camera poses on the 3D model of a vehicle and the effect of different camera poses on the 3D model. The results show that this method can be used to accurately predict and accurately predict the camera pose of a vehicle, and to find the correct camera pose for the vehicle. The new method can be used to accurately predict and predict the camera pose of a vehicle, and to find the correct camera pose for the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is an overall flow chart of the method of the present invention.

[0033] Figure 2 This is the rough camera pose pre-labeling flow chart in the present invention.

[0034] Figure 3 This is a flowchart of the reconstruction part in the present invention.

[0035] Figure 4 This is a diagram of the shape parameter encoder network structure in the present invention.

[0036] Figure 5 This is a network structure diagram of the camera parameter encoder in the present invention.

[0037] Figure 6 This is a network structure diagram of the material and lighting parameter encoder in the present invention.

[0038] Figure 7 This is a comparison diagram of the virtual data rendered by the reconstructed model in the present invention and the real image. DETAILED DESCRIPTION

[0039] In order to more clearly illustrate the purpose and technical solutions of the present invention, the present invention is further described in detail below with reference to the accompanying drawings.

[0040] like Figure 1 As shown, the present invention includes the following steps:

[0041] like Figure 2 As shown, 1) multiple original vehicle images of the same vehicle from different perspectives are preprocessed and foreground extracted to obtain corresponding vehicle foreground images. The preprocessing involves normalization to a mean of 0 and a variance of 1. A vehicle foreground image is extracted from the preprocessed vehicle images using a vehicle instance segmentation algorithm. While the original vehicle image contains both background and foreground, the vehicle foreground image contains only the foreground. The vehicle foreground image is a square image with a height of 800 pixels and a width of 800 pixels.

[0042] 2) Input multiple vehicle foreground images of the current vehicle into the camera coarse pose annotation module respectively, and output the coarse camera pose information corresponding to the multiple vehicle foreground images;

[0043] In the camera coarse pose annotation module, the current vehicle foreground image is first classified by vehicle type using a pre-trained vehicle type classification model to obtain the corresponding vehicle type. The pre-trained vehicle type classification model includes five vehicle types: sedan, hatchback, SUV, convertible sports car, and van, each corresponding to a base vehicle 3D model. Next, a base vehicle 3D model is selected based on the vehicle type, and its translation parameters (Translation, T, i.e., position coordinates in the world coordinate system) and rotation parameters (Rotation, R, i.e., Euler angles in the world coordinate system) are adjusted before rendering to obtain a rendered image of the vehicle model. The rendered image of the vehicle model has the same size as the vehicle foreground image, ensuring that the center of the vehicle in the rendered image coincides with that in the current foreground image and the vehicle bottom surfaces are parallel. The current translation and rotation parameters of the base vehicle 3D model in the world coordinate system are used as the coarse camera pose information for the current vehicle foreground image.

[0044] 3) Construct a 3D model reconstruction network, such as Figure 3 As shown, multiple vehicle foreground images of the current vehicle and the corresponding rough camera pose information are input into the 3D model reconstruction network for multiple iterations to obtain a trained 3D model reconstruction network. The trained 3D model reconstruction network outputs a high-precision 3D vehicle model. High precision is specifically defined as a Peak Signal to Noise Ratio (PSNR) value greater than 30 and a Structural Similarity (SSIM) value less than 1 between the rendered virtual image and the real image under the same camera parameters, thus achieving explicit 3D reconstruction of the vehicle.

[0045] The differentiable renderer in the 3D model reconstruction network outputs a virtual image. The virtual image is supervised by the input vehicle foreground image. The loss function is calculated based on the vehicle foreground image and the virtual image, and gradient backpropagation is performed to update the shape, material, lighting, and camera neural network encoder parameters. The loss function is the least square error (LSE), and the formula is:

[0046]

[0047] Among them, y m and y′ m Represent the color value of the mth pixel in the real image and the virtual image respectively, and n represents the total number of pixels in the virtual image.

[0048] The 3D model reconstruction network includes a normalization layer, a matrix transformation layer, a shape neural network encoder, a material neural network encoder, a lighting neural network encoder, a camera neural network encoder, a base sphere, a material model, a lighting model, and a differentiable renderer. Each vehicle foreground image is input into the normalization layer, where it is normalized to a mean of 0 and a variance of 1. The normalization layer is connected to the shape neural network encoder, the material neural network encoder, the lighting neural network encoder, and the camera neural network encoder. The coarse camera pose information corresponding to each vehicle foreground image is input into the matrix transformation layer, which transforms the rotation parameters of this coarse camera pose information from Euler angles to a rotation matrix in a right-handed coordinate system.

[0049] The formula is as follows:

[0050]

[0051] Among them, R matrix Represents the matrix form of the rotation parameters, p means that the camera rotates p° around the x-axis in the right-hand coordinate system, h means that the camera rotates h° around the y-axis in the right-hand coordinate system, and b means that the camera rotates b° around the z-axis in the right-hand coordinate system.

[0052] The output of the camera neural network encoder modifies the output of the matrix conversion layer to obtain fine camera pose information and input it into the differentiable renderer. The output of the shape neural network encoder modifies the current spherical model to obtain a modified spherical model and update the spherical model, which is then input into the differentiable renderer. The output of the material neural network encoder modifies the current material model to obtain a modified material model and update the material model, which is then input into the differentiable renderer. The output of the color neural network encoder modifies the current light model to obtain a modified light model and update the light model, which is then input into the differentiable renderer. The differentiable renderer then renders and outputs a virtual image.

[0053] like Figure 5 As shown, the camera neural network encoder includes a first feature extraction backbone network and two camera information encoding heads. The input of the camera neural network encoder serves as the input of the first feature extraction backbone network. The first feature extraction backbone network is connected to the two camera information encoding heads. The two camera information encoding heads respectively output the offset of the translation parameters and the offset of the rotation parameters, which are recorded as the camera pose compensation. The camera pose compensation serves as the output of the camera neural network encoder. The two camera information encoding heads have the same structure, both consisting of three layers of fully connected layers connected in sequence.

[0054] like Figure 4As shown, the shape neural network encoder includes a second feature extraction backbone network and two shape information encoding heads, the input of the shape neural network encoder is taken as the input of the second feature extraction backbone network, the second feature extraction backbone network is connected with the two shape information encoding heads respectively, the two shape information encoding heads respectively output the signed distance function (SDF) of each vertex and the corresponding vertex offset vector, each vertex offset vector includes the offset of the current vertex on the x, y and z axes. The output of the two shape information encoding heads is taken as the output of the shape neural network encoder. The two shape information encoding heads have the same structure and are composed of three fully connected layers and hyperbolic tangent activation layers connected in sequence. The hyperbolic tangent activation layer is used to ensure that the final output is within the range of (-1, 1). In specific implementation, the first and second feature extraction backbone networks both adopt the RepVGG-A0 network. Only the output channel number of the fully connected layer in the last layer is different in the three fully connected layers of the camera information encoding head and the three fully connected layers of the shape information encoding head, and the output parameter is set according to the corresponding output parameter.

[0055] As shown in Figure 6 , the material neural network encoder and the light neural network encoder have the same structure and are composed of six fully connected layers connected in sequence. In specific implementation, the output channel number of the last layer of the material neural network encoder and the light neural network encoder is different, and the output parameter is set according to the corresponding output parameter. The light neural network encoder outputs the light parameters required for rendering.

[0056] The high-precision vehicle three-dimensional model is composed of the base sphere, the material model and the light model in the trained three-dimensional model reconstruction network.

[0057] The output of the camera neural network encoder is the camera pose compensation amount, including the offset of the translation parameter and the offset of the rotation parameter, the modified translation parameter is obtained by adding the translation parameter in the rough camera pose information and the offset of the translation parameter; the offset rotation matrix is obtained by transforming the offset of the rotation parameter, and the modified rotation parameter is obtained by adding the offset rotation matrix and the rotation parameter in the rough camera pose information, the fine camera pose information is composed of the modified translation parameter and the modified rotation parameter, and the specific modification formula is as follows:

[0058] T′=(T x +ΔT x ,T y +ΔT y ,T z +ΔT z )

[0059] R′=R+ΔR

[0060]

[0061]

[0062] B1 = N(A1)

[0063] B2 = N(A2 - (B1 * A2) * B1)

[0064] B3 = B1 * B2

[0065]

[0066] R' = R + AR

[0067] where T x , T y , T z are the components of the annotated translation parameters in x, y, z axis direction respectively, AT x , AT y , AT z are the offsets of the translation parameters in x, y, z axis direction respectively, Rot out represents the offset of the rotation parameters output by the camera pose encoder, A i represents the i-th row in the offset of the rotation parameters, Rot out has two rows, A1 and A2, represent the first, second and third values of the i-th row of the offset of the rotation parameters, i = 1, 2 B1, B2, B3 are the first column, the second column and the third column of the offset rotation matrix of the rotation parameters respectively, Trans represents the transpose operation, represent the row vectors obtained by transposing B1, B2 and B3 respectively, N(·) represents the operation of normalizing the vector to a unit vector, T' and R' are the modified translation parameters and rotation parameters respectively, AR represents the offset rotation matrix of the rotation parameters, and R represents the rotation matrix form of the rotation parameters in the rough camera pose information.

[0068] The output of the shape neural network encoder is the effective distance field of each vertex and the corresponding vertex offset vector. According to the effective distance field of each vertex and the corresponding vertex offset vector, the spherical model is modified, and each vertex obtains an effective distance field (i.e. SDF value) and a vertex offset vector. If SDF > 0, it indicates that the vertex is outside the surface, if SDF < 0, it indicates that it is inside the surface, and if SDF = 0, it indicates that it is on the surface. The method reconstructs the target to ensure that the point is on the surface, i.e. the target SDF value is 0. The modification formula is as follows:

[0069] V k = (x + Ax k , y + Ayk , z + Δz k )

[0070]

[0071] wherein, V k and V j represent two vertex coordinates of the same triangular plane, V k is the kth vertex, V j is the jth vertex, and xyz represent the original coordinate values of the vertex V k on the x-axis, y-axis, and z-axis, respectively, Δx k , Δy k , and Δz k represent the coordinate offsets of the vertex V k on the x-axis, y-axis, and z-axis, respectively, V k and V j , S k and S j represent the effective distance fields of the vertex V k and the vertex V j , V kj represents the new vertex coordinates obtained by interpolation according to the effective distance fields of the vertex V k and the vertex V j and the offset coordinate values, and the updating of the base sphere is realized.

[0072] The output of the material neural network encoder is material information, including the offsets of color, transparency, roughness, and metallicity factors, and the material information modifies the random base material model, and the formula is as follows:

[0073] (Red', Green', Blue', Alpha')

[0074] = (Red + ΔRed, Green + ΔGredn, Blue + ΔBlue, Alpha + ΔAlpha)

[0075] (r', m') = (r + Δr, m + Δm)

[0076] Among them, (Red′, Green′, Blue′, Alpha′) represents the updated three-channel color and transparency, (r′, m′) represent the updated roughness and metalness factors respectively, Red, Green, Blue and Alpha represent the red, green, blue and transparency of the color in the original material information of the vertex respectively, r represents the transparency in the original material information, m represents the metalness factor in the original material information, ΔRed, ΔGreen, ΔBlue and ΔAlpha represent the red, green, blue and transparency offsets of the color in the material information output by the material neural network encoder respectively, Δr represents the transparency offset output by the material neural network encoder, and Δm represents the metalness factor offset output by the material neural network encoder.

[0077] The output of the lighting neural network encoder is lighting information, which modifies the random basic lighting model. In a specific implementation, the output is a high-dynamic range (HDR) lighting file.

[0078] like Figure 7 As shown, Figure 7 The left and right images in the middle are images under the same camera pose. The left image (a) is a virtual image rendered by the reconstructed vehicle model, and the right image (b) is a vehicle foreground image taken and cut out under real conditions.

Claims

1. A method for explicit 3D reconstruction of a vehicle with rough camera pose annotation, characterized by: The following steps are involved: 1) After preprocessing and foreground extraction of multiple original vehicle images from different perspectives of the same vehicle, the corresponding vehicle foreground images are obtained; 2) Input multiple vehicle foreground images of the current vehicle into the camera coarse pose annotation module respectively, and output the coarse camera pose information corresponding to the multiple vehicle foreground images; 3) Constructing a 3D model reconstruction network. Multiple vehicle foreground images and corresponding rough camera pose information are input into the 3D model reconstruction network and repeated multiple times to obtain a trained 3D model reconstruction network. The trained 3D model reconstruction network outputs a high-precision 3D model of the vehicle, achieving explicit 3D reconstruction of the vehicle. The three-dimensional model reconstruction network includes a normalization layer, a matrix conversion layer, a shape neural network encoder, a material neural network encoder, a light neural network encoder, a camera neural network encoder, a basic sphere, a material model, a light model and a differentiable renderer; each vehicle foreground image is input into the normalization layer, the normalization layer is connected to the shape neural network encoder, the material neural network encoder, the light neural network encoder and the camera neural network encoder, the rough camera pose information corresponding to each vehicle foreground image is input into the matrix conversion layer, and the output of the camera neural network encoder modifies the output of the matrix conversion layer to obtain fine camera pose information. And input it into the differentiable renderer, after the output of the shape neural network encoder modifies the current spherical model, the modified spherical model is obtained and the spherical model is updated, and the updated spherical model is input into the differentiable renderer, after the output of the material neural network encoder modifies the current material model, the modified material model is obtained and the material model is updated, and the updated material model is input into the differentiable renderer, after the output of the color neural network encoder modifies the current light model, the modified light model is obtained and the light model is updated, and the updated light model is input into the differentiable renderer, the differentiable renderer renders and outputs a virtual image.

2. The method for explicit 3D reconstruction of a vehicle with rough camera pose annotation according to claim 1, characterized in that: In the camera coarse pose annotation module, first, the vehicle type of the current vehicle foreground image is classified to obtain the corresponding vehicle type, then the corresponding basic vehicle 3D model is selected according to the vehicle type, and the translation parameters and rotation parameters of the basic vehicle 3D model in the world coordinate system are adjusted before rendering to obtain a vehicle model rendered image, so that the vehicle in the rendered image coincides with the center of the vehicle in the current foreground image and the bottom surface of the vehicle is parallel; finally, the current translation parameters and rotation parameters of the basic vehicle 3D model in the world coordinate system are used as the coarse camera pose information of the current vehicle foreground image.

3. The method for explicit 3D reconstruction of a vehicle with rough camera pose annotation according to claim 1, characterized in that: The camera neural network encoder includes a first feature extraction backbone network and two camera information encoding heads. The input of the camera neural network encoder serves as the input of the first feature extraction backbone network. The first feature extraction backbone network is connected to the two camera information encoding heads respectively. The two camera information encoding heads respectively output the offset of the translation parameter and the offset of the rotation parameter and are recorded as the camera pose compensation. The camera pose compensation serves as the output of the camera neural network encoder.

4. The method for explicit 3D reconstruction of a vehicle with rough camera pose annotation according to claim 1, wherein: The shape neural network encoder includes a second feature extraction backbone network and two shape information encoding heads. The input of the shape neural network encoder serves as the input of the second feature extraction backbone network. The second feature extraction backbone network is connected to the two shape information encoding heads respectively. The two shape information encoding heads respectively output the effective distance field of each vertex and the corresponding vertex offset vector. The output of the two shape information encoding heads serves as the output of the shape neural network encoder.

5. The method for explicit 3D reconstruction of a vehicle with rough camera pose annotation according to claim 1, wherein: The material neural network encoder and the light neural network encoder have the same structure, both consisting of 6 layers of fully connected layers connected in sequence.

6. The method for explicit 3D reconstruction of a vehicle with rough camera pose annotation according to claim 1, characterized in that: The high-precision vehicle three-dimensional model is composed of a basic sphere, a material model and a lighting model in a trained three-dimensional model reconstruction network.

7. The method for explicit 3D reconstruction of a vehicle with rough camera pose annotation according to claim 1, wherein: The output of the camera neural network encoder is the camera pose compensation, including the offset of the translation parameter and the offset of the rotation parameter. The translation parameter in the coarse camera pose information is added to the offset of the translation parameter to obtain the modified translation parameter; the offset of the rotation parameter is transformed to obtain the offset rotation matrix, and the offset rotation matrix is ​​added to the rotation parameter in the coarse camera pose information to obtain the modified rotation parameter. The fine camera pose information is composed of the modified translation parameter and the modified rotation parameter. The specific modification formula is as follows: T′=(T x +ΔT x ,T y +ΔT y ,T z +ΔT z ) R′=R+ΔR B1=N(A1) B2=N(A2-(B1·A2)·B1) B3=B1×B2 Among them, T x ,T y ,T z are the components of the labeled translation parameters in the x, y, and z directions, ΔT x ,ΔT y ,ΔT z are the offsets of the translation parameters in the x, y, and z axis directions, Rot out Represents the offset of the rotation parameter output by the camera pose encoder, A i The i-th row in the offset representing the rotation parameter, Rot out There are two rows, A1 and A2; Represents the 1st, 2nd, and 3rd values ​​of the i-th row of the offset of the rotation parameter, where i = 1 and 2; B1, B2, and B3 are the 1st, 2nd, and 3rd columns of the offset rotation matrix of the rotation parameter, respectively. Trans represents the transpose operation, N(·) represents the operation of normalizing a vector to a unit vector, T′ and R′ are the modified translation parameter and rotation parameter, respectively. ΔR represents the offset rotation matrix of the rotation parameter, and R represents the rotation matrix form of the rotation parameter in the coarse camera pose information.

8. The method for explicit 3D reconstruction of a vehicle with rough camera pose annotation according to claim 1, wherein: The output of the shape neural network encoder is the effective distance field of each vertex and the corresponding vertex offset vector. The spherical model is modified according to the effective distance field of each vertex and the corresponding vertex offset vector. The modification formula is as follows: V k =(x+Δx k ,y+Δy k ,z+Δz k ) Among them, V k and V j Represents the coordinates of two vertices of the same triangular plane. In the entire three-dimensional model, V k is the kth vertex, V j is the jth vertex, xyz represents the vertex V k The original coordinate values ​​on the x-axis, y-axis, and z-axis, Δx k ,Δy k ,Δz k Represents the vertex V k The offset of the coordinates on the x-axis, y-axis, and z-axis, S k and S j Represents vertex V k and vertex V j The effective distance field, V kj According to the vertex V k and vertex V j The new vertex coordinates are interpolated from the effective distance field and the offset coordinate values.

9. The method for explicit 3D reconstruction of a vehicle with rough camera pose annotation according to claim 1, characterized in that: The output of the material neural network encoder is material information, which modifies the material model; the output of the light neural network encoder is light information, which modifies the light model.

Images