Three-dimensional reconstruction method, device, computer equipment and storage medium

Abstract

The present application relates to a three-dimensional reconstruction method, device, computer equipment and storage medium. The method comprises: obtaining a target two-dimensional image; the target two-dimensional image comprises a target object to be three-dimensionally reconstructed; inputting the target two-dimensional image into a pre-trained target neural network to obtain a reconstruction result output by the target neural network; the reconstruction result comprises three-dimensional point cloud data of the target object; wherein the target neural network is obtained by neural network training based on a sample two-dimensional image, a sample rendered image and a sample point cloud data. The method can reduce the cost of three-dimensional reconstruction and improve the efficiency of three-dimensional reconstruction.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular to a three-dimensional reconstruction method, apparatus, computer equipment and storage medium. Background Art

[0002] 3D reconstruction refers to the simulation of 3D objects in the real world using computers. At present, 3D reconstruction technology has been widely used in medical systems, autonomous navigation, aviation and remote sensing measurement, industrial automation and other fields, bringing great convenience to people's work and life.

[0003] In the related art, the process of three-dimensional reconstruction includes: using a camera with a depth information acquisition function to capture an image of a target object, and then reconstructing the target object in three dimensions based on the image with the depth information.

[0004] However, since cameras with depth information acquisition functions are relatively expensive and difficult to operate, the cost of three-dimensional reconstruction is high and the efficiency is low. Summary of the invention

[0005] Based on this, it is necessary to provide a 3D reconstruction method, apparatus, computer equipment and storage medium that can reduce the cost of 3D reconstruction and improve efficiency in response to the above technical problems.

[0006] A three-dimensional reconstruction method, the method comprising:

[0007] Acquire a target two-dimensional image; the target two-dimensional image includes a target object to be three-dimensionally reconstructed;

[0008] Inputting the target two-dimensional image into the pre-trained target neural network to obtain a reconstruction result output by the target neural network; the reconstruction result includes three-dimensional point cloud data of the target object;

[0009] The target neural network is obtained by training the neural network based on sample two-dimensional images, sample rendering images and sample point cloud data.

[0010] In one embodiment, before inputting the target two-dimensional image into the pre-trained target neural network to obtain the reconstruction result output by the target neural network, the method further includes:

[0011] Acquire a training sample set; the training sample set includes a plurality of sample two-dimensional images, a plurality of sample rendering images, and sample point cloud data corresponding to each sample rendering image; the sample two-dimensional images include training objects;

[0012] Inputting the sample two-dimensional image, the sample rendered image and the sample point cloud data corresponding to the sample rendered image into the neural network to be trained, and obtaining the training result output by the neural network to be trained; the training result includes the three-dimensional point cloud data of the training object;

[0013] The neural network is trained based on the training results to obtain the target neural network.

[0014] In one embodiment, the neural network to be trained includes an encoding subnetwork, a decision subnetwork and a decoding subnetwork; the sample two-dimensional image, the sample rendered image and the sample point cloud data corresponding to the sample rendered image are input into the neural network to be trained to obtain the training result output by the neural network to be trained, including:

[0015] The sample two-dimensional image, the sample rendered image, and the sample point cloud data corresponding to the sample rendered image are input into the encoding subnetwork for dimension transformation, and a first latent space vector corresponding to the sample two-dimensional image, a second latent space vector corresponding to the sample rendered image, and a third latent space vector corresponding to the sample point cloud data output by the encoding subnetwork are obtained;

[0016] The first latent space vector, the second latent space vector and the third latent space vector are input into the decision sub-network for judgment, and a judgment result output by the decision sub-network is obtained; the judgment result includes whether the first latent space vector comes from the sample rendering image and whether the second latent space vector comes from the sample point cloud data;

[0017] The second latent space vector is input into the decoding sub-network to obtain the training result output by the decoding sub-network.

[0018] In one embodiment, the encoding subnetwork includes a first encoder, a second encoder, and a third encoder, and the sample two-dimensional image, the sample rendered image, and the sample point cloud data corresponding to the sample rendered image are input into the encoding subnetwork for dimension transformation, and a first latent space vector corresponding to the sample two-dimensional image, a second latent space vector corresponding to the sample rendered image, and a third latent space vector corresponding to the sample point cloud data output by the encoding subnetwork are obtained, including:

[0019] Inputting the sample two-dimensional image into the first encoder to obtain a first latent space vector output by the first encoder;

[0020] Input the sample rendered image into the second encoder to obtain a second latent space vector output by the second encoder;

[0021] The sample point cloud data is input into the third encoder to obtain a third latent space vector output by the third encoder.

[0022] In one embodiment, the decision subnetwork includes a first decider and a second decider; the first latent space vector, the second latent space vector and the third latent space vector are input into the decision subnetwork to obtain a decision result output by the decision subnetwork, including:

[0023] Inputting the first latent space vector and the second latent space vector into a first decision device to obtain a first decision result output by the first decision device; the first decision result includes whether the first latent space vector comes from a sample rendered image;

[0024] The second latent space vector and the third latent space vector are input into the second decision device to obtain a second decision result output by the second decision device; the second decision result includes whether the second latent space vector comes from the sample point cloud data.

[0025] In one embodiment, the training of the neural network based on the training results to obtain the target neural network includes:

[0026] The neural network is trained according to the first judgment result, the second judgment result and the training result to obtain a target neural network.

[0027] In one embodiment, the training of the neural network according to the first judgment result, the second judgment result and the training result to obtain the target neural network includes:

[0028] Calculate according to the training result and the sample point cloud data to obtain the chamfer distance between the training result and the sample point cloud data;

[0029] The adjustable parameters in the neural network are adjusted according to the chamfer distance, the first judgment result and the second judgment result until the chamfer distance, the first judgment result and the second judgment result meet the preset convergence conditions to obtain the target neural network.

[0030] A three-dimensional reconstruction device, comprising:

[0031] An image acquisition module, used to acquire a target two-dimensional image; the target two-dimensional image includes a target object to be three-dimensionally reconstructed;

[0032] A reconstruction module, used for inputting the target two-dimensional image into a pre-trained target neural network to obtain a reconstruction result output by the target neural network; the reconstruction result includes three-dimensional point cloud data of the target object;

[0033] The target neural network is obtained by training the neural network based on sample two-dimensional images, sample rendering images and sample point cloud data.

[0034] In one embodiment, the device further comprises:

[0035] The sample acquisition module is used to acquire a training sample set; the training sample set includes a plurality of sample two-dimensional images, a plurality of sample rendering images and sample point cloud data corresponding to each sample rendering image; the sample two-dimensional image includes a training object;

[0036] A training result acquisition module is used to input the sample two-dimensional image, the sample rendering image and the sample point cloud data corresponding to the sample rendering image into the neural network to be trained, and obtain the training result output by the neural network to be trained; the training result includes the three-dimensional point cloud data of the training object;

[0037] The training module is used to train the neural network based on the training results to obtain the target neural network.

[0038] In one embodiment, the neural network to be trained includes an encoding subnetwork, a decision subnetwork and a decoding subnetwork; and the training result obtaining module includes:

[0039] The encoding submodule is used to input the sample two-dimensional image, the sample rendered image and the sample point cloud data corresponding to the sample rendered image into the encoding subnetwork for dimensional transformation, and obtain the first latent space vector corresponding to the sample two-dimensional image, the second latent space vector corresponding to the sample rendered image and the third latent space vector corresponding to the sample point cloud data output by the encoding subnetwork;

[0040] A decision submodule is used to input the first latent space vector, the second latent space vector and the third latent space vector into the decision subnetwork for judgment, and obtain a decision result output by the decision subnetwork; the decision result includes whether the first latent space vector comes from the sample rendering image and whether the second latent space vector comes from the sample point cloud data;

[0041] The decoding submodule is used to input the second latent space vector into the decoding subnetwork to obtain the training result output by the decoding subnetwork.

[0042] In one embodiment, the encoding subnetwork includes a first encoder, a second encoder and a third encoder, and the encoding submodule is specifically used to input the sample two-dimensional image into the first encoder to obtain a first latent space vector output by the first encoder; input the sample rendered image into the second encoder to obtain a second latent space vector output by the second encoder; input the sample point cloud data into the third encoder to obtain a third latent space vector output by the third encoder.

[0043] In one embodiment, the above-mentioned decision subnetwork includes a first decider and a second decider; the above-mentioned decision submodule is specifically used to input the first latent space vector and the second latent space vector into the first decider to obtain a first decision result output by the first decider; the first decision result includes whether the first latent space vector comes from the sample rendered image; the second latent space vector and the third latent space vector are input into the second decider to obtain a second decision result output by the second decider; the second decision result includes whether the second latent space vector comes from the sample point cloud data.

[0044] In one of the embodiments, the training module is specifically used to train the neural network according to the first judgment result, the second judgment result and the training result to obtain the target neural network.

[0045] In one of the embodiments, the above-mentioned training module is specifically used to perform calculations based on the training results and the sample point cloud data to obtain the chamfer distance between the training results and the sample point cloud data; the adjustable parameters in the neural network are adjusted according to the chamfer distance, the first judgment result and the second judgment result until the chamfer distance, the first judgment result and the second judgment result meet the preset convergence conditions, thereby obtaining the target neural network.

[0046] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0047] Acquire a target two-dimensional image; the target two-dimensional image includes a target object to be three-dimensionally reconstructed;

[0048] Inputting the target two-dimensional image into the pre-trained target neural network to obtain a reconstruction result output by the target neural network; the reconstruction result includes three-dimensional point cloud data of the target object;

[0049] The target neural network is obtained by training the neural network based on sample two-dimensional images, sample rendering images and sample point cloud data.

[0050] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0051] Acquire a target two-dimensional image; the target two-dimensional image includes a target object to be three-dimensionally reconstructed;

[0052] Inputting the target two-dimensional image into the pre-trained target neural network to obtain a reconstruction result output by the target neural network; the reconstruction result includes three-dimensional point cloud data of the target object;

[0053] The target neural network is obtained by training the neural network based on sample two-dimensional images, sample rendering images and sample point cloud data.

[0054] In the above-mentioned three-dimensional reconstruction method, apparatus, computer equipment and storage medium, the server obtains a target two-dimensional image, inputs the target two-dimensional image into a pre-trained target neural network, and obtains a reconstruction result output by the target neural network. Through the embodiments of the present disclosure, only a two-dimensional image is needed to perform three-dimensional reconstruction of the target object. Since there is no need to use an expensive camera with a depth information acquisition function, the cost of three-dimensional reconstruction can be reduced. Since there is no need to collect the depth information of the target object, the efficiency of three-dimensional reconstruction can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A diagram showing an application environment of a three-dimensional reconstruction method in an embodiment;

[0056] Figure 2 is a schematic flow chart of a three-dimensional reconstruction method in one embodiment;

[0057] Figure 3 A schematic diagram of a flow chart of the steps of training a target neural network in one embodiment;

[0058] Figure 4 is one of the structural schematic diagrams of a neural network to be trained in one embodiment;

[0059] Figure 5 A schematic flow chart of the steps of obtaining a training result output by a neural network to be trained in one embodiment;

[0060] Figure 6 FIG2 is a second structural diagram of a neural network to be trained in one embodiment;

[0061] Figure 7 A flowchart of steps for training a neural network according to a first judgment result, a second judgment result and a training result to obtain a target neural network in one embodiment;

[0062] Figure 8 One of the structural block diagrams of a three-dimensional reconstruction device in one embodiment;

[0063] Fig. 9 This is a second structural block diagram of a three-dimensional reconstruction device in an embodiment;

[0064] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] The three-dimensional reconstruction method provided in this application can be applied to Figure 1 In the application environment shown. The application environment includes a terminal 102 and a server 104, and the terminal 102 communicates with the server 104 through a network. In one embodiment, the terminal 102 sends the target two-dimensional image selected by the user to the server 104, and the server receives the target two-dimensional image sent by the terminal 102. Among them, the terminal 102 can be but is not limited to various personal computers, laptops, smart phones, tablet computers and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers.

[0067] In one embodiment, Figure 2 As shown, a three-dimensional reconstruction method is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0068] Step 201: The server obtains a target two-dimensional image.

[0069] The target two-dimensional image includes a target object to be three-dimensionally reconstructed. The target object may be a pedestrian, a building, a vehicle, etc. The embodiment of the present disclosure does not limit the target object.

[0070] The terminal can communicate with the server, and the user selects a target two-dimensional image from multiple two-dimensional images stored in the terminal. After that, the terminal sends the target two-dimensional image to the server; the server receives the target two-dimensional image sent by the terminal. Alternatively, the user sends a selection instruction to the server through the terminal, and the server receives the selection instruction sent by the terminal and selects the target two-dimensional image from multiple two-dimensional images stored locally according to the selection instruction. The disclosed embodiment does not limit the method for obtaining the target two-dimensional image.

[0071] Step 202, input the target two-dimensional image into a pre-trained target neural network to obtain a reconstruction result output by the target neural network.

[0072] The reconstruction result includes three-dimensional point cloud data of the target object; the target neural network is obtained by training the neural network based on sample two-dimensional images, sample rendering images and sample point cloud data.

[0073] The server pre-trains the neural network based on the sample two-dimensional image, the sample rendered image and the sample point cloud data to obtain the target neural network. In practical applications, the target neural network can be trained by combining the sample two-dimensional image with the dataset ShapeNet or the dataset ModelNet. The above dataset includes sample rendered images and sample point cloud data. It can be understood that if the neural network is trained only based on the above dataset, the trained neural network can perform three-dimensional reconstruction based on the rendered image to obtain point cloud data. However, due to the difference between the two-dimensional image and the rendered image, the point cloud data obtained by inputting the target two-dimensional image into the neural network trained based on the above dataset will be different from the actual situation. By combining the sample two-dimensional image with the above dataset to train the neural network, the three-dimensional point cloud data of the target object output by the trained target neural network is closer to the actual situation, that is, the reconstruction result output by the target neural network is more accurate.

[0074] After the target neural network is trained, the server inputs the acquired target two-dimensional image into the target neural network. The target neural network reconstructs the target object in three dimensions based on the target two-dimensional image and outputs the reconstruction result.

[0075] In the above three-dimensional reconstruction method, the server obtains a target two-dimensional image, inputs the target two-dimensional image into a pre-trained target neural network, and obtains a reconstruction result output by the target neural network. Through the disclosed embodiment, only a two-dimensional image is needed to perform three-dimensional reconstruction of the target object. Since there is no need to use an expensive camera with a depth information acquisition function, the cost of three-dimensional reconstruction can be reduced. Since there is no need to acquire the depth information of the target object, the efficiency of three-dimensional reconstruction can be improved.

[0076] In one embodiment, Figure 3 As shown, before inputting the target two-dimensional image into the pre-trained target neural network to obtain the reconstruction result output by the target neural network, the step of training the target neural network may also be included:

[0077] Step 301: The server obtains a training sample set.

[0078] The training sample set includes a plurality of sample two-dimensional images, a plurality of sample rendering images and sample point cloud data corresponding to each sample rendering image; and the sample two-dimensional images include training objects.

[0079] The server may obtain multiple sample two-dimensional images from the terminal. For example, the terminal acquires a sample two-dimensional image by using an image acquisition device to acquire an image of the training object, and then the terminal sends the sample two-dimensional image to the server, and the server receives the sample two-dimensional image sent by the terminal. Alternatively, the server selects the sample two-dimensional image from multiple two-dimensional images stored locally. This disclosure embodiment does not limit this.

[0080] After acquiring multiple sample two-dimensional images, the server combines the multiple sample two-dimensional images with the dataset ShapeNet or the dataset ModelNet to obtain a training sample set. The sample rendering image and the sample point cloud data are one-to-one corresponding, and the training objects contained in the sample two-dimensional image may be the same as the objects corresponding to the sample image and the sample point cloud data, or may be different from the objects corresponding to the sample image and the sample point cloud data. This disclosure embodiment does not limit this.

[0081] Step 302: input the sample two-dimensional image, the sample rendered image, and the sample point cloud data corresponding to the sample rendered image into the neural network to be trained, and obtain the training result output by the neural network to be trained.

[0082] The training results include three-dimensional point cloud data of the training object.

[0083] The server inputs a sample two-dimensional image, a sample rendered image and sample point cloud data corresponding to the sample rendered image into the neural network to be trained. The neural network reconstructs the training object in three dimensions based on the input sample two-dimensional image, sample rendered image and sample point cloud data, and outputs the training result, that is, the three-dimensional point cloud data of the training object.

[0084] In practical applications, the structure of the neural network may be a generative adversarial network, which is not limited in the embodiments of the present disclosure.

[0085] Step 303: Train the neural network based on the training results to obtain a target neural network.

[0086] The server makes a judgment based on the training result output by the neural network. If the training result meets the preset convergence condition, the training is completed and the target neural network is obtained. If the training result does not meet the preset convergence condition, the adjustable parameters in the neural network are adjusted. After that, the server inputs another sample two-dimensional image, a sample rendered image, and the sample point cloud data corresponding to the sample rendered image into the neural network after adjusting the parameters to obtain the second training result output by the neural network. The server makes a judgment based on the second training result again until the training result output by the neural network meets the preset convergence condition and the target neural network is obtained.

[0087] In practical applications, the adjustable parameters in the neural network are adjusted, and the gradient can be determined and back-propagated according to the training results. This embodiment of the present disclosure does not limit this.

[0088] In the above embodiment, the server obtains a training sample set; the sample two-dimensional image, the sample rendered image, and the sample point cloud data corresponding to the sample rendered image are input into the neural network to be trained, and the training result output by the neural network to be trained is obtained. If the neural network is trained only based on the sample rendered image and the sample point cloud data, and the target two-dimensional image is input into the trained neural network, the obtained point cloud data will be different from the actual situation. In the embodiment of the present disclosure, the training sample set includes multiple sample two-dimensional images, multiple sample rendered images, and the sample point cloud data corresponding to each sample rendered image. The three-dimensional point cloud data of the target object output by the trained target neural network is closer to the actual situation, that is, the reconstruction result output by the target neural network is more accurate.

[0089] In one embodiment, Figure 4 As shown, the neural network to be trained includes an encoding subnetwork, a decision subnetwork and a decoding subnetwork; Figure 5 As shown, the step of inputting the sample two-dimensional image, the sample rendered image, and the sample point cloud data corresponding to the sample rendered image into the neural network to be trained to obtain the training result output by the neural network to be trained may include:

[0090] In step 401, the server inputs the sample two-dimensional image, the sample rendered image, and the sample point cloud data corresponding to the sample rendered image into the encoding subnetwork for dimensional transformation, and obtains the first latent space vector corresponding to the sample two-dimensional image, the second latent space vector corresponding to the sample rendered image, and the third latent space vector corresponding to the sample point cloud data output by the encoding subnetwork.

[0091] The encoding subnetwork is used to transform the dimensions of the sample two-dimensional image, the sample rendered image, and the sample point cloud data, so that the sample two-dimensional image, the sample rendered image, and the sample point cloud data are transformed into high-dimensional latent space vectors. The transformed dimension can be 124 dimensions or 512 dimensions, and the embodiment of the present disclosure does not limit the transformed dimension.

[0092] In one embodiment, if Figure 6 As shown, the encoding subnetwork includes a first encoder, a second encoder and a third encoder. The sample two-dimensional image is input into the first encoder to obtain a first latent space vector output by the first encoder; the sample rendered image is input into the second encoder to obtain a second latent space vector output by the second encoder; the sample point cloud data is input into the third encoder to obtain a third latent space vector output by the third encoder.

[0093] Step 402: input the first latent space vector, the second latent space vector and the third latent space vector into the decision sub-network for judgment, and obtain the judgment result output by the decision sub-network.

[0094] The judgment result includes whether the first latent space vector comes from the sample rendered image and whether the second latent space vector comes from the sample point cloud data.

[0095] The server inputs the first latent space vector, the second latent space vector and the third latent space vector into the decision subnetwork. The decision subnetwork determines whether the first latent space vector comes from the sample rendered image based on the first latent space vector and the second latent space vector, and determines whether the second latent space vector comes from the sample point cloud data based on the second latent space vector and the third latent space vector, and outputs the decision result based on the judgment. In this process, the adaptation of the two-dimensional image and the rendered image in the latent space domain, as well as the adaptation of the rendered image and the point cloud data in the latent space domain are realized. That is, the decision subnetwork makes the first latent space vector close to the second latent space vector, makes the second latent space vector close to the third latent space vector, and finally makes the first latent space vector close to the third latent space vector, so that the point cloud data output by the target neural network for three-dimensional reconstruction based on the two-dimensional image can be more in line with the actual situation and more accurate.

[0096] In one embodiment, if Figure 6 As shown, the decision subnetwork includes a first decision device and a second decision device; the first latent space vector and the second latent space vector are input into the first decision device to obtain a first decision result output by the first decision device; the second latent space vector and the third latent space vector are input into the second decision device to obtain a second decision result output by the second decision device.

[0097] The first judgment result includes whether the first latent space vector comes from the sample rendered image; the second judgment result includes whether the second latent space vector comes from the sample point cloud data.

[0098] Step 403: input the second latent space vector into the decoding sub-network to obtain the training result output by the decoding sub-network.

[0099] The server inputs the second latent space vector into the decoding subnetwork, which performs dimensionality reduction on the second latent space vector and outputs the training result. In practical applications, the training result output by the decoding subnetwork is point cloud data in the form of an N*3 matrix, where N is the number of point clouds and 3 is the x-coordinate, y-coordinate, and z-coordinate of the point cloud point.

[0100] In one embodiment, if Figure 6 As shown, the decoding subnetwork includes a decoder, and the second latent space vector is input into the decoder to obtain a training result output by the decoder.

[0101] like Figure 6 As shown, the first encoder, the second encoder, the first decision device and the decoder constitute a generative adversarial neural network, and the second encoder, the third encoder, the second decision device and the decoder constitute another generative adversarial neural network.

[0102] In the process of inputting the sample two-dimensional image, the sample rendered image and the sample point cloud data corresponding to the sample rendered image into the neural network to be trained and obtaining the training result output by the neural network to be trained, the server inputs the sample two-dimensional image, the sample rendered image and the sample point cloud data corresponding to the sample rendered image into the encoding subnetwork for dimensional transformation, and obtains the first latent space vector corresponding to the sample two-dimensional image output by the encoding subnetwork, the second latent space vector corresponding to the sample rendered image and the third latent space vector corresponding to the sample point cloud data; inputs the first latent space vector, the second latent space vector and the third latent space vector into the decision subnetwork for judgment, and obtains the decision result output by the decision subnetwork; inputs the second latent space vector into the decoding subnetwork, and obtains the training result output by the decoding subnetwork. Through the embodiment of the present invention, the adaptation of the two-dimensional image and the rendered image in the latent space domain, as well as the adaptation of the rendered image and the point cloud data in the latent space domain are realized, so that the point cloud data output by the target neural network for three-dimensional reconstruction based on the two-dimensional image can be more in line with the actual situation and more accurate.

[0103] In one embodiment, training the neural network based on the training result to obtain the target neural network may include: training the neural network according to the first judgment result, the second judgment result and the training result to obtain the target neural network.

[0104] In actual applications, the first decision device outputs a first decision result, the second decision device outputs a second decision result, and the decoder outputs a training result; thereafter, the server determines whether the first decision result, the second decision result, and the training result meet a preset convergence condition, and if they do not meet the preset convergence condition, the gradient is determined and back propagation is performed to train the neural network; if they meet the preset convergence condition, the target neural network is obtained. The embodiments of the present disclosure do not limit the preset convergence condition.

[0105] In one embodiment, if Figure 7 As shown, the step of training the neural network according to the first judgment result, the second judgment result and the training result to obtain the target neural network may include:

[0106] Step 501: The server performs calculations based on the training results and the sample point cloud data to obtain the chamfer distance between the training results and the sample point cloud data.

[0107] Among them, the chamfer distance in three-dimensional space is mainly used for three-dimensional reconstruction. The larger the chamfer distance, the greater the difference between the two sets of point cloud data; the smaller the chamfer distance, the smaller the difference between the two sets of point cloud data, that is, the better the reconstruction effect.

[0108] Step 502, adjusting the adjustable parameters in the neural network according to the chamfer distance, the first judgment result and the second judgment result until the chamfer distance, the first judgment result and the second judgment result meet the preset convergence conditions, thereby obtaining the target neural network.

[0109] It can be understood that the first judgment result and the second judgment result meet the preset convergence conditions, indicating that the first latent space vector is relatively close to the second latent space vector, and the second latent space vector is relatively close to the third latent space vector, that is, the first latent space vector is relatively close to the third latent space vector; and the chamfer distance meets the preset convergence conditions, indicating that the point cloud data obtained by the decoder through three-dimensional reconstruction based on the sample rendered image is slightly different from the sample point cloud data, and the reconstruction effect is better.

[0110] The neural network is trained according to the first judgment result, the second judgment result and the training result to obtain the target neural network. The server calculates according to the training result and the sample point cloud data to obtain the chamfer distance between the training result and the sample point cloud data. The adjustable parameters in the neural network are adjusted according to the chamfer distance, the first judgment result and the second judgment result until the chamfer distance, the first judgment result and the second judgment result meet the preset convergence conditions to obtain the target neural network. Through the embodiments of the present disclosure, not only can the target neural network achieve domain adaptation in the three-dimensional reconstruction process, but also the training result output by the target neural network can achieve better reconstruction effect.

[0111] It should be understood that although Figure 2-Figure 7 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 7 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0112] In one embodiment, Figure 8 As shown, a three-dimensional reconstruction device is provided, comprising:

[0113] An image acquisition module 601 is used to acquire a target two-dimensional image; the target two-dimensional image includes a target object to be three-dimensionally reconstructed;

[0114] A reconstruction module 602 is used to input the target two-dimensional image into a pre-trained target neural network to obtain a reconstruction result output by the target neural network; the reconstruction result includes three-dimensional point cloud data of the target object;

[0115] The target neural network is obtained by training the neural network based on sample two-dimensional images, sample rendering images and sample point cloud data.

[0116] In one embodiment, if Fig. 9 As shown, the device also includes:

[0117] The sample acquisition module 603 is used to acquire a training sample set; the training sample set includes a plurality of sample two-dimensional images, a plurality of sample rendering images and sample point cloud data corresponding to each sample rendering image; the sample two-dimensional image includes a training object;

[0118] The training result obtaining module 604 is used to input the sample two-dimensional image, the sample rendering image and the sample point cloud data corresponding to the sample rendering image into the neural network to be trained, and obtain the training result output by the neural network to be trained; the training result includes the three-dimensional point cloud data of the training object;

[0119] The training module 605 is used to train the neural network based on the training results to obtain the target neural network.

[0120] In one embodiment, the neural network to be trained includes an encoding subnetwork, a decision subnetwork and a decoding subnetwork; the training result obtaining module 604 includes:

[0121] The encoding submodule is used to input the sample two-dimensional image, the sample rendered image and the sample point cloud data corresponding to the sample rendered image into the encoding subnetwork for dimensional transformation, and obtain the first latent space vector corresponding to the sample two-dimensional image, the second latent space vector corresponding to the sample rendered image and the third latent space vector corresponding to the sample point cloud data output by the encoding subnetwork;

[0122] A decision submodule is used to input the first latent space vector, the second latent space vector and the third latent space vector into the decision subnetwork for judgment, and obtain a decision result output by the decision subnetwork; the decision result includes whether the first latent space vector comes from the sample rendering image and whether the second latent space vector comes from the sample point cloud data;

[0123] The decoding submodule is used to input the second latent space vector into the decoding subnetwork to obtain the training result output by the decoding subnetwork.

[0124] In one embodiment, the encoding subnetwork includes a first encoder, a second encoder and a third encoder, and the encoding submodule is specifically used to input the sample two-dimensional image into the first encoder to obtain a first latent space vector output by the first encoder; input the sample rendered image into the second encoder to obtain a second latent space vector output by the second encoder; input the sample point cloud data into the third encoder to obtain a third latent space vector output by the third encoder.

[0125] In one embodiment, the above-mentioned decision subnetwork includes a first decider and a second decider; the above-mentioned decision submodule is specifically used to input the first latent space vector and the second latent space vector into the first decider to obtain a first decision result output by the first decider; the first decision result includes whether the first latent space vector comes from the sample rendered image; the second latent space vector and the third latent space vector are input into the second decider to obtain a second decision result output by the second decider; the second decision result includes whether the second latent space vector comes from the sample point cloud data.

[0126] In one of the embodiments, the training module is specifically used to train the neural network according to the first judgment result, the second judgment result and the training result to obtain the target neural network.

[0127] In one of the embodiments, the above-mentioned training module is specifically used to perform calculations based on the training results and the sample point cloud data to obtain the chamfer distance between the training results and the sample point cloud data; the adjustable parameters in the neural network are adjusted according to the chamfer distance, the first judgment result and the second judgment result until the chamfer distance, the first judgment result and the second judgment result meet the preset convergence conditions, thereby obtaining the target neural network.

[0128] For the specific definition of the three-dimensional reconstruction device, please refer to the definition of the three-dimensional reconstruction method above, which will not be repeated here. Each module in the above three-dimensional reconstruction device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0129] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store three-dimensional reconstruction data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional reconstruction method is implemented.

[0130] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0131] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0132] Acquire a target two-dimensional image; the target two-dimensional image includes a target object to be three-dimensionally reconstructed;

[0133] Inputting the target two-dimensional image into the pre-trained target neural network to obtain a reconstruction result output by the target neural network; the reconstruction result includes three-dimensional point cloud data of the target object;

[0134] The target neural network is obtained by training the neural network based on sample two-dimensional images, sample rendering images and sample point cloud data.

[0135] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0136] Acquire a training sample set; the training sample set includes a plurality of sample two-dimensional images, a plurality of sample rendering images, and sample point cloud data corresponding to each sample rendering image; the sample two-dimensional images include training objects;

[0137] Inputting the sample two-dimensional image, the sample rendered image and the sample point cloud data corresponding to the sample rendered image into the neural network to be trained, and obtaining the training result output by the neural network to be trained; the training result includes the three-dimensional point cloud data of the training object;

[0138] The neural network is trained based on the training results to obtain the target neural network.

[0139] In one embodiment, the neural network to be trained includes an encoding subnetwork, a decision subnetwork and a decoding subnetwork; when the processor executes the computer program, the following steps are also implemented:

[0140] The sample two-dimensional image, the sample rendered image, and the sample point cloud data corresponding to the sample rendered image are input into the encoding subnetwork for dimension transformation, and a first latent space vector corresponding to the sample two-dimensional image, a second latent space vector corresponding to the sample rendered image, and a third latent space vector corresponding to the sample point cloud data output by the encoding subnetwork are obtained;

[0141] The first latent space vector, the second latent space vector and the third latent space vector are input into the decision sub-network for judgment, and a judgment result output by the decision sub-network is obtained; the judgment result includes whether the first latent space vector comes from the sample rendering image and whether the second latent space vector comes from the sample point cloud data;

[0142] The second latent space vector is input into the decoding sub-network to obtain the training result output by the decoding sub-network.

[0143] In one embodiment, the encoding subnetwork includes a first encoder, a second encoder, and a third encoder, and the processor further implements the following steps when executing the computer program:

[0144] Inputting the sample two-dimensional image into the first encoder to obtain a first latent space vector output by the first encoder;

[0145] Input the sample rendered image into the second encoder to obtain a second latent space vector output by the second encoder;

[0146] The sample point cloud data is input into the third encoder to obtain a third latent space vector output by the third encoder.

[0147] In one embodiment, the decision subnetwork includes a first decider and a second decider; when the processor executes the computer program, the processor further implements the following steps:

[0148] Inputting the first latent space vector and the second latent space vector into a first decision device to obtain a first decision result output by the first decision device; the first decision result includes whether the first latent space vector comes from a sample rendered image;

[0149] The second latent space vector and the third latent space vector are input into the second decision device to obtain a second decision result output by the second decision device; the second decision result includes whether the second latent space vector comes from the sample point cloud data.

[0150] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0151] The neural network is trained according to the first judgment result, the second judgment result and the training result to obtain a target neural network.

[0152] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0153] Calculate according to the training result and the sample point cloud data to obtain the chamfer distance between the training result and the sample point cloud data;

[0154] The adjustable parameters in the neural network are adjusted according to the chamfer distance, the first judgment result and the second judgment result until the chamfer distance, the first judgment result and the second judgment result meet the preset convergence conditions to obtain the target neural network.

[0155] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0156] Acquire a target two-dimensional image; the target two-dimensional image includes a target object to be three-dimensionally reconstructed;

[0157] Inputting the target two-dimensional image into the pre-trained target neural network to obtain a reconstruction result output by the target neural network; the reconstruction result includes three-dimensional point cloud data of the target object;

[0158] The target neural network is obtained by training the neural network based on sample two-dimensional images, sample rendering images and sample point cloud data.

[0159] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0160] Acquire a training sample set; the training sample set includes a plurality of sample two-dimensional images, a plurality of sample rendering images, and sample point cloud data corresponding to each sample rendering image; the sample two-dimensional images include training objects;

[0161] Inputting the sample two-dimensional image, the sample rendered image and the sample point cloud data corresponding to the sample rendered image into the neural network to be trained, and obtaining the training result output by the neural network to be trained; the training result includes the three-dimensional point cloud data of the training object;

[0162] The neural network is trained based on the training results to obtain the target neural network.

[0163] In one embodiment, the neural network to be trained includes an encoding subnetwork, a decision subnetwork and a decoding subnetwork; when the computer program is executed by a processor, the following steps are also implemented:

[0164] The sample two-dimensional image, the sample rendered image, and the sample point cloud data corresponding to the sample rendered image are input into the encoding subnetwork for dimension transformation, and a first latent space vector corresponding to the sample two-dimensional image, a second latent space vector corresponding to the sample rendered image, and a third latent space vector corresponding to the sample point cloud data output by the encoding subnetwork are obtained;

[0165] The first latent space vector, the second latent space vector and the third latent space vector are input into the decision sub-network for judgment, and a judgment result output by the decision sub-network is obtained; the judgment result includes whether the first latent space vector comes from the sample rendering image and whether the second latent space vector comes from the sample point cloud data;

[0166] The second latent space vector is input into the decoding sub-network to obtain the training result output by the decoding sub-network.

[0167] In one embodiment, the encoding subnetwork includes a first encoder, a second encoder, and a third encoder, and when the computer program is executed by a processor, the following steps are further implemented:

[0168] Inputting the sample two-dimensional image into the first encoder to obtain a first latent space vector output by the first encoder;

[0169] Input the sample rendered image into the second encoder to obtain a second latent space vector output by the second encoder;

[0170] The sample point cloud data is input into the third encoder to obtain a third latent space vector output by the third encoder.

[0171] In one embodiment, the decision subnetwork includes a first decider and a second decider; when the computer program is executed by a processor, the following steps are also implemented:

[0172] Inputting the first latent space vector and the second latent space vector into a first decision device to obtain a first decision result output by the first decision device; the first decision result includes whether the first latent space vector comes from a sample rendered image;

[0173] The second latent space vector and the third latent space vector are input into the second decision device to obtain a second decision result output by the second decision device; the second decision result includes whether the second latent space vector comes from the sample point cloud data.

[0174] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0175] The neural network is trained according to the first judgment result, the second judgment result and the training result to obtain a target neural network.

[0176] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0177] Calculate according to the training result and the sample point cloud data to obtain the chamfer distance between the training result and the sample point cloud data;

[0178] The adjustable parameters in the neural network are adjusted according to the chamfer distance, the first judgment result and the second judgment result until the chamfer distance, the first judgment result and the second judgment result meet the preset convergence conditions to obtain the target neural network.

[0179] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0180] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0181] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A three-dimensional reconstruction method, characterized in that: The method comprises: Acquire a training sample set; the training sample set includes a plurality of sample two-dimensional images, a plurality of sample rendering images and sample point cloud data corresponding to each of the sample rendering images; the sample two-dimensional images include training objects; Inputting the sample two-dimensional image, the sample rendered image and the sample point cloud data corresponding to the sample rendered image into the neural network to be trained, and obtaining a training result output by the neural network to be trained; the training result includes the three-dimensional point cloud data of the training object; Training the neural network based on the training results to obtain a target neural network; Acquire a target two-dimensional image; the target two-dimensional image includes a target object to be three-dimensionally reconstructed; Inputting the target two-dimensional image into a pre-trained target neural network to obtain a reconstruction result output by the target neural network; the reconstruction result includes three-dimensional point cloud data of the target object; Wherein, the target neural network is obtained by performing neural network training based on sample two-dimensional images, sample rendering images and sample point cloud data; The neural network to be trained includes a coding subnetwork, a decision subnetwork and a decoding subnetwork; the step of inputting the sample two-dimensional image, the sample rendered image and the sample point cloud data corresponding to the sample rendered image into the neural network to be trained to obtain the training result output by the neural network to be trained includes: Input the sample two-dimensional image, the sample rendered image, and the sample point cloud data corresponding to the sample rendered image into the encoding subnetwork for dimension transformation, and obtain a first latent space vector corresponding to the sample two-dimensional image, a second latent space vector corresponding to the sample rendered image, and a third latent space vector corresponding to the sample point cloud data output by the encoding subnetwork; Inputting the first latent space vector, the second latent space vector and the third latent space vector into the decision subnetwork for judgment, and obtaining a judgment result output by the decision subnetwork; the judgment result includes whether the first latent space vector comes from the sample rendered image and whether the second latent space vector comes from the sample point cloud data; The second latent space vector is input into the decoding subnetwork to obtain the training result output by the decoding subnetwork.

2. The method according to claim 1, characterized in that The encoding subnetwork includes a first encoder, a second encoder, and a third encoder. The sample two-dimensional image, the sample rendered image, and the sample point cloud data corresponding to the sample rendered image are input into the encoding subnetwork for dimension transformation, and a first latent space vector corresponding to the sample two-dimensional image, a second latent space vector corresponding to the sample rendered image, and a third latent space vector corresponding to the sample point cloud data output by the encoding subnetwork are obtained, including: Inputting the sample two-dimensional image into the first encoder to obtain the first latent space vector output by the first encoder; Inputting the sample rendered image into the second encoder to obtain the second latent space vector output by the second encoder; The sample point cloud data is input into the third encoder to obtain the third latent space vector output by the third encoder.

3. The method according to claim 1, characterized in that The decision subnetwork includes a first decision device and a second decision device; the first latent space vector, the second latent space vector and the third latent space vector are input into the decision subnetwork to obtain a decision result output by the decision subnetwork, including: Inputting the first latent space vector and the second latent space vector into the first decision device to obtain a first decision result output by the first decision device; the first decision result includes whether the first latent space vector comes from the sample rendered image; The second latent space vector and the third latent space vector are input into the second decision device to obtain a second decision result output by the second decision device; the second decision result includes whether the second latent space vector comes from the sample point cloud data.

4. The method according to claim 3, characterized in that The training of the neural network based on the training result to obtain the target neural network includes: The neural network is trained according to the first judgment result, the second judgment result and the training result to obtain the target neural network.

5. The method according to claim 4, characterized in that The training of the neural network according to the first judgment result, the second judgment result and the training result to obtain the target neural network includes: Calculating according to the training result and the sample point cloud data to obtain a chamfer distance between the training result and the sample point cloud data; The adjustable parameters in the neural network are adjusted according to the chamfer distance, the first judgment result and the second judgment result until the chamfer distance, the first judgment result and the second judgment result meet the preset convergence conditions, thereby obtaining the target neural network.

6. A three-dimensional reconstruction device, characterized in that: The device comprises: The sample acquisition module is used to acquire a training sample set; the training sample set includes a plurality of sample two-dimensional images, a plurality of sample rendering images and sample point cloud data corresponding to each sample rendering image; the sample two-dimensional image includes a training object; A training result acquisition module is used to input the sample two-dimensional image, the sample rendering image and the sample point cloud data corresponding to the sample rendering image into the neural network to be trained, and obtain the training result output by the neural network to be trained; the training result includes the three-dimensional point cloud data of the training object; A training module, used to train the neural network based on the training results to obtain a target neural network; An image acquisition module, used to acquire a target two-dimensional image; the target two-dimensional image includes a target object to be three-dimensionally reconstructed; A reconstruction module, used for inputting the target two-dimensional image into a pre-trained target neural network to obtain a reconstruction result output by the target neural network; the reconstruction result includes three-dimensional point cloud data of the target object; Wherein, the target neural network is obtained by performing neural network training based on sample two-dimensional images, sample rendering images and sample point cloud data; The neural network to be trained includes a coding subnetwork, a decision subnetwork and a decoding subnetwork; and the training result acquisition module includes: The encoding submodule is used to input the sample two-dimensional image, the sample rendered image and the sample point cloud data corresponding to the sample rendered image into the encoding subnetwork for dimensional transformation, and obtain the first latent space vector corresponding to the sample two-dimensional image, the second latent space vector corresponding to the sample rendered image and the third latent space vector corresponding to the sample point cloud data output by the encoding subnetwork; A decision submodule is used to input the first latent space vector, the second latent space vector and the third latent space vector into the decision subnetwork for judgment, and obtain a decision result output by the decision subnetwork; the decision result includes whether the first latent space vector comes from the sample rendering image and whether the second latent space vector comes from the sample point cloud data; The decoding submodule is used to input the second latent space vector into the decoding subnetwork to obtain the training result output by the decoding subnetwork.

7. The device according to claim 6, characterized in that The encoding subnetwork includes a first encoder, a second encoder and a third encoder; the encoding submodule is specifically used to input the sample two-dimensional image into the first encoder to obtain a first latent space vector output by the first encoder; input the sample rendered image into the second encoder to obtain a second latent space vector output by the second encoder; input the sample point cloud data into the third encoder to obtain a third latent space vector output by the third encoder.

8. The device according to claim 6, characterized in that The decision subnetwork includes a first decider and a second decider; the decision submodule is specifically used to input the first latent space vector and the second latent space vector into the first decider to obtain a first decision result output by the first decider; the first decision result includes whether the first latent space vector comes from a sample rendered image; the second latent space vector and the third latent space vector are input into the second decider to obtain a second decision result output by the second decider; the second decision result includes whether the second latent space vector comes from sample point cloud data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images