3D Reconstruction Method, System, Device and Storage Medium Based on 2D Images

Abstract

The present invention provides a three-dimensional reconstruction method, system, device and storage medium based on two-dimensional images. The method includes: collecting two-dimensional images; performing dimensionality reduction encoding on the input images to obtain a latent space tensor; performing dimensionality increase encoding on the latent space tensor to generate three feature planes, and obtaining orthogonal three-feature planes through orthogonal arrangement; and rendering the orthogonal three-feature planes according to the device parameters of the two-dimensional image acquisition device that captures the two-dimensional images to obtain three-dimensional perspective information corresponding to the two-dimensional images, where the three-dimensional perspective information includes multi-angle images and / or three-dimensional geometric structures. The present invention can make up for the information gap in the process of two-dimensional image to three-dimensional reconstruction by using a generative adversarial network, realize the feasibility of three-dimensional reconstruction from two-dimensional images, and expand the usage scenarios and functionality of two-dimensional images.

Description

Background Art

[0002] With the continuous development of the Internet, more and more scenarios have been moved onto the virtual world platform, such as online meetings, remote online courses, online shopping, etc. At the same time, the virtual world platform is gradually transitioning from two-dimensional plane displays to three-dimensional. Such as future technologies like digital humans, digital twins, and the metaverse. The development of these new technologies currently highly depends on modelers for repetitive modeling work, which is time-consuming and laborious. Therefore, how to obtain a three-dimensional model from two-dimensional images through algorithms has become an urgent problem to be solved. Currently, most algorithmic techniques rely on special hardware devices, such as high-cost radars or commercial-grade depth cameras, etc. These have erected technical barriers for acquisition tasks and cannot be widely applied on a large scale. The following problems exist in the existing related technologies:

[0003] (1) Rely on special hardware, such as radars, commercial-grade depth cameras, etc.

[0004] (2) Have special requirements for the acquisition angle and high requirements for acquisition personnel.

[0005] (3) The algorithm structure is not reasonable enough, and there is no special improvement for the rendering effect in 3D reconstruction. The reconstruction result may not be sufficient for practical application.

[0006] (4) The accuracy of existing face recognition only through two-dimensional pictures is not high. Using depth images for face recognition requires additional hardware, which is accurate but too costly.

[0007] For example: The existing patent solution is a 3D semantic scene reconstruction method based on a generative adversarial network. It encodes the input SUNCG-RGBD, that is, image data containing voxel information and depth information, to obtain corresponding encoded data. Then, the encoded data is input into the generative adversarial network to output a 3D semantic scene. This method requires special hardware support, namely a depth camera and a device that can collect voxel information, which is expensive and has a high learning cost. This method does not have promotional value.

[0008] In view of this, the present invention proposes a 3D reconstruction method, system, device, and storage medium based on two-dimensional images.

[0009] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present invention. Therefore, it may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0010] In view of the problems in the prior art, the purpose of the present invention is to provide a three-dimensional reconstruction method, system, device and storage medium based on two-dimensional images, which overcomes the difficulties of the prior art, can make up for the information gap in the process of two-dimensional image to three-dimensional reconstruction by using a generative adversarial network, realizes the feasibility of three-dimensional reconstruction from two-dimensional images, and expands the usage scenarios and functionality of two-dimensional images.

[0011] An embodiment of the present invention provides a three-dimensional reconstruction method based on two-dimensional images, including the following steps:

[0012] Collect two-dimensional images;

[0013] Perform dimensionality reduction encoding on the input image to obtain a latent space tensor;

[0014] Perform dimensionality increase encoding on the latent space tensor to generate three feature planes, and obtain orthogonal three-feature planes through orthogonal arrangement; and

[0015] Render the orthogonal three-feature planes according to the device parameters of the two-dimensional image acquisition device that captured the two-dimensional image to obtain the three-dimensional perspective information corresponding to the two-dimensional image, where the three-dimensional perspective information includes multi-angle images and / or three-dimensional geometric structures.

[0016] Preferably, the collecting of two-dimensional images includes:

[0017] Capture a two-dimensional image through a two-dimensional image acquisition device;

[0018] Collect the two-dimensional image information and the camera parameters of the two-dimensional image acquisition device.

[0019] Preferably, the performing of dimensionality reduction encoding on the input image to obtain a latent space tensor includes:

[0020] Input the input image into an encoder;

[0021] Perform dimensionality reduction encoding on the input image through the encoder;

[0022] The encoder outputs the encoded latent space tensor corresponding to the input image.

[0023] Preferably, the performing of dimensionality increase encoding on the latent space tensor to generate three feature planes and obtaining orthogonal three-feature planes through orthogonal arrangement includes:

[0024] Input the latent space tensor into a generator

[0025] Perform dimensionality increase processing on the latent space tensor through the generator;

[0026] The generator outputs three feature planes;

[0027] An orthogonal three-feature plane is obtained through orthogonal arrangement, and the feature planes are orthogonal to each other.

[0028] Preferably, rendering the orthogonal three-feature plane according to the camera parameters of the two-dimensional image acquisition device that captures the two-dimensional image to obtain the three-dimensional perspective information corresponding to the two-dimensional image, including:

[0029] Inputting the three feature planes into a neural rendering module for decoding to obtain a latent space tensor;

[0030] Rendering the latent space tensor according to the camera parameters of the two-dimensional image acquisition device that captures the two-dimensional image to obtain the three-dimensional perspective information corresponding to the two-dimensional image, and the three-dimensional perspective information includes multi-angle images and / or three-dimensional geometric structures.

[0031] Preferably, rendering the orthogonal three-feature plane according to the camera parameters of the two-dimensional image acquisition device that captures the two-dimensional image to obtain the three-dimensional perspective information corresponding to the two-dimensional image, further includes:

[0032] The neural rendering module includes a decoder and a renderer. The decoder is composed of a multi-layer perceptron MLP and is used for decoding the input three-feature plane to obtain a latent space tensor. The renderer is composed of a fully connected layer and renders the latent space tensor into three-dimensional perspective information and outputs it.

[0033] Preferably, it further includes:

[0034] Performing discrimination on the multi-angle images to obtain a loss value, obtaining a gradient according to the loss value, and updating parameters according to the gradient.

[0035] Preferably, the performing discrimination on the multi-angle images to obtain a loss value, obtaining a gradient according to the loss value, and updating parameters according to the gradient includes:

[0036] Performing image discrimination on the multi-angle images through a discriminator to obtain a loss value of the reconstruction process. The discriminator and the generator used to generate the feature plane jointly form an adversarial network structure, and both the encoder and the discriminator are composed of a VGG-16 network structure;

[0037] Obtaining a gradient according to the loss value;

[0038] Updating the parameters of the adversarial network according to the gradient.

[0039] Preferably, it further includes:

[0040] The two-dimensional image is a face image, and user authentication is performed through the first similarity between the two-dimensional image and a preset face image and the second similarity between the three-dimensional perspective information and a preset three-dimensional perspective information.

[0041] An embodiment of the present invention further provides a three-dimensional reconstruction system based on a two-dimensional image, which is used to implement the above-mentioned three-dimensional reconstruction method based on a two-dimensional image. The three-dimensional reconstruction system based on a two-dimensional image includes:

[0042] An image acquisition module, which acquires two-dimensional images;

[0043] A latent space tensor module, which performs dimensionality reduction encoding on the input image to obtain a latent space tensor;

[0044] An orthogonal plane module, which performs dimensionality increase encoding on the latent space tensor to generate three feature planes, and obtains orthogonal three-feature planes through orthogonal arrangement;

[0045] A three-dimensional information module, which renders the orthogonal three-feature planes according to the device parameters of the two-dimensional image acquisition device that captures the two-dimensional image, and obtains three-dimensional perspective information corresponding to the two-dimensional image. The three-dimensional perspective information includes multi-angle images and / or three-dimensional geometric structures.

[0046] An embodiment of the present invention further provides a three-dimensional reconstruction device based on a two-dimensional image, including:

[0047] A processor;

[0048] A memory, in which executable instructions of the processor are stored;

[0049] Wherein, the processor is configured to execute the steps of the above-mentioned three-dimensional reconstruction method based on a two-dimensional image by executing the executable instructions.

[0050] An embodiment of the present invention further provides a computer-readable storage medium, which is used to store a program, and when the program is executed, the steps of the above-mentioned three-dimensional reconstruction method based on a two-dimensional image are implemented.

[0051] The purpose of the present invention is to provide a three-dimensional reconstruction method, system, device and storage medium based on a two-dimensional image, which can make up for the information gap in the process of two-dimensional image to three-dimensional reconstruction by using a generative adversarial network, realize the feasibility of two-dimensional image to three-dimensional reconstruction, and expand the usage scenarios and functionality of two-dimensional images. Description of the Drawings

[0052] By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, objects and advantages of the present invention will become more obvious.

[0053] Figure 1 It is a flowchart of the first embodiment of the three-dimensional reconstruction method based on a two-dimensional image of the present invention.

[0054] Figure 2It is a schematic flowchart of step S110 in the first embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention.

[0055] Figure 3 It is a schematic flowchart of step S120 in the first embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention.

[0056] Figure 4 It is a schematic flowchart of step S130 in the first embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention.

[0057] Figure 5 It is a schematic flowchart of step S140 in the first embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention.

[0058] Figure 6 It is a flowchart of the second embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention.

[0059] Figure 7 It is a schematic flowchart of step S150 in the second embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention.

[0060] Figure 8 It is a flowchart of the third embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention.

[0061] Figure 9 It is a schematic flowchart of step S150 in the third embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention.

[0062] Figure 10 It is a schematic diagram of the modules of the first embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention.

[0063] Figure 11 It is a schematic diagram of the image acquisition module in the first embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention.

[0064] Figure 12 It is a schematic diagram of the latent space tensor module in the first embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention.

[0065] Figure 13 It is a schematic diagram of the orthogonal plane module in the first embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention.

[0066] Figure 14 It is a schematic diagram of the three-dimensional information module in the first embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention.

[0067] Figure 15It is a schematic diagram of the parameter update module in the second embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention.

[0068] Figure 16 It is a schematic diagram of the module of the second embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention.

[0069] Figure 17 It is a schematic diagram of the module of the third embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention.

[0070] Figure 18 It is a schematic diagram of the user authentication module in the third embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention.

[0071] Figure 19 It is a schematic diagram of the implementation process of the three-dimensional reconstruction method based on two-dimensional images of the present invention.

[0072] Figure 20 It is a step flow chart of the implementation process of the three-dimensional reconstruction method based on two-dimensional images of the present invention.

[0073] Figure 21 It is a schematic diagram of the three-dimensional reconstruction device based on two-dimensional images of the present invention. Specific implementation manners

[0074] The following uses specific examples to illustrate the implementation manners of the present application. Those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the present application. The present application can also be implemented or applied through other different specific implementation manners. Various details in the present application can also be modified or changed according to different viewpoints and application systems without departing from the spirit of the present application. It should be noted that, without conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0075] The following takes the accompanying drawings as a reference and details the embodiments of the present application so that those skilled in the technical field to which the present application belongs can easily implement it. The present application can be embodied in many different forms and is not limited to the embodiments described herein.

[0076] In the description of the present application, the descriptions referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples", etc. mean that the specific features, structures, materials, or characteristics represented in connection with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics represented can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples represented in the present application and the features of different embodiments or examples.

[0077] In addition, the terms "first" and "second" are only used for indicating purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" can explicitly or implicitly include at least one of such features. In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specifically defined.

[0078] To clearly illustrate the present application, devices irrelevant to the description are omitted, and the same or similar components throughout the specification are given the same reference signs.

[0079] Throughout the specification, when it is said that a certain device is "connected" to another device, this includes not only the case of "direct connection", but also the case of "indirect connection" with other elements placed therebetween. In addition, when it is said that a certain device "includes" a certain component, unless there is a particularly contrary record, it does not exclude other components, but means that other components can also be included.

[0080] When it is said that a certain device is "above" another device, this can be directly above the other device, but there can also be other devices therebetween. When it is said that a certain device is "directly" "above" another device, there are no other devices therebetween.

[0081] Although, in some instances, the terms first, second, etc. are used herein to denote various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first interface and a second interface, etc. are indicated. Further, as used herein, the singular forms "a," "an," and "the" are also intended to include the plural forms, unless the context clearly dictates otherwise. It should be further understood that the terms "comprising," "including" indicate the presence of the features, steps, operations, elements, components, items, kinds, and / or groups, but do not preclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are to be construed as inclusive, or meaning any one or any combination. Thus, "A, B, or C" or "A, B, and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B, and C." An exception to this definition occurs only when the combination of elements, functions, steps, or operations is inherently mutually exclusive in some manner.

[0082] The technical terms used herein are only for referring to specific embodiments and are not intended to limit the present application. The singular forms used herein also include the plural forms as long as the statement does not clearly indicate the contrary meaning. The meaning of "including" used in the specification is to embody specific characteristics, regions, integers, steps, operations, elements, and / or components, and does not exclude the existence or addition of other characteristics, regions, integers, steps, operations, elements, and / or components.

[0083] Although not defined differently, including the technical terms and scientific terms used herein, all terms have the same meaning as generally understood by those skilled in the technical field to which the present application pertains. Terms defined in commonly used dictionaries are additionally interpreted to have a meaning consistent with the relevant technical literature and the content currently presented. As long as they are not defined, they should not be over-interpreted as ideal or overly formulaic meanings.

[0084] Figure 1 is a flowchart of the first embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention. As Figure 1 shown, the three-dimensional reconstruction method based on two-dimensional images of the present invention relates to the field of network configuration and is a method for three-dimensional reconstruction based on two-dimensional images applied to a mobile terminal. The process of the present invention includes:

[0085] S110. Collect two-dimensional images.

[0086] S120. Perform dimensionality reduction encoding on the input image to obtain a latent space tensor.

[0087] S130. Perform dimensionality - increasing encoding based on the latent - space tensor to generate three feature planes, and obtain orthogonal three - feature planes through orthogonal arrangement. And

[0088] S140. According to the device parameters of the two - dimensional image acquisition device for capturing two - dimensional images, render the orthogonal three - feature planes to obtain three - dimensional perspective information corresponding to the two - dimensional image. The three - dimensional perspective information includes multi - angle images and / or three - dimensional geometric structures. In this embodiment, the device parameters of the two - dimensional image acquisition device can be the camera parameters of the camera for capturing two - dimensional images, such as: lens view angle, lens focal length, sensor size, etc., and are not limited thereto.

[0089] In view of the existing problems, the present invention provides a three - dimensional reconstruction system and method based on a generative adversarial network and neural rendering. This system and method reconstruct the corresponding three - dimensional geometric structure through the input two - dimensional image and its camera parameters p. The core technology is to fill the information gap through the generative ability of the generative adversarial network, realizing the technical feasibility of three - dimensional reconstruction from two - dimensional images. Since the generative adversarial network itself does not have three - dimensional rendering capabilities, this patent realizes the conversion from two - dimensional to three - dimensional by orthogonalizing the three feature planes obtained by the generator. At the same time, combined with neural rendering technology, the neural rendering module is used to learn the three - dimensional implicit structure in the input image, making the finally reconstructed three - dimensional geometric structure more accurate, meeting the requirements of high - quality modeling and satisfying diverse needs. It does not need to rely on special image acquisition devices, the required devices are simple, and the requirements for acquisition conditions are low, having commercial value for implementation.

[0090] In the present invention, through the innovative combination of the generative adversarial network and neural rendering technology, the modeling work from two - dimensional images to three - dimensional geometric structures without special devices is realized. Among them, the technical point of constructing orthogonal three - feature planes through the generation module enables the acquisition to no longer require special angle requirements. Breaking through the technical barriers, it lays a solid foundation for human body modeling, object modeling, scene modeling, etc.

[0091] Figure 2 is a schematic flowchart of step S110 in the first embodiment of the three - dimensional reconstruction method based on two - dimensional images of the present invention. Figure 3 is a schematic flowchart of step S120 in the first embodiment of the three - dimensional reconstruction method based on two - dimensional images of the present invention. Figure 4 is a schematic flowchart of step S130 in the first embodiment of the three - dimensional reconstruction method based on two - dimensional images of the present invention. Figure 5 is a schematic flowchart of step S140 in the first embodiment of the three - dimensional reconstruction method based on two - dimensional images of the present invention. Figures 2 to 5 As shown in Figure 1In an embodiment, based on steps S110, S120, S130, and S140, step S110 is replaced by S111 and S112, step S120 is replaced by S121, S122, and S123, step S130 is replaced by S131, S132, S133, and S134, and step S140 is replaced by S141 and S142. The following is an explanation for each step:

[0092] S111: Capture a two-dimensional image through a two-dimensional image acquisition device.

[0093] S112: Collect two-dimensional image information and the camera parameters of the two-dimensional image acquisition device.

[0094] S121: Input the input image into an encoder.

[0095] S122: Perform dimensionality reduction encoding on the input image through the encoder.

[0096] S123: The encoder outputs an encoded latent space tensor corresponding to the input image.

[0097] S131: Input the latent space tensor into a generator

[0098] S132: Perform dimensionality increase processing on the latent space tensor through the generator.

[0099] S133: The generator outputs three feature planes.

[0100] S134: Obtain orthogonal three-feature planes through orthogonal arrangement, and the feature planes are orthogonal to each other.

[0101] S141: Input the three feature planes into a neural rendering module for decoding to obtain a latent space tensor.

[0102] S142: According to the camera parameters of the two-dimensional image acquisition device that captured the two-dimensional image, render the latent space tensor to obtain three-dimensional perspective information corresponding to the two-dimensional image. The three-dimensional perspective information includes multi-angle images and / or three-dimensional geometric structures. The neural rendering module includes a decoder and a renderer. The decoder is composed of a multi-layer perceptron (MLP) and is used to decode the input three-feature planes to obtain a latent space tensor. The renderer is composed of fully connected layers and renders the latent space tensor into three-dimensional perspective information and outputs it.

[0103] Figure 6 It is a flowchart of the second embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention. Figure 6 As shown in Figure 1 In the embodiment, step S150 is added. The following is an explanation for each step: S150: Discriminate the multi-angle images to obtain a loss value, obtain a gradient according to the loss value, and update parameters according to the gradient.

[0104] Figure 7 It is a schematic flowchart of step S150 in the second embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention. Figure 7 As shown in Figure 6 In the embodiment of

[0105] S151. The discriminator is used to perform image discrimination on multi-angle images to obtain the loss value of the reconstruction process. The discriminator and the generator for generating the feature plane jointly form an adversarial network structure. Both the encoder and the discriminator are composed of the VGG-16 network structure.

[0106] S152. Obtain the gradient according to the loss value.

[0107] S153. Update the parameters of the adversarial network according to the gradient.

[0108] Figure 8 It is a flowchart of the third embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention. Figure 8 As shown in Figure 1 In the embodiment of

[0109] Figure 9 It is a schematic flowchart of step S150 in the third embodiment of the three-dimensional reconstruction method based on two-dimensional images of the present invention. Figure 9 As shown in Figure 8 In the embodiment of

[0110] S161. The two-dimensional image is a face image, and the first similarity between the two-dimensional image and the preset user face image is obtained.

[0111] S162. Obtain the second similarity between the three-dimensional perspective information and the preset user face three-dimensional perspective information.

[0112] S163. Obtain the reference similarity after weighted calculation according to the first similarity and the second similarity.

[0113] S164. If the reference similarity meets the preset threshold, the user authentication is successful; otherwise, the user authentication fails.

[0114] Figure 10 It is a schematic diagram of the modules of the first embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention. As Figure 10 shown, the three-dimensional reconstruction system based on two-dimensional images of the present invention includes, but is not limited to:

[0115] An image acquisition module 51 for acquiring two-dimensional images;

[0116] A latent space tensor module 52 for performing dimensionality reduction encoding on the input image to obtain a latent space tensor;

[0117] An orthogonal plane module 53 for performing dimensionality increase encoding on the latent space tensor to generate three feature planes, and obtaining orthogonal three-feature planes through orthogonal arrangement;

[0118] A three-dimensional information module 54 for rendering the orthogonal three-feature planes according to the device parameters of the two-dimensional image acquisition device that captures the two-dimensional image, and obtaining the three-dimensional perspective information corresponding to the two-dimensional image. The three-dimensional perspective information includes multi-angle images and / or three-dimensional geometric structures. In this embodiment, the device parameters of the two-dimensional image acquisition device may be the camera parameters of the camera that captures the two-dimensional image, such as: lens view angle, lens focal length, sensor size, etc., which is not limited thereto.

[0119] For the implementation principles of the above modules, refer to the relevant introduction in the three-dimensional reconstruction system based on two-dimensional images, which will not be elaborated here.

[0120] The three-dimensional reconstruction system based on two-dimensional images of the present invention can make up for the information gap in the process of two-dimensional image to three-dimensional reconstruction by using the adversarial generation network, realize the feasibility of two-dimensional image to three-dimensional reconstruction, and expand the usage scenarios and functionality of two-dimensional images.

[0121] Figure 11 It is a schematic diagram of the image acquisition module in the first embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention. Figure 12 It is a schematic diagram of the latent space tensor module in the first embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention. Figure 13 It is a schematic diagram of the orthogonal plane module in the first embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention. Figure 14 It is a schematic diagram of the three-dimensional information module in the first embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention. Figures 11 to 14 shown, in Figure 10Based on the device embodiment, in the three-dimensional reconstruction system based on two-dimensional images of the present invention, the image acquisition module 51 is replaced by an image shooting module 511 and an information acquisition module 512; the latent space tensor module 52 is replaced by an encoder module 521, a dimensionality reduction encoding module 522, and a spatial tensor module 523; the orthogonal plane module 53 is replaced by a generator module 531, a dimensionality increase processing module 532, a feature plane module 533, and an orthogonal plane module 534; the three-dimensional information module 54 is replaced by a tensor decoding module 541 and a three-dimensional perspective module 542. The following is an explanation for each module:

[0122] The image shooting module 511 shoots a two-dimensional image through a two-dimensional image acquisition device.

[0123] The information acquisition module 512 acquires two-dimensional image information and the camera parameters of the two-dimensional image acquisition device.

[0124] The encoder module 521 inputs the input image into an encoder.

[0125] The dimensionality reduction encoding module 522 performs dimensionality reduction encoding on the input image through the encoder.

[0126] The spatial tensor module 523 outputs the encoded latent space tensor corresponding to the input image by the encoder.

[0127] The generator module 531 inputs the latent space tensor into a generator.

[0128] The dimensionality increase processing module 532 performs dimensionality increase processing on the latent space tensor through the generator.

[0129] The feature plane module 533 outputs three feature planes by the generator.

[0130] The orthogonal plane module 534 obtains three orthogonal feature planes through orthogonal arrangement, and the feature planes are orthogonal to each other.

[0131] The tensor decoding module 541 inputs the three feature planes into a neural rendering module for decoding to obtain the latent space tensor.

[0132] The three-dimensional perspective module 542 renders the latent space tensor according to the camera parameters of the two-dimensional image acquisition device that shoots the two-dimensional image to obtain the three-dimensional perspective information corresponding to the two-dimensional image. The three-dimensional perspective information includes multi-angle images and / or three-dimensional geometric structures.

[0133] For the implementation principle of the above steps, please refer to the relevant introduction in the three-dimensional reconstruction system based on two-dimensional images, which will not be elaborated here.

[0134] Figure 15 It is a schematic diagram of the parameter update module in the second embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention. As Figure 15As shown in Figure 10 Based on the device embodiment, the three-dimensional reconstruction system based on two-dimensional images of the present invention further includes: a parameter update module 55, which discriminates multi-angle images to obtain a loss value, obtains a gradient according to the loss value, and updates parameters according to the gradient.

[0135] Figure 16 It is a schematic diagram of the modules of the second embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention. Figure 16 As shown in Figure 15 Based on the device embodiment, the three-dimensional reconstruction system based on two-dimensional images of the present invention replaces the parameter update module 55 with an image discrimination module 551, a gradient obtaining module 552, and a parameter update module 553. The following is an explanation for each module:

[0136] The image discrimination module 551 discriminates multi-angle images through a discriminator to obtain a loss value of the reconstruction process. The discriminator and a generator for generating a feature plane together form an adversarial network structure. Both the encoder and the discriminator are composed of a VGG-16 network structure.

[0137] The gradient obtaining module 552 obtains a gradient according to the loss value.

[0138] The parameter update module 553 updates the parameters of the adversarial network according to the gradient.

[0139] For the implementation principle of the above steps, refer to the relevant introduction in the three-dimensional reconstruction system based on two-dimensional images, which will not be elaborated here.

[0140] Figure 17 It is a schematic diagram of the modules of the user authentication of the third embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention. As Figure 17 shown in Figure 10 Based on the device embodiment, the three-dimensional reconstruction system based on two-dimensional images of the present invention further includes: a user authentication module 56. When the two-dimensional image is a face image, user authentication is performed through a first similarity between the two-dimensional image and a preset face image and a second similarity between the three-dimensional perspective information and the preset three-dimensional perspective information.

[0141] Figure 18 It is a schematic diagram of the modules of the user authentication in the third embodiment of the three-dimensional reconstruction system based on two-dimensional images of the present invention. Figure 18 As shown in Figure 17 Based on the device embodiment, the three-dimensional reconstruction system based on two-dimensional images of the present invention replaces the user authentication module 56 with a first similarity module 561, a second similarity module 562, a reference similarity module 563, and a user comprehensive authentication module 564. The following is an explanation for each module:

[0142] The first similarity module 561, where the two-dimensional image is a face image, obtains the first similarity between the two-dimensional image and a preset user face image.

[0143] The second similarity module 562 obtains the second similarity between the three-dimensional perspective information and the preset user face three-dimensional perspective information.

[0144] The reference similarity module 563 obtains a reference similarity after weighted calculation based on the first similarity and the second similarity.

[0145] The user comprehensive authentication module 564 determines that the user authentication is successful when the reference similarity meets a preset threshold, and fails if not.

[0146] For the implementation principle of the above steps, refer to the relevant introduction in the three-dimensional reconstruction system based on two-dimensional images, which will not be elaborated here.

[0147] Another specific implementation manner of the present invention is as follows:

[0148] Figure 19 is a schematic diagram of the implementation process of the three-dimensional reconstruction method based on two-dimensional images of the present invention. Figure 20 is a flowchart of the steps of the implementation process of the three-dimensional reconstruction method based on two-dimensional images of the present invention. The method flowchart involved in this patent is as Figure 19 、 20 shown. Compared with the original scheme, the scheme of this patent application mainly includes the following parts:

[0149] Encoder module: This module is used to perform dimensionality reduction operations on the input two-dimensional image, that is, the encoding process, and then obtain the encoded latent space tensor corresponding to the input image for subsequent processes. Its input is the image to be processed input by an external device, and the output is the latent space tensor. This encoder module can be constructed by network structures such as VGG-16, GoogLeNet, and ResNet.

[0150] Generator module: This module is used to perform dimensionality increase operations on the input tensor, and then obtain three feature planes containing feature information for subsequent processes. Its input is the latent space tensor output by the encoder module, and the output is three feature planes. This generator module and the discriminator module together constitute a generative adversarial network structure. This generator module can be constructed using generator structures such as CGAN, Wasserstein-GAN, and Style-GAN.

[0151] Neural Rendering Module: This module is used to perform a dimensionality increase operation on the input orthogonal three-feature planes, that is, the decoding process. Then, according to the input camera parameters p, the corresponding multi-angle images are rendered for subsequent processes. Its input is the orthogonal three-feature planes, and the output is the multi-angle images. This module consists of two parts: a decoder and a renderer. The decoder is used to decode the input orthogonal three-feature planes to obtain the latent space tensor. The renderer renders the latent space tensor according to the camera parameters p to obtain multi-angle images. The decoder can be composed of a radial basis network or a multi-layer perceptron, etc., and the renderer can be constructed by fully connected layers.

[0152] Discriminator Module: This module is used to distinguish the authenticity of the generated multi-angle images and calculate the corresponding loss value for subsequent training iteration updates. This discriminator module and the generator module together constitute the structure of a generative adversarial network. This discriminator module can be constructed by network structures such as VGG-16 and ResNet.

[0153] The implementation of this patent includes the following steps, as Figure 20 shown:

[0154] Orthogonal Three-Feature Plane Generation Stage

[0155] S1-1: Input Image and Parameters

[0156] The image to be processed and its corresponding camera parameters p are input by an external device.

[0157] S1-2: Construct Encoder Module and Generator Module

[0158] The image to be processed obtained in S1-1 is input into the encoder module, and the corresponding latent space tensor is obtained after encoding. This encoder module can be constructed by network structures such as VGG-16, GoogLeNet, and ResNet. Then, the latent space tensor is passed into the generator module to generate three feature planes containing feature information. This generator module can be constructed using generator structures such as CGAN, Wasserstein-GAN, and Style-GAN.

[0159] S1-3: Construct Orthogonal Three-Feature Planes

[0160] The three feature planes generated by the generator are orthogonally arranged to obtain the orthogonal three-feature planes. The purpose of this step is to map the two-dimensional features to the three-dimensional space through orthogonal arrangement.

[0161] Training Stage of Each Module:

[0162] S2-1: Construct Neural Rendering Module

[0163] Input the orthogonal three - feature planes generated by S1 - 3 and the camera parameters p input by S1 - 1 into the neural rendering module, which consists of a decoder and a renderer. The decoder can be composed of a radial basis network or a multi - layer perceptron, etc., and the renderer can be constructed by fully - connected layers.

[0164] S2 - 2: Output multi - angle images

[0165] The neural rendering module decodes the input orthogonal three - feature planes to obtain a latent space tensor, and then renders the latent space tensor into multi - angle images through the renderer and outputs them.

[0166] S2 - 3: Construct a discriminator module

[0167] Input the multi - angle images generated by S2 - 2 into the discriminator module, and this module determines their authenticity and calculates the corresponding loss value. The discriminator module can be constructed by network structures such as VGG - 16, ResNet, etc.

[0168] S2 - 4: Train the model until convergence

[0169] Use the loss value obtained in S2 - 3 to calculate the gradient, and iteratively update all the above - mentioned modules according to the gradient until the model converges.

[0170] Three - dimensional geometric structure output stage:

[0171] S3 - 1: Output three - dimensional geometric structure

[0172] After the model finishes training in the second stage, its neural rendering module will learn the implicit three - dimensional structure in the input images. Output the learned implicit three - dimensional structure, that is, obtain the three - dimensional geometric structure corresponding to the input images.

[0173] Compared with the prior art, this patent has the following advantages:

[0174] It realizes the three - dimensional reconstruction of input two - dimensional images without special hardware devices. Compared with existing solutions, it greatly reduces the limitations on acquisition devices and reduces hardware costs.

[0175] By using the generative adversarial network, it makes up for the information gap in the process of two - dimensional image to three - dimensional reconstruction and realizes the technical feasibility. At the same time, it outputs the three - dimensional geometric structure through neural rendering technology, realizing true three - dimensional reconstruction and having certain commercial value.

[0176] The specific implementation scheme of the present invention is as follows:

[0177] In the first stage, the mobile phone captures an object image as the image to be processed and queries the camera parameters p of the mobile phone. The image to be processed is transmitted to the encoder module constructed by the VGG16 network structure, and the corresponding latent space tensor is obtained after encoding by the encoder module. Then, the latent space tensor obtained in the previous step is input into the generator module constructed by the Style-GAN generator structure to generate three feature planes containing feature information. The three feature planes are orthogonally arranged to obtain an orthogonal three-feature plane. The purpose of this step is to be able to map from two dimensions to three dimensions and achieve the dimension elevation operation.

[0178] The orthogonal three-feature plane generated in the first stage and the camera parameters p input by the external device are input into the neural rendering module, which is composed of a decoder and a renderer. The decoder is composed of a multi-layer perceptron MLP, and the renderer is constructed by a fully connected layer. The neural rendering module decodes the input three-feature plane to obtain a latent space tensor, and then renders the latent space tensor into multi-angle images through the renderer and outputs them. Subsequently, the generated multi-angle images are input into the discriminator module constructed by the VGG-16 network structure, and this module judges the authenticity of the images and calculates the corresponding loss value. The loss value calculated in the previous step is used to calculate the gradient, and all the above modules are iteratively updated according to the gradient until the model converges.

[0179] After the model finishes training in the second stage, its neural rendering module will learn the implicit three-dimensional structure in the input image. The learned implicit three-dimensional structure is output, that is, the three-dimensional geometric structure corresponding to the input image is obtained.

[0180] This patent application proposes a three-dimensional reconstruction system and method based on a generative adversarial network and neural rendering, including the following technical advantages:

[0181] (1) Three-dimensional reconstruction is realized based on a generative adversarial network. In the process of two-dimensional image to three-dimensional reconstruction, a large amount of information differences need to be made up. Currently, most patent technologies need to rely on special acquisition devices to provide richer original information. In the present invention, through a generative adversarial network, its generation ability is used to realize the information supplement, so that the technology from two-dimensional image to three-dimensional reconstruction is feasible and no special acquisition device is required.

[0182] (2) A three-dimensional basis is constructed through an orthogonal three-feature plane. For the two-dimensional image to three-dimensional geometric structure, a dimension elevation operation needs to be performed. In the present invention, by orthogonally combining the three feature planes generated by the generator module, an orthogonal three-feature plane is obtained, realizing the dimension expansion. Based on this technical point, there is no additional requirement for the acquisition angle of the input image, reducing the operation requirements for the acquisition personnel.

[0183] (3) A neural rendering module is introduced. By introducing the neural rendering module, the implicit three-dimensional structure in the input image is learned through training, so that the finally reconstructed three-dimensional geometric structure is more accurate. At the same time, since the generative adversarial network itself does not have the ability of three-dimensional rendering, the neural rendering module can be used to render multi-angle views and output them to the discriminator module to calculate the loss value, and then jointly supervise and iteratively update with the generative adversarial network to improve the final reconstruction quality.

[0184] An embodiment of the present invention also provides a three-dimensional reconstruction device based on a two-dimensional image, including a processor and a memory, in which executable instructions of the processor are stored. Among them, the processor is configured to execute the steps of the three-dimensional reconstruction method based on a two-dimensional image via executing the executable instructions.

[0185] As shown above, the three-dimensional reconstruction system based on a two-dimensional image of the present invention can make up for the information gap in the process of two-dimensional image to three-dimensional reconstruction by using a generative adversarial network, realize the feasibility of two-dimensional image to three-dimensional reconstruction, and expand the usage scenarios and functionality of two-dimensional images.

[0186] Those skilled in the art can understand that various aspects of the present invention can be implemented as a system, a method or a program product. Therefore, various aspects of the present invention can be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which can be collectively referred to as "circuit", "module" or "platform" here.

[0187] Figure 21 is a schematic diagram of the three-dimensional reconstruction device based on a two-dimensional image of the present invention. The following refers to Figure 21 Describe the electronic device 600 according to this embodiment of the present invention. Figure 21 The electronic device 600 shown is only an example and should not bring any limitation to the functions and usage scope of the embodiments of the present invention.

[0188] As Figure 21 shown, the electronic device 600 is presented in the form of a general computing device. The components of the electronic device 600 may include but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0189] Among them, the storage unit stores program codes, and the program codes can be executed by the processing unit 610, so that the processing unit 610 executes the steps according to various exemplary embodiments of the present invention described in the above part of the electronic prescription transfer processing method of this specification. For example, the processing unit 610 can execute the steps as Figure 1 shown.

[0190] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only storage unit (ROM) 6203.

[0191] The storage unit 620 may further include a program / utility 6204 having a set (at least one) of program modules 6205. Such program modules 6205 include, but are not limited to: a processing system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment.

[0192] The bus 630 may represent one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus structures.

[0193] The electronic device 600 may also communicate with one or more external devices 700 (such as a keyboard, a pointing device, a Bluetooth device, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 600, and / or may communicate with any device that enables the electronic device 600 to communicate with one or more other computing devices (such as a router, a modem, etc.). Such communication may be through an input / output (I / O) interface 650. Moreover, the electronic device 600 may also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 660. The network adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms, etc.

[0194] An embodiment of the present invention further provides a computer-readable storage medium for storing a program, and when the program is executed, it implements the steps of a three-dimensional reconstruction method based on a two-dimensional image. In some possible implementation manners, various aspects of the present invention may also be implemented in the form of a program product, which includes program code. When the program product runs on a terminal device, the program code is used to cause the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above-mentioned electronic prescription circulation processing method part of this specification.

[0195] As shown above, the 3D reconstruction system based on 2D images of the present invention can make up for the information gap in the process of 2D image to 3D reconstruction by using the generative adversarial network, realize the feasibility of 3D reconstruction from 2D images, and expand the usage scenarios and functionality of 2D images.

[0196] A program product 800 for implementing the above method according to an embodiment of the present invention may be a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0197] The program product may employ any combination of one or more readable media. The readable media may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, but not be limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium include: an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0198] A computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, in which the readable program code is carried. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The readable storage medium may also be any readable medium other than the readable storage medium, which can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination of the foregoing.

[0199] The program code for performing the processing of the present invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java, C++, etc., and also including conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, executed as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user's computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., by using an Internet service provider to connect through the Internet).

[0200] In summary, the object of the present invention is to provide a three-dimensional reconstruction method, system, device and storage medium based on two-dimensional images, which can make up for the information gap in the process of two-dimensional image to three-dimensional reconstruction by using a generative adversarial network, realize the feasibility of three-dimensional reconstruction from two-dimensional images, and expand the usage scenarios and functionality of two-dimensional images.

[0201] The above content is a further detailed description of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is only limited to these descriptions. For those of ordinary skill in the technical field to which the present invention belongs, without departing from the concept of the present invention, several simple deductions or substitutions can be made, and all should be regarded as belonging to the protection scope of the present invention.

Claims

1. A three-dimensional reconstruction method based on two-dimensional images, characterized in that, Including the following steps: Collect a two-dimensional image; Perform dimensionality reduction encoding on the input image to obtain a latent space tensor; Perform dimensionality increase encoding based on the latent space tensor to generate three feature planes, and obtain orthogonal three-feature planes through orthogonal arrangement; and Input the three feature planes into a neural rendering module for decoding to obtain a latent space tensor, and render the latent space tensor according to the camera parameters of the two-dimensional image acquisition device that captured the two-dimensional image to obtain three-dimensional perspective information corresponding to the two-dimensional image. The three-dimensional perspective information includes multi-angle images and / or three-dimensional geometric structures. The neural rendering module includes a decoder and a renderer. The decoder is composed of a multi-layer perceptron (MLP) and is used to decode the input three-feature planes to obtain a latent space tensor. The renderer is composed of fully connected layers and renders the latent space tensor into three-dimensional perspective information and outputs it. The three-dimensional perspective information includes multi-angle images and / or three-dimensional geometric structures.

2. The three-dimensional reconstruction method based on a two-dimensional image according to claim 1, wherein The collecting of the two-dimensional image includes: Shoot a two-dimensional image through a two-dimensional image acquisition device; Collect two-dimensional image information and the camera parameters of the two-dimensional image acquisition device.

3. The three-dimensional reconstruction method based on a two-dimensional image according to claim 1, characterized in that, The performing of dimensionality reduction encoding on the input image to obtain a latent space tensor includes: Input the input image into an encoder; Perform dimensionality reduction encoding on the input image through the encoder; The encoder outputs the encoded latent space tensor corresponding to the input image.

4. The three-dimensional reconstruction method based on a two-dimensional image according to claim 1, wherein The performing of dimensionality increase encoding based on the latent space tensor to generate three feature planes and obtaining orthogonal three-feature planes through orthogonal arrangement includes: Input the latent space tensor into a generator; Perform dimensionality increase processing on the latent space tensor through the generator; The generator outputs three feature planes; Obtain orthogonal three-feature planes through orthogonal arrangement, and the feature planes are orthogonal to each other.

5. The 3D reconstruction method based on 2D images according to claim 3, wherein It also includes: Discriminate the multi-angle images to obtain a loss value, obtain a gradient according to the loss value, and update parameters according to the gradient.

6. The three-dimensional reconstruction method based on a two-dimensional image according to claim 5, wherein, The discriminating of the multi-angle images to obtain a loss value, obtaining a gradient according to the loss value, and updating parameters according to the gradient includes: Use a discriminator to perform image discrimination on the multi-angle images to obtain a loss value of the reconstruction process. The discriminator and the generator used to generate the feature planes jointly form an adversarial network structure. Both the encoder and the discriminator are composed of a VGG-16 network structure; Obtain a gradient according to the loss value; Update the parameters of the adversarial network according to the gradient.

7. The 3D reconstruction method based on 2D images according to claim 1, wherein It also includes: The two-dimensional image is a face image, and user authentication is performed through the first similarity between the two-dimensional image and a preset face image and the second similarity between the three-dimensional perspective information and a preset three-dimensional perspective information.

8. A three-dimensional reconstruction system based on two-dimensional images, characterized in that Including: An image acquisition module that collects two-dimensional images; A latent space tensor module that performs dimensionality reduction encoding on the input image to obtain a latent space tensor; An orthogonal plane module that performs dimensionality increase encoding based on the latent space tensor to generate three feature planes and obtains orthogonal three-feature planes through orthogonal arrangement; A three-dimensional information module inputs the three feature planes into a neural rendering module for decoding to obtain a latent space tensor, and renders the latent space tensor according to the camera parameters of the two-dimensional image acquisition device that captures the two-dimensional image to obtain three-dimensional perspective information corresponding to the two-dimensional image. The three-dimensional perspective information includes multi-angle images and / or three-dimensional geometric structures. The neural rendering module includes a decoder and a renderer. The decoder is composed of a multi-layer perceptron (MLP) and is used to decode the input three feature planes to obtain a latent space tensor. The renderer is composed of fully connected layers, renders the latent space tensor into three-dimensional perspective information and outputs it. The three-dimensional perspective information includes multi-angle images and / or three-dimensional geometric structures.

9. A three-dimensional reconstruction device based on two-dimensional images, characterized in that, Comprising: A processor; A memory in which executable instructions of the processor are stored; Wherein the processor is configured to execute the steps of the method for three-dimensional reconstruction based on a two-dimensional image according to any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the steps of the method for three-dimensional reconstruction based on a two-dimensional image according to any one of claims 1 to 7.

Images