Method, electronic device and computer program product for generating three-dimensional image

Abstract

The embodiment of the invention relates to a method for generating a three-dimensional image, electronic equipment and a computer program product. The method comprises the following steps: receiving a first image presenting a target object at a first view angle, wherein the first image is a two-dimensional image; and determining a transformed image of the first image under the target view angle, wherein the target view angle is the same as or different from the first view angle. The method further includes generating, based on the transformed image, a first representation using a first feature extraction layer in the encoder corresponding to the first viewing angle; and generating a second image based on the first representation, wherein the second image is a three-dimensional image and presents the target object at a target perspective. Through the embodiment of the invention, the three-dimensional image under the target view angle can be generated more quickly and accurately based on the two-dimensional image, so that time and computing resources are saved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computers, and more particularly, to a method, an electronic device, and a computer program product for generating a three-dimensional image. Background Art

[0002] With the development of technology, it has become possible to render three-dimensional (3D) images based on two-dimensional (2D) images, and this technology has many applications. For example, human body rendering has many applications such as virtual try-on, personalized shopping, virtual reality, etc. Human body rendering is a challenging problem that has attracted widespread attention in the fields of computer vision and graphics. Its purpose is to synthesize realistic human images or videos with controllable appearance, posture, expression, etc. Summary of the invention

[0003] Embodiments of the present disclosure provide a method, an electronic device, and a computer program product for generating a three-dimensional image.

[0004] According to a first aspect of the present disclosure, a method for generating a three-dimensional image is provided. The method includes receiving a first image presenting a target object at a first perspective, wherein the first image is a two-dimensional image. The method also includes determining a transformed image of the first image at a target perspective, wherein the target perspective is the same as or different from the first perspective. The method also includes generating a first representation based on the transformed image using a first feature extraction layer in an encoder corresponding to the first perspective. The method also includes generating a second image based on the first representation, wherein the second image is a three-dimensional image and presents the target object at the target perspective.

[0005] According to a second aspect of the present disclosure, an electronic device is also provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, and the instructions cause the device to perform an action when executed by the processor. The action includes receiving a first image presenting a target object at a first perspective, wherein the first image is a two-dimensional image. The action also includes determining a transformed image of the first image at a target perspective, wherein the target perspective is the same as or different from the first perspective. The action also includes generating a first representation based on the transformed image using a first feature extraction layer in an encoder corresponding to the first perspective. The action also includes generating a second image based on the first representation, wherein the second image is a three-dimensional image and presents the target object at the target perspective.

[0006] According to a third aspect of the present disclosure, there is provided a computer program product, which is tangibly stored on a computer-readable medium and comprises computer-executable instructions, which when executed by a device cause the device to perform the method according to the first aspect.

[0007] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram schematically illustrates an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 A schematic diagram schematically illustrates the principle of a method for generating a three-dimensional image according to an exemplary implementation of the present disclosure;

[0011] Figure 3 A flowchart of a method for generating a three-dimensional image according to an exemplary implementation of the present disclosure is schematically shown;

[0012] Figure 4 A schematic diagram schematically illustrates a workflow of a group equivariant convolutional neural network (G-CNN) according to an exemplary implementation of the present disclosure;

[0013] Figure 5 A schematic diagram schematically illustrates a framework for training a group equivariant convolutional neural network according to an exemplary implementation of the present disclosure; and

[0014] Figure 6 A block diagram of a device for generating a three-dimensional image according to an exemplary implementation of the present disclosure is schematically shown.

[0015] Throughout the drawings, the same or similar reference numbers denote the same or similar elements. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below. In addition, all specific numerical values ​​in this document are examples, which are only for the purpose of helping understanding, and are not intended to limit the scope.

[0018] As discussed in the background technology above, the purpose of human body rendering is to synthesize realistic human images or videos with controllable appearance, posture, expression, etc. Traditional human body rendering methods can be roughly divided into two categories: model-based methods and model-free methods. Among them, model-based methods rely on predefined 3D human body models, such as skinned multi-person linear (SMPL) models, to represent human body shape and posture. The SMPL model usually estimates 3D human body parameters from the input image or video, and then renders it using a differentiable renderer. Model-based methods can achieve high-quality results, but they also have some limitations. For example, they may encounter problems such as perspective ambiguity, limited expression capabilities, and high computational cost.

[0019] Model-free methods do not use any predefined 3D human model, but directly learn the mapping from input images or videos to output images or videos with the desired perspective or appearance. Model-free methods usually adopt deep neural networks, such as convolutional neural networks (CNNs) or generative adversarial networks (GANs), to learn this mapping in an end-to-end manner. Model-free methods can avoid some limitations of model-based methods, but they also face some challenges. For example, they may require a large amount of training data, lack geometric consistency, and produce artifacts or distortions.

[0020] As can be seen, human rendering is also a challenging problem due to the high complexity and variability of human appearance and motion. For example, one of the challenges of human rendering is to deal with perspective changes, such as the task of generating images or videos of the same person from different perspectives given an input image or video of a person from a certain perspective. This task requires not only capturing the 3D structure and appearance of the human body from a single perspective, but also transferring them to images from other perspectives in a consistent and realistic way.

[0021] Another challenge of human rendering is dealing with data scarcity. Collecting large-scale datasets with realistic 3D human structures or multi-view images is expensive and time-consuming. Moreover, such datasets may not cover all possible variations of human appearance and motion in different scenarios. Therefore, a method that can learn from limited or noisy data and generalize well to unseen situations is needed.

[0022] To address the above shortcomings, an embodiment of the present disclosure provides a scheme for generating three-dimensional images. The present scheme proposes an equivariant learning framework that can address at least one of these challenges. The present scheme predicts images of multiple perspectives from a single 2D image in the training phase, and generates a 3D image from a single 2D image in the inference phase. In some embodiments, for human body rendering, the present scheme does not rely on any predefined 3D human body model or real 3D human body structure or multi-perspective images. Instead, it utilizes a group equivariant convolutional neural network (G-CNN) and self-supervised learning to learn invariant and equivariant representations of human body rendering.

[0023] For example, this scheme uses a group equivariant convolutional neural network (G-CNN) to encode a 2D image from a single perspective to generate a 3D image. G-CNN is designed to exploit the symmetry of the data and reduce sample complexity by sharing weights between different transformations, so it can generate a 3D image from a target perspective more quickly and accurately based on a 2D image, saving time and computing resources.

[0024] In some embodiments, the present solution formulates the 2D image to 3D image problem as an equivariant learning task, in which it seeks to learn a representation that is invariant to the input perspective and equivariant to the output perspective. This enables the prediction of images of multiple perspectives of an object from a single input image during the training phase, and the generation of a 3D image from a single input image during the inference phase. In some embodiments, the present solution uses a self-supervised loss function to implement equivariance constraints between images at different perspectives during training. This enables the training of G-CNNs without any real 3D human structures or multi-perspective images. In some embodiments, the present solution uses a differentiable renderer to project 3D structures onto 2D images during the inference process. This enables the generation of realistic 3D images with consistent geometry and appearance in images from different perspectives.

[0025] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1As shown, the example environment 100 may include a computing device 102. The computing device 102 may be, for example, a computing system or a server. An application or software for generating a 3D image that executes the present disclosure may be installed on the computing device 102. The computing device 102 receives an image 104 (also referred to as a first image herein) as an input, wherein the image 104 is a 2D image and includes a target object, such as a human body, presented in an original perspective (also referred to as a first perspective). It can be understood that although the present disclosure uses the human body as an example to describe the present solution, the present solution may also be applied to 3D image generation of other objects.

[0026] The user may also input a target viewing angle 108 to the computing device 102. The target viewing angle 108 indicates the angle from which the user expects to see the target object. For example, the north direction may be 0 degrees, and the angle rotating in the clockwise direction may be the target viewing angle, that is, 90 degrees represents due east, 180 degrees represents due south, and 270 degrees represents due west.

[0027] The image 104 may be input to the G-CNN model 110. In the G-CNN model 110, a dihedral group 112 may be included. The dihedral group 112 may consist of eight transformations: four rotations (0°, 90°, 180°, and 270°) and four reflections (horizontal, vertical, and diagonal). These transformations convert the objects in the image 104 from the original perspective to other perspectives to obtain one or more images 114 (also referred to as transformed images) of the same object at different angles. One or more images 114 may be input to the encoder 116 of the shared G-CNN 110. The encoder 116 generates eight feature maps of the same size and dimension. These feature maps are invariant to the input image 104 and equivariant to the output image.

[0028] In some examples, the encoder 116 may include feature extraction layers specific to a certain view angle, such as feature layer 118-1 (also referred to as a first feature layer) and feature layer 118-2 (also referred to as a second feature layer), as well as more feature layers not shown (which may be collectively or individually referred to as feature layers or feature extraction layers). As an example, feature extraction layer 118-1 may be specific to a view angle of 0 degrees, feature extraction layer 118-2 may be specific to a view angle of 90 degrees, and so on. The feature extraction layer may be one or more layers of a convolutional neural network (CNN), which may better extract features of objects in an image at a specific view angle. The encoder 116 may generate a representation 120 (also referred to as a first representation 120) based on the image 114. The representation 120 may be decoded by a decoder 122 and generate an image 106. The image 106 is a 3D image and presents a target object at a target view angle 108.

[0029] It should be understood that the architecture and functions in the example environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. Embodiments of the present disclosure may also be applied to other environments with different structures and / or functions.

[0030] Figure 2 A schematic diagram of the principle 200 of a method for generating a three-dimensional image according to an exemplary implementation of the present disclosure is schematically shown. The present disclosure formulates the image generation problem from 2D to 3D as an equivariant learning task, wherein the present disclosure seeks to learn a representation that is invariant to the input image and equivariant to the output image. Specifically, a deep neural network can be trained to predict a 3D image from a single input image using equivariance constraints and a self-supervised learning strategy, and has advantages in accuracy, robustness, and efficiency.

[0031] In principle, the framework of this scheme consists of three main components: an encoder network, a decoder network, and a loss function. The encoder network takes an input image of an object and extracts a latent representation that is invariant to the input image. The decoder network takes the latent representation and the output image as input and generates a 3D image equivalent to the output image. The loss function measures the difference between the predicted 3D image and the real 3D image.

[0032] The framework exploits two key ideas: equivariance constraints and self-supervised learning. The equivariance constraint enforces that changes in the output image should lead to corresponding changes in the predicted 3D image. Self-supervised learning utilizes multiple images of the same object with different viewpoints as supervisory signals for training.

[0033] As an example, the original image 202 may be converted from a 0-degree viewing angle to an image 204 of a 180-degree viewing angle, and then a corresponding 3D image 208 may be generated. The original image 202 may also be converted from a 2D image of a 0-degree viewing angle to a 3D image 206 of a 0-degree viewing angle, and then a corresponding 3D image 208 may be generated. These two methods are equivalent.

[0034] After introducing the basic principles of this scheme, the following will combine Figures 3 to 5 The process according to the embodiment of the present disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or the actions shown may be omitted, and the scope of the present disclosure is not limited in this respect.

[0035] Figure 3A flowchart of a method 300 for generating a three-dimensional image according to an exemplary implementation of the present disclosure is schematically shown. At block 302, a first image presenting a target object at a first viewing angle is received, wherein the first image is a two-dimensional image. For example, the computing device 102 receives the image 104, wherein the target object in the image 104 is a human body and is presented at a viewing angle of 0 degrees.

[0036] At block 304, a transformed image of the first image at a target viewing angle is determined, wherein the target viewing angle is the same as or different from the first viewing angle. For example, computing device 102 determines a transformed image 114 of image 104 at target viewing angle 108. Transformed image 114 rotates image 104 90 degrees clockwise.

[0037] In some embodiments, multiple transformed images may be generated based on the first image and using a dihedral group, wherein the dihedral group includes rotation transformations at multiple angles and reflection transformations at multiple angles, for example, dihedral group 112 .

[0038] In some embodiments, the rotation transformation of multiple angles includes multiple rotation transformations that rotate the target object at different angles, and the reflection transformation of multiple angles includes multiple reflection transformations that reflect the target object in different directions. For example, the dihedron group 112 can be composed of eight transformations: four rotations (0°, 90°, 180°, and 270°) and four reflections (horizontal, vertical, and 2 diagonals). Based on the target perspective, multiple weights corresponding to the multiple transformed images can be determined, and the transformed images can be determined based on the multiple transformed images and the multiple weights. For example, the weights corresponding to the four rotations are r1, r2, r3, and r4, respectively, and the weights corresponding to the four reflections are r5, r6, r7, and r8, respectively. A transformed image can be synthesized based on these weights. It can be understood that not all of these 8 transformations need to be used every time, but which transformations to use can be determined based on the original perspective and the target perspective.

[0039] At block 306, a first representation is generated based on the transformed image using a first feature extraction layer in the encoder corresponding to the first viewing angle. As an example, computing device 102 generates representation 120 based on transformed image 114 using feature layer 118-1 in encoder 116 (assuming it is specific to a 0 degree viewing angle).

[0040] In some embodiments, the encoder further includes a second feature extraction layer corresponding to the second viewing angle, a third feature extraction layer corresponding to the third viewing angle, and a fourth feature extraction layer corresponding to the fourth viewing angle, wherein the first viewing angle, the second viewing angle, the third viewing angle, and the fourth viewing angle are all different from each other. For example, the first viewing angle is 0 degrees, the second viewing angle is 90 degrees, the second viewing angle is 180 degrees, and the fourth viewing angle is 270 degrees.

[0041] In some embodiments, a second representation may be generated based on the first image and using the second feature extraction layer, a third representation may be generated based on the first image and using the third feature extraction layer, and / or a fourth representation may be generated based on the first image and using the fourth feature extraction layer, and a fifth representation may be generated based on the first representation, the second representation, the third representation, and the fourth representation. In this way, the fifth representation incorporates more feature information, which may be referenced and corrected with each other to more accurately generate a 3D image at a target viewing angle.

[0042] At block 308, a second image is generated based on the first representation, wherein the second image is a three-dimensional image and presents the target object at the target perspective. As an example, computing device 102 generates image 106 based on representation 120, wherein image 106 is a 3D image and presents a human body at 90 degrees. In some embodiments, image 106 may be generated based on a fifth representation.

[0043] By implementing the embodiment provided by method 300, leading results can be obtained on various benchmark data sets using less data and computing resources than traditional methods, and three-dimensional images at a target perspective can be generated based on two-dimensional images more quickly and accurately, thereby saving time and computing resources.

[0044] In some embodiments, the encoder belongs to a group equivariant convolutional neural network (G-CNN) obtained through pre-training, and the first feature extraction layer, the second feature extraction layer, the third feature extraction layer and the fourth feature extraction layer are respectively at least one layer in the group equivariant convolutional neural network.

[0045] Now refer to Figure 4 To understand the group equivariant convolutional neural network of the present disclosure. Figure 4 A schematic diagram of a G-CNN workflow 400 according to an exemplary implementation of the present disclosure is schematically shown. G-CNN extends the convolutional neural network and has equivariance under certain transformations (rotation, translation, reflection, etc.). In other words, the result obtained by transforming the image and then sending it to the convolution layer is the same as the result obtained by directly sending the original image to the same convolution layer to obtain a feature map and then transforming the feature map.

[0046] As an example, Figure 4The images 402, 404, 406, and 408 in FIG. 4 show the effects of some possible transformations. For example, assuming that the original image is image 402 and the target object therein is F, transformation 410 translates the target object F to the right to obtain image 404. Transformation 412 rotates image 404 by 180 degrees to obtain image 406. Transformation 414 rotates image 406 90 degrees clockwise and translates it to the left to obtain image 408. Transformation 416 translates image 408 to the right and rotates it 90 degrees clockwise to obtain image 402. The effects of these transformations 410-416 on the original image 402 are equivalent to the transformation of the original image 402 on the feature map (also called representation) obtained after passing through the feature extraction layer (CNN).

[0047] In order to better understand the G-CNN of this scheme, the following will be combined Figure 5 To describe the process of training G-CNN. Figure 5 A schematic diagram of a framework 500 for training a group equivariant convolutional neural network according to an exemplary implementation of the present disclosure is schematically shown. A group equivariant convolutional neural network is a convolutional neural network that exploits the symmetry of data and reduces sample complexity by sharing weights between different transformations. For example, G-CNN 506 can be defined as: G-CNN 506 is a CNN that satisfies the following properties: for any input x and any transformation g in the group G, if applied before G-CNN, the output of G-CNN is unchanged for g, and is equivalent for g to if G-CNN is applied after. That is, formula (1):

[0048]

[0049] Among them, f represents the operation of G-CNN, X represents the input space, and G represents the transformation group.

[0050] In some embodiments, for example, for human body rendering, G-CNN can be used to encode images of multiple perspectives of a human from an image of a single perspective. Specifically, dihedral group D4 can be used, as mentioned above. These transformations can be applied to an input image of a human from a certain perspective (e.g., image 502 or image 504), and eight images of the same person from different perspectives are obtained. These images can then be input to a shared G-CNN encoder, which generates eight feature maps of the same size and dimension (e.g., representation 510 and representation 512). These feature maps are invariant to the input image and equivariant to the output image (e.g., image 516 and image 518).

[0051] The advantage of using a G-CNN encoder is that it can capture more information about the 3D structure and appearance of a person from a single image than a traditional CNN encoder. In addition, it can reduce the number of parameters and improve generalization ability by sharing weights between different transformations.

[0052] In some embodiments, in order to train the G-CNN encoder without any real 3D human structure or images from multiple perspectives, a self-supervised loss function can be used to enforce the equivariance constraint between images from different perspectives during training. The self-supervised loss function consists of two terms: an equivariance loss term 520 (also referred to as the first loss) and a perceptual loss term 522 (also referred to as the second loss).

[0053] The equivariance loss term measures how well the G-CNN encoder preserves the relative transformation between images from different viewpoints. For example, if an input image is rotated 90 degrees clockwise and then fed into the G-CNN encoder, it is expected that its corresponding feature map will also be rotated 90 degrees clockwise compared to its original feature map. To compute this loss term, an auxiliary network called an inverse transformation network 508 (ITN) can be used, which takes two feature maps as input and predicts their relative transformation in terms of rotation angle or reflection axis. The ITN can be trained jointly with the G-CNN encoder, using feature map pairs with known relative transformation as supervision. The equivariance loss term can be defined as follows:

[0054]

[0055] Among them, x i and x j are two images obtained by applying different transformations of the dihedral set D4 to the input image; f(x i ) and f(x j ) is the corresponding feature map generated by the G-CNN encoder; g ij is their true relative transformation; and represents the squared Euclidean distance.

[0056] The perceptual loss term measures how well the generated image matches its real image in terms of appearance similarity. To generate images from feature maps, another auxiliary network called a differentiable renderer can be used. The input of the differentiable renderer can be: (1) the feature map generated by the G-CNN encoder; (2) the desired viewpoint; and (3) the background image. The differentiable renderer outputs an image with consistent geometry and appearance across different viewpoints. The differentiable renderer can be jointly trained with the GCN-N encoder, using feature map pairs and real images as supervision. The perceptual loss term can be defined as follows:

[0057]

[0058] Where R is a differentiable renderer; f(x i ) is the GCNN encoder for image x i Generated feature map; v i is the image y i The required viewing angle; b is the background image; and |·|1 represents the L1 distance.

[0059] The total loss function (also called the third loss function) of the training framework of this scheme can be expressed as formula (4):

[0060] L=L eq +λL per (4)

[0061] Among them, λ is a trade-off parameter to balance the equivariance loss and perceptual loss.

[0062] In some embodiments, stochastic gradient descent (SGD) with momentum can be used to optimize the G-CNN encoder and auxiliary network. For example, a learning rate of 0.01, a momentum of 0.9, and a batch size of 32 are used.

[0063] In some embodiments, the decoder 514 can be a deconvolutional neural network (DeCNN) that takes the latent representation and the output image as input and outputs a 3D image of size H×W×3. The output image is represented by a rotation matrix R of size 3×3, which specifies the desired orientation of the structure of the object in 3D. The decoder ensures that the structure of the target object in the output 3D image is equivalent to the output image, which means that when the output image is rotated or translated, it changes accordingly.

[0064] In this way, the G-CNN trained by framework 500 can reduce the complexity of samples required for training. Equivariant learning can be used to exploit the symmetry of samples, and can learn from less data and better generalize unseen perspectives. In some embodiments, model complexity is also reduced, and equivariant learning can reduce the number of parameters and the amount of calculation required to share weights between images of different perspectives. In some embodiments, interpretability is improved, and equivariant learning can provide a more intuitive and meaningful data representation while retaining its geometric structure.

[0065] In addition, this solution also uses self-supervised learning for 3D human structure prediction. Self-supervised learning is a powerful technique that can learn useful features from unlabeled data by designing pretext tasks. In this case, it is possible to predict multiple views of a human from a single view using an equivariance constraint. This enables the solution to leverage a large amount of unlabeled data for training and improve its performance without the need for expensive 3D annotations.

[0066] Figure 6 1 shows a schematic block diagram of a device 600 that can be used to implement an embodiment of the present disclosure. The device 600 may be a device or apparatus described in an embodiment of the present disclosure. Figure 6 As shown, the device 600 includes a central processing unit / graphics processing unit (CPU / GPU) 601, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 602 or loaded from a storage unit 608 to a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The CPU / GPU 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604. Although not shown in FIG. Figure 6 As shown in FIG. 6 , device 600 may also include a co-processor.

[0067] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0068] The various methods or processes described above may be performed by the CPU / GPU 601. For example, in some embodiments, the methods may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the CPU / GPU 601, one or more steps or actions in the methods or processes described above may be performed.

[0069] In some embodiments, the methods and processes described above may be implemented as a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0070] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0071] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0072] The computer program instructions for performing the disclosed operation may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, programming languages ​​including object-oriented programming languages, and conventional procedural programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, executed as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In certain embodiments, by utilizing the state information of a computer-readable program instruction to customize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit may execute a computer-readable program instruction, thereby realizing various aspects of the present disclosure.

[0073] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0074] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0075] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the equipment, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the frame can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous frames can actually be executed substantially in parallel, and they can also be executed in the opposite order sometimes, depending on the functions involved. It should also be noted that each frame in the block diagram and / or flow chart, and the combination of frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0076] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for generating a three-dimensional image, comprising: Receiving a first image presenting a target object at a first perspective, wherein the first image is a two-dimensional image; Determining a transformed image of the first image at a target viewing angle, wherein the target viewing angle is the same as or different from the first viewing angle; Based on the transformed image, generating a first representation using a first feature extraction layer in an encoder corresponding to the first view; as well as A second image is generated based on the first representation, wherein the second image is a three-dimensional image and presents the target object at the target viewing angle.

2. The method according to claim 1, wherein determining a transformed image of the first image at a second viewing angle comprises: generating a plurality of transformed images based on the first image and using a dihedron group, wherein the dihedron group includes rotation transformations at multiple angles and reflection transformations at multiple angles; as well as Based on the target viewing angle, determining a plurality of weights corresponding to the plurality of transformed images; as well as Based on the plurality of transformed images and the plurality of weights, the transformed image is determined. 3 . The method according to claim 2 , wherein the rotation transformations at multiple angles include multiple rotation transformations that respectively rotate the target object at different angles, and the reflection transformations at multiple angles include multiple reflection transformations that respectively reflect the target object in different directions.

4. The method according to claim 1, wherein the encoder further comprises a second feature extraction layer corresponding to a second viewing angle, a third feature extraction layer corresponding to a third viewing angle, and a fourth feature extraction layer corresponding to a fourth viewing angle, wherein the first viewing angle, the second viewing angle, the third viewing angle, and the fourth viewing angle are all different from each other.

5. The method according to claim 4, further comprising: generating a second representation based on the first image and using the second feature extraction layer; generating a third representation based on the first image and using the third feature extraction layer; generating a fourth representation based on the first image and using the fourth feature extraction layer; as well as generating a fifth representation based on the first representation, the second representation, the third representation, and the fourth representation, and Wherein generating a second image based on the first representation comprises: The second image is generated based on the fifth representation.

6. The method according to claim 5, wherein the encoder belongs to a group equivariant convolutional neural network obtained through pre-training, and the first feature extraction layer, the second feature extraction layer, the third feature extraction layer and the fourth feature extraction layer are respectively at least one layer in the group equivariant convolutional neural network.

7. The method according to claim 6, wherein the pre-training comprises: Based on the sample two-dimensional image set, determining a first loss function corresponding to the sample two-dimensional image set using an inverse transform network, wherein the first loss function represents the degree to which the encoder retains relative transformations between multiple images at different viewing angles; as well as Based on the sample two-dimensional image set, a second loss function corresponding to the sample two-dimensional image set is determined using a differentiable renderer, and the second loss function represents the degree of matching between the generated multiple training three-dimensional images and the multiple reference three-dimensional images in terms of appearance similarity.

8. The method of claim 7, wherein the first loss function comprises a set of distances between: a plurality of training transformed images generated by the inverse transform network and based on a plurality of representations determined by a feature extraction layer in the encoder corresponding to a plurality of sample transformed images; and Multiple reference transformed images.

9. The method of claim 7, wherein the second loss function comprises a distance set between: a plurality of training three-dimensional images generated by the differentiable renderer and based on a plurality of representations generated by a feature extraction layer in the encoder corresponding to a plurality of sample transformed images, a target view angle, and a background image; and Multiple reference 3D images.

10. The method according to claim 7, wherein the pre-training further comprises: Determining a first weight corresponding to the first loss function and a second weight corresponding to the second loss function; Determining a third loss function based on the first weight and the second weight; and At least one parameter of the third loss function is adjusted.

11. An electronic device, comprising: processor; as well as a memory coupled to the processor, the memory having instructions stored therein, the instructions, when executed by the processor, causing the device to perform actions, the actions comprising: Receiving a first image presenting a target object at a first perspective, wherein the first image is a two-dimensional image; Determining a transformed image of the first image at a target viewing angle, wherein the target viewing angle is the same as or different from the first viewing angle; Based on the transformed image, generating a first representation using a first feature extraction layer in an encoder corresponding to the first view; and A second image is generated based on the first representation, wherein the second image is a three-dimensional image and presents the target object at the target viewing angle.

12. The electronic device according to claim 11, wherein determining a transformed image of the first image at a second viewing angle comprises: generating a plurality of transformed images based on the first image and using a dihedron group, wherein the dihedron group includes rotation transformations at multiple angles and reflection transformations at multiple angles; as well as Based on the target viewing angle, determining a plurality of weights corresponding to the plurality of transformed images; as well as Based on the plurality of transformed images and the plurality of weights, the transformed image is determined. 13 . The electronic device according to claim 12 , wherein the rotation transformations at multiple angles include multiple rotation transformations that respectively rotate the target object at different angles, and the reflection transformations at multiple angles include multiple reflection transformations that respectively reflect the target object in different directions.

14. An electronic device according to claim 11, wherein the encoder further includes a second feature extraction layer corresponding to a second viewing angle, a third feature extraction layer corresponding to a third viewing angle, and a fourth feature extraction layer corresponding to a fourth viewing angle, wherein the first viewing angle, the second viewing angle, the third viewing angle, and the fourth viewing angle are all different from each other.

15. The electronic device according to claim 14, wherein the actions further comprise: generating a second representation based on the first image and using the second feature extraction layer; generating a third representation based on the first image and using the third feature extraction layer; generating a fourth representation based on the first image and using the fourth feature extraction layer; as well as generating a fifth representation based on the first representation, the second representation, the third representation, and the fourth representation, and Wherein generating a second image based on the first representation comprises: The second image is generated based on the fifth representation.

16. An electronic device according to claim 15, wherein the encoder belongs to a group equivariant convolutional neural network obtained through pre-training, and the first feature extraction layer, the second feature extraction layer, the third feature extraction layer and the fourth feature extraction layer are respectively at least one layer in the group equivariant convolutional neural network.

17. The electronic device according to claim 16, wherein the pre-training comprises: Based on the sample two-dimensional image set, determining a first loss function corresponding to the sample two-dimensional image set using an inverse transform network, wherein the first loss function represents the degree to which the encoder retains relative transformations between multiple images at different viewing angles; as well as Based on the sample two-dimensional image set, a second loss function corresponding to the sample two-dimensional image set is determined using a differentiable renderer, and the second loss function represents the degree of matching between the generated multiple training three-dimensional images and the multiple reference three-dimensional images in terms of appearance similarity.

18. The electronic device according to claim 17, wherein: The first loss function includes the distance set between the following two items: a plurality of training transformed images generated by the inverse transform network and based on a plurality of representations determined by a feature extraction layer in the encoder corresponding to a plurality of sample transformed images; as well as multiple reference transformed images, and The second loss function includes the distance set between the following two items: a plurality of training three-dimensional images generated by the differentiable renderer and based on a plurality of representations generated by a feature extraction layer in the encoder corresponding to a plurality of sample transformed images, a target view angle, and a background image; as well as Multiple reference 3D images.

19. The electronic device according to claim 17, wherein the pre-training further comprises: Determining a first weight corresponding to the first loss function and a second weight corresponding to the second loss function; Determining a third loss function based on the first weight and the second weight; and At least one parameter of the third loss function is adjusted.

20. A computer program product tangibly stored on a computer readable medium and comprising computer executable instructions which, when executed by a device, cause the device to: Receiving a first image presenting a target object at a first perspective, wherein the first image is a two-dimensional image; Determining a transformed image of the first image at a target viewing angle, wherein the target viewing angle is the same as or different from the first viewing angle; generating a first representation based on the transformed image using a first feature extraction layer in an encoder corresponding to the first view; and A second image is generated based on the first representation, wherein the second image is a three-dimensional image and presents the target object at the target viewing angle.

Images