An unsupervised single-viewpoint synthesis method and apparatus
By employing an unsupervised single-viewpoint compositing method guided by design style and prior distillation, this method utilizes a single source view for view transformation, overcoming the limitation of existing methods that require multiple source views and improving the quality and real-time performance of view compositing.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-01
- Publication Date
- 2026-04-03
AI Technical Summary
Existing unsupervised view synthesis methods require multiple source views to learn view transitions between different poses, which limits their application when only a single source view exists.
An unsupervised single-view synthesis method based on style guidance and prior distillation was designed. Using a single source view as input, the method synthesizes a view at any target pose through style embedding strategy, 3D rotation operation and view mapping operation, and trains the model through the total loss function.
It enables view transformation using only a single source view under unsupervised conditions, improving the quality of synthesized views and reducing inference time, achieving performance comparable to fully supervised single-view synthesis methods.
Smart Images
Figure CN115272148B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of deep learning and viewpoint synthesis, and in particular to an unsupervised single-viewpoint synthesis method and apparatus. Background Technology
[0002] With the rapid development of multi-view video technology, free-viewpoint displays have gradually gained researchers' attention. Free-viewpoint displays, due to their ability to provide immersive visual experiences, are widely used in fields such as the metaverse and virtual reality. In practical applications, obtaining high-quality free-viewpoint displays typically requires acquiring a large number of views from different poses. However, deploying multiple cameras to capture views from different poses is costly, posing a challenge to achieving high-quality free-viewpoint displays. Viewpoint synthesis technology, capable of synthesizing target views at arbitrary poses given a finite number of source views, has become a current research hotspot in the field of free-viewpoint displays.
[0003] In recent years, thanks to the powerful feature representation capabilities of convolutional neural networks, supervised view synthesis methods based on deep learning have made significant progress. Liu et al. proposed a geometry-aware view synthesis network to synthesize target views. Wiles et al. synthesized target views by learning point cloud transformations between different poses. Kyle et al. designed a bottleneck transformation network to learn 3D spatial transformations to synthesize target views. However, the above methods typically rely on expensive labels to supervise the view transformation learning process. Unlike supervised methods, unsupervised view synthesis methods can learn view transformations between different poses without label supervision. Therefore, research on unsupervised view synthesis methods has significant importance and practical value.
[0004] For unsupervised view synthesis, Nicolai et al. used multiple source views as input to the network to obtain an object representation at a reference pose, and then reconstructed the multiple input source views based on this object representation. Finally, they used the reconstruction error between the synthesized source views and the real source views to constrain the training of the network.
[0005] While the aforementioned methods can train view synthesis networks in an unsupervised manner, they still require multiple source views to perceive the spatial transformation relationships between different views. This significantly limits the application of view synthesis methods when only a single source view is available. Therefore, it is crucial to research unsupervised single-view synthesis methods that train view synthesis networks using only a single source view. Current unsupervised view synthesis methods require multiple source views to learn view transformations between different poses, which greatly limits their application when only a single source view is available. Summary of the Invention
[0006] This invention provides an unsupervised single-viewpoint synthesis method and apparatus. This invention utilizes only a single source view as input and designs an unsupervised single-viewpoint synthesis method based on style guidance and prior distillation to synthesize views at arbitrary target poses. It learns the transformation between views using only a single source view. Details are described below:
[0007] An unsupervised single-viewpoint synthesis method, characterized in that the method includes:
[0008] Construct a style-guided view composition model consisting of a style embedding strategy, 3D rotation operations, and view mapping operations, with the real source view I. S Source P S and Gaussian noise G N Used as input to synthesize the source view The method involves learning view transformations and then synthesizing pseudo-target views at arbitrary target poses from the learned view transformations.
[0009] Construct a priori distillation view synthesis model consisting of a volume generation module, 3D rotation operations, and view mapping operations, with the real source view I. S Source P S and arbitrary target pose P T As input, to synthesize arbitrary target pose P T The final target view at the location
[0010] Based on the style-guided view synthesis model and the prior distillation view synthesis model, a total loss function is constructed. The style-guided view synthesis model and the prior distillation view synthesis model are then trained based on the total loss function to synthesize the target view.
[0011] The style embedding strategy includes a style generation part and a style manipulation part. The style generation part will use the real source view I. S As input, the style vector S of the source view is extracted as follows:
[0012] S = O SGM (I S )
[0013] Among them, O SGM The style generation part consists of six stacked 2D convolutional layers and four stacked 1D convolutional layers.
[0014] Gaussian noise G N The style vector S is used as input to the style manipulation part to generate the intrinsic representation V. R ,as follows:
[0015] V R =OSMM (G N ,S)
[0016] Among them, O SMM The style control section includes a 2D style control unit and a 3D style control unit.
[0017] Furthermore,
[0018] The 2D style manipulation unit employs an encoder-decoder network structure with skip connections. The encoder network consists of eight 2D convolutional layers and four adaptive instance normalization layers. An adaptive instance normalization layer is stacked after every two 2D convolutional layers to embed the style vector S into Gaussian noise G in 2D space. N The decoder network consists of six stacked 2D convolutional layers.
[0019] The 3D style manipulation unit consists of two 3D convolutional layers and two adaptive instance normalization layers, wherein an adaptive instance normalization layer is stacked after each 3D convolutional layer to further embed the style vector S in 3D space.
[0020] Among them, the source view I S As input to the volume generation module to generate P S The 3D volume at V is represented by S Based on the generated 3D volume representation V S At source pose P S and target pose P T Guided by 3D rotation and view mapping operations, the final target view is synthesized. as follows:
[0021]
[0022] The loss function for training the prior distillation view synthesis model is as follows:
[0023]
[0024]
[0025]
[0026]
[0027]
[0028] in, This is a pseudo-target view. This is the final target view.
[0029] An unsupervised single-viewpoint synthesis apparatus, the apparatus comprising: a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the apparatus to perform any of the steps of the method described above.
[0030] The beneficial effects of the technical solution provided by this invention are:
[0031] 1. This invention utilizes the feature representation capabilities of deep learning to design a style-guided view synthesis model, thereby enabling the construction of view transformations between different poses to synthesize views at any target pose without label supervision.
[0032] 2. This invention designs a priori distillation view synthesis model to learn direct conversion between views, thereby improving the quality of synthesized views while reducing inference time in the synthesis process.
[0033] 3. The present invention achieves performance comparable to fully supervised single-viewpoint synthesis methods with only a single source view required. Attached Figure Description
[0034] Figure 1 A flowchart of an unsupervised single-viewpoint synthesis method;
[0035] Figure 2 This is a schematic diagram showing the comparison results between this method and other methods;
[0036] Figure 3 A diagram illustrating style manipulation;
[0037] Figure 4 This is a schematic diagram of view mapping operations;
[0038] Figure 5 This is a schematic diagram of a 3D rotation operation;
[0039] Figure 6 A schematic diagram of volume generation. Detailed Implementation
[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below.
[0041] The following examples illustrate the specific implementation of the unsupervised single-viewpoint synthesis method in this invention.
[0042] I. Constructing a Style-Guided View Composite Model
[0043] When only a single source view is available, the key to unsupervised single-view synthesis methods lies in how to learn view transformations between different poses. Therefore, this embodiment of the invention assumes a reference pose P. R To learn the intrinsic representation VR The intrinsic representation V R Able to reference pose P R The object is described here. Based on this, the intrinsic representation V R A reference pose P can be constructed. R and source pose P S View transitions between views are used to achieve unsupervised single-viewpoint composition.
[0044] To achieve the above objectives, embodiments of the present invention design a style-guided view synthesis model to learn view transformations in an unsupervised manner.
[0045] This invention utilizes a real source view I S Source P S and Gaussian noise G N The source view is synthesized as input to the style-guided view synthesis model. The method involves learning view transformations and then synthesizing pseudo-target views at arbitrary target poses from the learned view transformations. The style-guided view composition model consists of a style embedding strategy, 3D rotation operations, and view mapping operations.
[0046] First, the object's pose P in the reference position is obtained through the designed style embedding strategy. R The intrinsic representation of V at the location R .
[0047] Specifically, the designed style embedding strategy includes: a style generation part and a style manipulation part. The style generation part will use the real source view I S As input, the style vector S of the source view is extracted, as shown in the following formula:
[0048] S = O SGM (I S (1)
[0049] Among them, O SGM The style generation part consists of six stacked 2D convolutional layers and four stacked 1D convolutional layers.
[0050] Then Gaussian noise G N The style vector S is used as input to the style manipulation part to generate the intrinsic representation V. R The formula is as follows:
[0051] V R =O SMM (G N ,S) (2)
[0052] Among them, O SMM This refers to the style control section, such as... Figure 3As shown, it includes a 2D style manipulation unit and a 3D style manipulation unit. The 2D style manipulation unit employs an encoder-decoder network structure with skip connections. The encoder network consists of eight 2D convolutional layers and four adaptive instance normalization layers, wherein an adaptive instance normalization layer is stacked after every two 2D convolutional layers to embed the style vector S into Gaussian noise G in 2D space. N The decoder network consists of six stacked 2D convolutional layers. The 3D style manipulation unit consists of two 3D convolutional layers and two adaptive instance normalization layers, wherein each 3D convolutional layer is followed by an adaptive instance normalization layer to further embed the style vector S in 3D space.
[0053] Finally, through arbitrary pose P A Guided by 3D rotation operations and view mapping operations, this inherently represents V. R Synthetic target view The formula is as follows:
[0054]
[0055] Among them, O P Indicates view mapping operation, O R This indicates a 3D rotation operation. For example... Figure 4 and Figure 5 As shown, the view mapping operation consists of two 3D convolutional layers, eight 2D convolutional layers, and three residual blocks, while the 3D rotation operation consists of a trilinear sampling layer.
[0056] Since in unsupervised single-view synthesis methods, only a single source view can be used to train the network, therefore, the input real source view I... S Supervision should be provided for the network. To this end, the style-guided view synthesis model will incorporate P from the above formula. A Set as source pose P S Synthetic source view The formula is as follows:
[0057]
[0058] This invention employs multiple loss functions to constrain the synthesized source views obtained by the style-guided view synthesis model. With the real source view I S Consistency between them is required to achieve unsupervised single-viewpoint synthesis.
[0059] First, using photometric loss L P Feature loss L F and shape loss L S Constrain the composite source view respectively and the real source view I SThe formulas for appearance consistency, feature consistency, and shape consistency are as follows:
[0060]
[0061]
[0062]
[0063] Where V represents VGG-19 and Seg represents the segmentation of the computed view.
[0064] Furthermore, to further improve the quality of the synthesized view, embodiments of the present invention also use a similarity loss L. SSIM and combat losses L A To train the network, the formula is as follows:
[0065]
[0066]
[0067] Here, SSIM represents similarity calculation, D represents the discriminator, which consists of six convolutional layers, and E represents mathematical expectation.
[0068] In a single source view I S Under the given conditions, this embodiment of the invention designs a style-guided view synthesis model. First, in order to achieve unsupervised learning, this embodiment of the invention uses formula (4) to synthesize the source pose P. S View at the location Then, embodiments of the present invention use formulas (5) and (6) to constrain the synthesized source view. and the real source view I S Consistency between them guides the learning process of the view synthesis model by instructing it to follow a guiding style.
[0069] II. Constructing a Synthetic Model of Prior Distillation Views
[0070] Because unsupervised single-viewpoint synthesis requires the generation of an intrinsic representation V R Then from this inherent representation V R Synthetic target pose P T The target view at that location. However, this intrinsic representation V is generated. R Unsupervised single-viewpoint synthesis methods typically require additional inference time, resulting in poor real-time performance. To further reduce the inference time of viewpoint synthesis methods and improve their real-time performance, this invention proposes a priori distillation view synthesis model that synthesizes the target view directly.
[0071] The prior distillation view synthesis model will use the true source view I S Source PS and target pose P T As input, to synthesize P T Target view at the location The prior distillation view synthesis model includes: volume generation module, 3D rotation operation and view mapping operation.
[0072] First, view source I S As input to the volume generation module to generate P S The 3D volume at V is represented by S Based on the generated 3D volume representation V S At source pose P S and target pose P T Guided by 3D rotation and view mapping operations, the final target view is synthesized. The formula is as follows:
[0073]
[0074] Among them, O VGM This indicates the volume generation module, such as Figure 6 As shown, it contains seven 2D convolutional layers, three residual blocks, and two 3D convolutional layers.
[0075] Due to the lack of actual target view labels, obtaining the target view... Then, in order to learn from the source pose P S To the target pose P T The direct view transformation, in this embodiment of the invention, first transforms the P in the style-guided view composition model. A Set the target pose P T To synthesize a pseudo target view The formula is as follows:
[0076]
[0077] Then, in order to supervise the prior distillation view synthesis model in learning direct view transitions, this embodiment of the invention uses the same loss function as the style-guided view synthesis model to constrain the target view. and pseudo-target view To ensure consistency between the prior distilled view synthesis model, the loss function is formulated as follows:
[0078]
[0079]
[0080]
[0081]
[0082]
[0083] Thanks to the designed prior distillation view synthesis model (i.e., formula (7)), the embodiments of the present invention are able to synthesize high-quality target views with less inference time.
[0084] III. Constructing the Total Loss Function
[0085] To train the proposed style-guided view synthesis model, a loss function was constructed. The formula is as follows:
[0086]
[0087] Where α, β, γ and λ are set to 1, 5, 10 and 0.5 respectively.
[0088] Furthermore, a loss function was constructed to train the proposed prior distillation view synthesis model. The formula is as follows:
[0089]
[0090] IV. Networks for Training Unsupervised Single-Viewpoint Synthesis Methods
[0091] The training process of the unsupervised single-viewpoint synthesis method is divided into two stages. In the first stage, the style-guided view synthesis model is trained until convergence. In the second stage, the parameters of the model in the previous stage are fixed, and the prior distillation view synthesis model is trained until convergence.
[0092] V. Based on the source pose P S and target pose P T Guided by source view I S Composite target view
[0093] During the testing phase, this embodiment of the invention only uses the prior distillation view synthesis model for unsupervised single-viewpoint synthesis; the style-guided view synthesis model does not participate in the testing process. The comparison results between this embodiment and other methods are as follows: Figure 2 As shown.
[0094] An unsupervised single-viewpoint synthesis apparatus includes a processor and a memory, wherein the memory stores program instructions, and the processor invokes the program instructions stored in the memory to cause the apparatus to perform the following method steps:
[0095] Construct a style-guided view composition model consisting of a style embedding strategy, 3D rotation operations, and view mapping operations, with the real source view I. S Source P S and Gaussian noise G NUsed as input to synthesize the source view The method involves learning view transformations and then synthesizing pseudo-target views at arbitrary target poses from the learned view transformations.
[0096] Construct a priori distillation view synthesis model consisting of a volume generation module, 3D rotation operations, and view mapping operations, with the real source view I. S Source P S and arbitrary target pose P T As input, to synthesize arbitrary target pose P T The final target view at the location
[0097] Based on the style-guided view synthesis model and the prior distillation view synthesis model, a total loss function is constructed. The style-guided view synthesis model and the prior distillation view synthesis model are trained based on the total loss function, and then the target view is synthesized.
[0098] The style embedding strategy includes a style generation part and a style manipulation part. The style generation part will use the real source view I S As input, the style vector S of the source view is extracted as follows:
[0099] S = O SGM (I S )
[0100] Among them, O SGM The style generation part consists of six stacked 2D convolutional layers and four stacked 1D convolutional layers.
[0101] Gaussian noise G N The style vector S is used as input to the style manipulation part to generate the intrinsic representation V. R ,as follows:
[0102] V R =O SMM (G N ,S)
[0103] Among them, O SMM The style control section includes a 2D style control unit and a 3D style control unit.
[0104] Furthermore,
[0105] The 2D style manipulation unit employs an encoder-decoder network structure with skip connections. The encoder network consists of eight 2D convolutional layers and four adaptive instance normalization layers. An adaptive instance normalization layer is stacked after every two 2D convolutional layers to embed the style vector S into Gaussian noise G in 2D space. NThe decoder network consists of six stacked 2D convolutional layers.
[0106] The 3D style manipulation unit consists of two 3D convolutional layers and two adaptive instance normalization layers, wherein each 3D convolutional layer is followed by an adaptive instance normalization layer to further embed the style vector S in 3D space.
[0107] Among them, the source view I S As input to the volume generation module to generate P S The 3D volume at V is represented by S Based on the generated 3D volume representation V S At source pose P S and target pose P T Guided by 3D rotation and view mapping operations, the final target view is synthesized. as follows:
[0108]
[0109] The loss function for training the prior distillation view synthesis model is as follows:
[0110]
[0111]
[0112]
[0113]
[0114]
[0115] in, This is a pseudo-target view. This is the final target view.
[0116] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.
[0117] The execution entities of the aforementioned processor and memory can be devices with computing functions such as computers, microcontrollers, and single-chip microcomputers. In specific implementations, the embodiments of the present invention do not limit the execution entities and can select them according to the needs of actual applications.
[0118] Data signals are transmitted between the memory and the processor via a bus, which will not be elaborated upon in this embodiment of the invention.
[0119] Unless otherwise specified, the model numbers of the various devices in this embodiment of the invention are not limited, and any device that can perform the above functions is acceptable.
[0120] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0121] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. An unsupervised single-viewpoint synthesis method, characterized in that, The method includes: Construct a style-guided view composition model consisting of a style embedding strategy, 3D rotation operations, and view mapping operations, with a real source view. , source pose and Gaussian noise Used as input to synthesize the source view The method involves learning view transformations and then synthesizing pseudo-target views at arbitrary target poses from the learned view transformations. ; Construct a priori distillation view synthesis model consisting of a volume generation module, 3D rotation operations, and view mapping operations, with the real source view. , source pose and arbitrary target pose As input, to synthesize arbitrary target poses The final target view at the location ; Based on the style-guided view synthesis model and the prior distillation view synthesis model, a total loss function is constructed. The style-guided view synthesis model and the prior distillation view synthesis model are trained based on the total loss function, and then the target view is synthesized. The style embedding strategy includes a style generation part and a style manipulation part, wherein the style generation part will use the real source view. As input, extract the style vector from the source view. ,as follows: ； in, The style generation part consists of six stacked 2D convolutional layers and four stacked 1D convolutional layers. Gaussian noise and style vector As input to the style manipulation part, to generate the intrinsic representation ,as follows: ； in, The style control section includes a 2D style control unit and a 3D style control unit. Source view As input to the volume generation module to generate 3D volume representation at the location Based on the generated 3D volume representation In the source pose and target pose Guided by 3D rotation and view mapping operations, the final target view is synthesized. ,as follows: ; This indicates a view mapping operation. Indicates 3D rotation operation; The 2D style manipulation unit employs an encoder-decoder network structure with skip connections. The encoder network consists of eight 2D convolutional layers and four adaptive instance normalization layers. An adaptive instance normalization layer is stacked after every two 2D convolutional layers to normalize the style vector in 2D space. Embedded into Gaussian noise The decoder network consists of six stacked 2D convolutional layers. The 3D style manipulation unit consists of two 3D convolutional layers and two adaptive instance normalization layers, wherein each 3D convolutional layer is followed by an adaptive instance normalization layer to further embed style vectors in 3D space. .
2. The unsupervised single-viewpoint synthesis method according to claim 1, characterized in that, The loss function for training the prior distillation view synthesis model is as follows: ； ； ； ； ； in, This is a pseudo-target view. For the final target view; This indicates similarity calculation; This indicates the VGG-19 network.
3. An unsupervised single-viewpoint synthesis device, characterized in that, The device includes a processor and a memory, the memory storing program instructions, the processor calling the program instructions stored in the memory to cause the device to perform the steps of the method according to any one of claims 1-2.