A method for quickly constructing virtual reality scenes

Through the improved generative adversarial network model, combining perspective consistency and style coding, the problems of consistency and personalization in VR scene construction are solved, and efficient and low-cost VR scene automatic generation is achieved. The generated scenes are consistent from different perspectives. And the style is controllable and supports complex scene construction.

CN118762131BActive Publication Date: 2024-11-22CHENGDU FUQIAN TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411244520.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-22
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

The scenes generated by existing VR scene construction methods lack consistency and personalized customization capabilities, are cost-effective and inefficient, and are difficult to support complex and detailed scenario construction.

Method used

The improved generative adversarial network model is adopted, combining perspective consistency and style coding, and through multi-view training and high-resolution texture generation, the automatic generation of VR scenes with coherent structures and rich details is achieved, and the scene style is supported flexibly adjustable.

Benefits of technology

The generated VR scene has good spatial continuity and structural consistency from different perspectives, and has a controllable style, which greatly improves the customized generation ability, with higher efficiency, lower cost and better results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118762131B_ABST
    Figure CN118762131B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of virtual reality, and in particular to a method for rapidly constructing a virtual reality scene. The method firstly annotates the collected data set at the pixel level and the instance level, and expands the data set to obtain a training set. Then, #imgabs0# is used to generate a low-resolution scene structure map from random noise, and #imgabs1# is used to generate a high-resolution texture map based on the structure map. Among them, #imgabs2# introduces a perspective consistency loss to ensure the consistency of the structure map under different perspectives; #imgabs3# utilizes adaptive instance normalization and adaptive spatial normalization to integrate style and structural features and enhance texture realism. Finally, the generated structure map and texture map are input into a rendering engine to synthesize a realistic virtual reality scene.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to a method for quickly constructing a virtual reality scene. Background Art

[0002] Virtual Reality (VR) technology aims to create an interactive and immersive virtual environment, allowing users to feel as if they are actually there. In recent years, VR technology has become increasingly mature, showing broad application prospects in many fields such as education and training, medical health, engineering simulation, and entertainment games.

[0003] High-quality VR content is the foundation of user experience, and scene construction is the most critical link. A realistic, detailed, and immersive virtual scene can greatly enhance the user's sense of immersion and interactivity. On the contrary, rough, distorted, and fragmented scenes will seriously damage the effect of VR applications.

[0004] Traditional VR scene construction mainly relies on graphics algorithms, such as geometric modeling, texture mapping, etc. Designers need to use modeling software such as 3D Studio Max and Maya, and spend a lot of time manually drawing scene elements. The whole process requires interdisciplinary professional skills, and places high demands on designers in various fields such as art, programming, and interaction.

[0005] In addition, traditional methods are difficult to achieve rich details and personalized customization, and the supported interaction methods are relatively single. For large and complex scenes, it is extremely costly to build them with manpower. Therefore, there is an urgent need to explore more automated and intelligent VR scene construction methods.

[0006] In recent years, some researchers have tried to use deep learning to generate VR scenes and have made some progress. The most representative one is based on generative adversarial networks (GANs). ) method. The basic idea of ​​this method is to train a The model takes the RGB image as input and generates the corresponding depth map. The 3D scene structure is restored through the depth information, and then the texture is synthesized to finally render the VR scene. The distribution of a large number of real scene images can be learned to generate virtual scenes.

[0007] However, the existing There are two main problems with this method: first, the generated scenes lack consistency, and the effects from different perspectives are not naturally connected; second, the style of the generated scenes is relatively single and lacks the ability to be customized. Summary of the invention

[0008] Regarding the aforementioned In order to overcome the limitations of the method, this paper proposes a novel VR scene automatic generation framework. The core idea is to and Two improved The models are combined to generate high-quality VR scenes from two levels: consistent structure and texture details.

[0009] First, use To capture the consistent structure of the scene. exist During the training process, the consistency constraints of multiple viewpoints in the scene are introduced. Specifically, the same scene is photographed from different viewpoints as training data, forcing the generator to learn the consistency of the scene structure when the viewpoint changes, thereby avoiding the fragmentation of rendering results from different viewpoints.

[0010] Secondly, use Generates high-resolution scene detail maps. Different from generating images directly from random noise, Style encoding is introduced to control the high-level semantic features of the generated image. The generated scene structure diagram is sent to , guiding it to generate detailed and realistic texture maps, and then synthesizing the final VR scene through the rendering engine.

[0011] comprehensive and The method of the present invention can automatically generate VR scenes with coherent structure and rich details, and support flexible adjustment of scene style. Compared with other methods, it is more efficient, less costly and more effective.

[0012] Specifically, the virtual reality scene rapid construction method proposed in the present invention is divided into three stages: data collection and preprocessing, Modeling and Modeling. The input of the entire model is random noise and reference texture , the output is a high-resolution scene texture map , and finally the final VR scene is synthesized through the rendering engine.

[0013] The training data comes from ScanNet, Matterport3D (MP3D) and self-collected outdoor scene data.

[0014] In the preprocessing stage, the original image Perform pixel-level and instance-level annotation. The former obtains the scene element segmentation map , which obtains the object mask . , , are the image height, width and number of scene element categories respectively.

[0015] 30% of the data is randomly labeled, and the rest is used for unsupervised training. Three types of data enhancement are performed at the same time: image transformation, scene synthesis, and perspective interpolation. The final dataset is obtained , the data set is divided into a training set, a validation set and a test set. Among them, For the A perspective.

[0016] To remove random noise Generate low-resolution scene structure diagram ,in is the noise dimension.

[0017] Generator Using U-Net structure, The discriminator use structure. The goal is to minimize the following loss:

[0018] ;

[0019] in, is the balance coefficient, is the perspective consistency loss:

[0020] ;

[0021] in, for and The interpolation perspective, is the corresponding real structure diagram.

[0022] by As the condition, for Generate a low-resolution scene structure map and generate a high-resolution texture map . It contains the encoder , Mapping Network and Generator .

[0023] The reference texture Encoded as style vector . Alternate use and Layers to merge and :

[0024] ;

[0025] ;

[0026] in, For the Layer feature map, , are the affine transformation parameters, , for The convolutional layer parameters.

[0027] The discriminator Adopt a multi-scale structure. The total loss is:

[0028] ;

[0029] in, is the generator adversarial loss,

[0030] is the discriminator adversarial loss,

[0031] is the style reconstruction loss,

[0032] For perceived loss. and is the balance coefficient, For pre-trained VGG network.

[0033] and The training process is:

[0034] Input: Dataset , initial parameters , , number of training rounds .

[0035] (1) Pre-training ;

[0036] (2) Fixed ,optimization , , of , , , ;

[0037] (3) Fixed ,optimization , of ;

[0038] (4) Repeat (2)-(3) Wheel until convergence.

[0039] Output: trained , , , .

[0040] During the testing phase, random noise is input and reference texture ,use Generate structure diagram , and then use , , Generating Texture Maps . Finally, the final VR scene is synthesized using a rendering engine.

[0041] In general, the beneficial technical effects of the present invention are as follows:

[0042] (1) Used for VR scene generation, it effectively solves the problem of scene fragmentation under different perspectives. The generated VR scene has good spatial continuity and structural consistency under any perspective.

[0043] (2) The style control capability is introduced into the VR scene synthesis process, making the scene style controllable and greatly improving the customized generation capability of VR scenes.

[0044] (3) Integration and Two different The improved model fully leverages their respective strengths to generate high-quality VR scenes at both the consistency and detail levels. DETAILED DESCRIPTION

[0045] The virtual reality scene rapid construction method proposed in the present invention is further explained in detail below, but the explanation is only for explanation and description, and it is not the only limitation to the implementation process of the present invention.

[0046] In order to train a high-quality scene generation model, it is necessary to collect a large amount of image data of real scenes. The data of this invention mainly comes from two public datasets: ScanNet and Matterport3D (MP3D).

[0047] In addition to the public dataset, we also used high-definition panoramic cameras to collect multiple outdoor scenes in multiple cities, covering different scene types such as streets, squares, parks, etc. During the collection process, 4-6 different perspectives were selected for each scene to supplement the perspective change data for the expansion of the dataset.

[0048] In order to better guide the model to learn the scene structure semantics, the original image data needs to be annotated. Specifically, the annotation content is divided into two levels:

[0049] (1) Pixel-level annotation. Using the FCN-based image segmentation model, each RGB image is annotated as semantic categories (such as ground, wall, ceiling, etc.). Suppose the input image is , pixel-level annotation is denoted as . Each element in Indicates that it is located Whether the pixel belongs to semantic categories.

[0050] (2) Instance-level annotation. Use the Mask R-CNN model to detect object instances (such as chairs, tables, potted plants, etc.) in each image and generate corresponding binary masks. The mask of an object is , Represents pixels Belong to object.

[0051] After pixel-level and instance-level annotation, each image is associated with a semantic segmentation map and several object masks, which can provide richer prior knowledge for the model to learn scene composition.

[0052] To further increase the size and diversity of training data, the following data augmentation strategies were adopted:

[0053] (1) Image transformation: The original image is randomly cropped, translated, rotated, mirrored, and color adjusted to generate 5-10 enhanced copies of each image. The transformed image shares semantic annotations with the original image. This transformation can improve the model's robustness to scene layout.

[0054] (2) Scene synthesis: Randomly extract foreground objects from scene images collected from different sources, scale them, rotate them, and place them in another background scene image to obtain a synthetic image. Through this "cut and paste" method, more complex scenes with object hierarchies can be obtained.

[0055] (3) Perspective interpolation: For different perspective images of the same scene, a new perspective is generated through spherical linear interpolation, and the image under the interpolated perspective is rendered from the RGB-D data of the existing perspective using techniques such as optical flow. This helps to learn continuous perspective transformations.

[0056] Of all the collected data, 80% were randomly selected as the training set, 10% as the validation set, and 10% as the test set.

[0057] The core of the method for rapidly generating virtual reality scenes proposed in this invention is to and The two generative models are cascaded to form a model that directly fuses consistency and detail textures. Responsible for random noise Generate a low-resolution scene structure diagram , by As a condition, further generate high-resolution scene texture map . Finally, and Enter the physically based rendering engine to synthesize realistic VR scenes.

[0058] The generator of , the noise space Mapping to scene structure space ; The generator of ,Will and style space Mapping to scene texture space . Then the entire scene generation process can be expressed as:

[0059] ;

[0060] in yes The style encoding vector of .

[0061] The advantage of this cascade structure is that it makes full use of the complementary characteristics of the two generative models: Good at learning the layout structure of the scene, and Good at controlling texture details. By decoupling them, the difficulty of generation can be significantly reduced. Output Guide , which ensures the consistency of texture and scene structure.

[0062] It is an improved method for VR scene generation tasks. model. It uses random noise As input, generate a low-resolution scene structure graph . is a The feature map of the channel indicates that each pixel belongs to The probability of a scene element category (such as ground, wall, ceiling, etc.)

[0063] Generator The U-Net structure is used. It is encoded into the latent space through a series of downsampling convolutional layers and then decoded to the original resolution through upsampling convolutional layers, and skip connections are added between the downsampling and upsampling paths to achieve the fusion of features of different scales.

[0064] The discriminator The input is Or real data , the output is a scalar representing the probability that the input comes from the true data distribution. Adopted The structure divides the input into multiple blocks for separate discrimination and then averages the results.

[0065] With the original different, Introduced perspective consistency loss during training . Using the structural diagram of the same scene in the dataset under different perspectives , this loss term encourages the generator to learn the consistency of scene structure under perspective changes, defined as:

[0066] ;

[0067] in yes and The interpolated viewing angle is is the corresponding real structure diagram.

[0068] In summary, The generator loss function is:

[0069] ;

[0070] The discriminator loss function is:

[0071] ;

[0072] in is the balance coefficient. By assigning weights to the above two loss functions, adversarial learning is performed, and finally Able to generate realistic and consistent scene structure diagrams.

[0073] It is an image generation model based on style transfer, which is good at capturing texture styles at different semantic levels. In this invention, it is used to generate high-resolution scene texture maps. .

[0074] It contains three key components: style encoder , Mapping Network and generator . The reference texture Encoded as style vector , depicted Texture features; Will In Mapping to latent space In , Enhance ability to express; by and As input, generate Consistent structure and Textures with similar style .

[0075] The structure and 2, including multiple adaptive instance normalization ( ) residual block. In each residual block, the feature map of the previous layer is first pass Normalized, then applied by The predicted affine transformation incorporates style information into texture features:

[0076] ;

[0077] in and yes Predicted affine transformation parameters.

[0078] Apart from , still Spatial adaptive normalization is added to ) layer to inject structural information. Layer For the condition, Perform pixel-by-pixel normalization and affine transformation:

[0079] ;

[0080] in and So The two convolutional layers are input to predict the pixel-by-pixel scale and offset parameters respectively.

[0081] By alternating and layer, The style and structure information are gradually integrated during the generation process, and finally a high-quality scene texture map is generated. .

[0082] The discriminator Contains multiple downsampling residual blocks and a pooling layer. The input is Or real texture map , the output is a scalar representing the probability that the input comes from the true data distribution.

[0083] The training loss consists of four items: the adversarial loss of the generator , adversarial loss of the discriminator , style reconstruction loss and perceived loss .

[0084] in, ;

[0085] ;

[0086] and They are used to constrain the reconstruction errors of generated images in the style space and perceptual feature space respectively:

[0087] ;

[0088] ;

[0089] in It is a pre-trained VGG network used to extract high-level semantic features.

[0090] Combining the above four items, The overall loss function is:

[0091] ;

[0092] in and To balance the weight.

[0093] Given a scene image and viewpoint dataset superior, and The training process is as follows:

[0094] Input: A set of image-view triples , and Initial parameters and , number of training rounds .

[0095] (1) Pre-training: On training of , extract semantic features of scene texture;

[0096] (2) Fixed , joint training , and ,optimization , , and , learn from and Generate realistic texture maps ;

[0097] (3) Fixed ,train of and ,optimization , and , learn from Generates structural consistency from different perspectives ;

[0098] (4) Repeat steps (2) to (3) wheel until convergence.

[0099] Output: trained , , , .

[0100] In the test phase, for any random noise and reference texture ,use Generate scene structure diagram , and then use , , Generate the corresponding texture map , and finally use the rendering engine to synthesize the VR scene.

[0101] In general, and By modeling the structure and texture of the scene separately, direct generation from random noise to high-quality VR scenes is achieved. Use perspective consistency loss to ensure the consistency of the generated structure graph under different perspectives; Style encoding and adaptive normalization layers are used to flexibly control the texture style of the scene.

[0102] In order to evaluate the proposed The effectiveness of the method in VR scene generation is demonstrated through experiments on two public datasets and compared with the current state-of-the-art methods.

[0103] (1) Dataset:

[0104] The following two commonly used VR / AR scene datasets were selected:

[0105] ;

[0106] The MP3D dataset contains 90 different indoor scenes, each with RGB-D images and 3D reconstruction models from 2-3 different perspectives; ScanNet is a larger indoor scene dataset, containing 1513 scenes, each with scan data from 3-5 perspectives. The dataset is randomly divided into training set, validation set and test set in a ratio of 8:1:1.

[0107] (2) Module comparison experiment:

[0108] In order to verify the effectiveness of each component and loss function, a series of module experiments were conducted, and the results are shown in the following table:

[0109] ;

[0110] It can be seen that removing (i.e. directly using Generating a texture map from noise) results in , , and The score dropped significantly, indicating The generated structure map is crucial for constraining texture synthesis; removing it slightly reduces , indicating that the perspective consistency loss helps to improve the structural consistency; removing or will be reduced to varying degrees , and , which shows that incorporating style and consistency information into different generation stages can improve texture quality.

[0111] (3) Comparison with other methods:

[0112] Will The results are compared with the following existing scene generation methods:

[0113] ;

[0114] In terms of generation quality, the indicators of the method of the present invention are superior to those of other methods. This is mainly due to: 1) The generated structure map provides more accurate prior information for subsequent texture synthesis; 2) The style information is integrated to make the texture richer and more realistic.

[0115] The present invention proposes a method based on and A fast virtual reality scene generation method. In view of the shortcomings of existing methods in scene structure consistency and texture realism, two improved The models are combined to achieve high-quality VR scene generation from random noise.

[0116] Specifically, first use Generate a low-resolution scene structure map from the noise, and then use A high-resolution texture map is generated based on the structure map, and finally a realistic VR scene is synthesized through the rendering engine. In , the perspective consistency loss is introduced to make the generated structure graph consistent under different perspectives; In this paper, adaptive instance normalization and adaptive spatial normalization are combined to incorporate style encoding and structural features respectively, enhancing the diversity and realism of texture.

[0117] Nevertheless, the current method still has some limitations. First, the generation of large-scale realistic scenes still faces the problems of high computational overhead and low rendering efficiency, which requires continuous optimization at both the hardware and software levels. Second, due to the lack of dynamic information and interactive functions, the currently generated VR scenes are static, and it is planned to introduce temporal dynamic information in subsequent research.

Claims

1. A method for rapidly constructing a virtual reality scene, characterized in that: The method is based on a generative adversarial network, which includes both styleGAN and cirCleGAN. The method first uses cirCleGAN to capture the consistent structure of the scene, then uses styleGAN to generate a high-resolution scene detail map, and then uses a rendering engine to synthesize the final VR scene. The generator G of cirCleGAN cir Using the U-Net structure, the discriminator D of cirCleGAN cir Using Patch GAN structure, G cir The goal is to minimize the following loss: Among them, λ view is the balance coefficient, L view is the perspective consistency loss: Where z is random noise, t ij t i and t j The interpolation perspective, is the corresponding real structure diagram, styleGAN is Generate a high-resolution texture map for the condition where A low-resolution scene structure diagram generated by cirCleGAN. StyleGAN includes encoder E, mapping network M and styleGAN generator G. sty , encoder E will refer to texture I r Encoded as style vector w, G sty Alternately use AdaIN and SPADE layers to fuse w and StyleGAN’s discriminator D sty Using a multi-scale structure, the total loss of styleGAN is: in, is the generator adversarial loss, is the discriminator adversarial loss, is the style reconstruction loss, is the perceptual loss, λ sty and λ per is the balance coefficient, and F is the pre-trained VGG network.

2. A method for rapidly constructing a virtual reality scene according to claim 1, characterized in that: The method also includes data collection and preprocessing, wherein the preprocessing includes: in the preprocessing stage, the original image I∈R H×W×3 Perform pixel-level and instance-level annotation. Pixel-level annotation obtains the scene element segmentation map S∈{0,1} H×W×N , instance-level annotation obtains the object mask M (m) ∈{0, 1} H×W , H, W, N are respectively the image height, width and the number of scene element categories; the method also includes data enhancement: the data enhancement method includes image transformation, scene synthesis and perspective interpolation, and finally obtains a data set The data set is divided into a training set, a validation set, and a test set, where t i is the i-th perspective.