Virtual reality data acquisition and repair method
By constructing a context-aware mechanism for multimodal adaptive fusion and a multi-scale guided diffusion model, the problem of insufficient semantic perception and multimodal information fusion in VR visual enhancement technology is solved, achieving high-fidelity visual enhancement under low-quality input conditions, and improving the generation quality of VR content and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QINGDAO KAILOT TECHNOLOGY CO LTD
- Filing Date
- 2025-12-12
- Publication Date
- 2026-07-07
AI Technical Summary
Existing VR visual enhancement technologies struggle to achieve high-fidelity visual enhancement with semantic perception and user control under low-quality input conditions. Furthermore, insufficient multimodal information fusion results in poor performance in terms of detail restoration, semantic consistency, and controllability.
A multimodal adaptive fusion context-aware mechanism is constructed. The content description text and user instruction vector are deeply fused through the self-attention mechanism of the adaptive Transformer. A multi-scale cross-attention layer is embedded in the U-Net decoder, and feature scale alignment is performed by combining a multi-scale fully connected neural network. An end-to-end trainable framework is constructed, and the sample library is weakly supervised for training by simulating the degradation process of real VR content.
It significantly improves the content relevance and user intent alignment of the augmented results, avoids issues of blurred details and structural distortion, meets the low latency and high interactivity requirements of VR applications, and enhances the model's generalization ability in real VR scenarios.
Smart Images
Figure CN121707869B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer data processing technology, and in particular relates to a method for virtual reality data acquisition and repair. Background Technology
[0002] Virtual reality (VR) technology, as the core carrier of immersive interactive experiences, has been widely used in education, healthcare, entertainment, and industrial simulation. With the continuous improvement of VR content resolution and real-time interaction requirements, the generation and enhancement of high-quality visual content has become a key technological challenge. Especially in real-time rendering environments, VR content often suffers from various forms of visual degradation due to transmission compression, rendering power limitations, or sensor noise, such as image blurring, loss of detail, and color distortion, severely impacting the user's immersion and visual comfort.
[0003] Currently, VR visual content enhancement technologies mainly rely on two mainstream methods: one is based on traditional image processing methods, such as filtering and denoising, super-resolution reconstruction, and contrast enhancement. These methods are computationally efficient but have limited ability to understand complex semantic content, making them difficult to adapt to the adaptive enhancement needs of various scenarios. The other is based on deep learning methods, especially generative models (such as Generative Adversarial Networks (GANs) and diffusion models), which can achieve high-quality image generation and restoration through end-to-end learning, but still have significant limitations in VR scenarios. Although generative models perform well in terms of visual quality, their generation process often lacks explicit modeling of the semantic context of VR content and user interaction intentions, resulting in insufficient consistency between the enhancement results and the scene content, or an inability to adapt to dynamically specified enhancement strategies (such as stylization and local enhancement). In addition, existing methods mostly rely on large-scale, high-quality paired data, while obtaining high-quality-degraded sample pairs in the VR field is costly and difficult to annotate, further limiting the generalization ability and practical deployment of the models.
[0004] On the other hand, existing technologies have significant shortcomings in handling multimodal information fusion. VR visual enhancement often requires combining content description text with user commands, but current methods mostly employ simple feature splicing or attention mechanisms, failing to achieve deep cross-modal alignment and guided generation. This results in poor performance in terms of detail restoration, semantic consistency, and controllability. Furthermore, generation methods such as diffusion models lack refined guidance on multi-scale features during the generation process, easily leading to structural distortion or blurred details, which is particularly prominent under the high-resolution output requirements of VR.
[0005] Based on the above-mentioned current state and shortcomings of the technology, there is an urgent need for a new VR visual content enhancement method that can achieve semantic perception and user-controllable high-fidelity visual enhancement under low-quality input conditions. Summary of the Invention
[0006] To address the above problems, this invention proposes a virtual reality data acquisition and repair method, comprising the following processes:
[0007] S1, through data collection and self-built methods, constructs an original VR visual dataset containing various virtual reality content;
[0008] S2 constructs a pairwise supervised sample library by simulating various degradation methods, and provides a description vector of the labeled content and a processing strategy vector for each sample pair in the pairwise supervised sample library;
[0009] S3: Obtain sample pairs and their corresponding content description vectors and processing strategy vectors from the paired supervised sample library; construct a context awareness and feature extraction module based on adaptive Transformer, and transform the degraded VR samples, content description vectors and processing strategy vectors in the sample pairs into image words, description words and instruction words respectively; finally, use the self-attention mechanism of Transformer to deeply fuse image words, description words and instruction words, and output a high-dimensional guided context feature sequence.
[0010] S4, a visual content restoration module is built and trained based on U-Net; firstly, the degraded samples are input into the virtual reality enhancement network, and the virtual reality enhancement network encodes the degraded samples layer by layer through the encoder and then inputs them into the decoder for feature enhancement to obtain the reconstructed VR samples;
[0011] During the feature enhancement process of the decoder, a multi-scale fully connected neural network is used to transform the feature scale of the guiding context feature sequence and align it with the output feature scale of each layer in the decoder.
[0012] Preferably, in step S2, a paired supervised sample library is constructed by simulating various degradation methods. This is achieved by taking the 8K 360° panoramic video dataset, 4K+PBR texture map set, high-precision 3D digital human asset set, multi-view light field dataset, self-built 8K spherical array original material set, self-built high-fidelity 360° video source, and self-built high-resolution PBR texture set from the original VR visual dataset obtained in step S1 as inputs. Degraded versions are generated according to a unified parameter scheme and paired with reference samples. This includes six operations: encoding / decoding degradation, stitching seam simulation, resolution degradation processing, texture downsampling, noise injection processing, and light field link simulation, resulting in six sets of degraded sample pairs.
[0013] Preferably, the seam simulation involves injecting controllable geometric misalignment and photometric discontinuity into the overlapping area of the self-built 8K spherical array original material set to obtain a simulated seam self-built 8K spherical array original material set. The degraded material set and the original self-built 8K spherical array original material set form a seam degraded sample pair set. Then, Mistika VR software is used for stitching to reflect the disturbance in the stitched result, forming a simulated seam self-built high-fidelity 360° video source. The degraded video source and the original self-built high-fidelity 360° video source simultaneously form a seam degraded sample pair set.
[0014] Preferably, the resolution degradation processing involves downsampling the 8K 360° panoramic video dataset and the self-built high-fidelity 360° video source three times to a specified level, including 4K, 2K, and 1K, to obtain a downsampled 8K 360° panoramic video dataset and a downsampled self-built high-fidelity 360° video source. These are then combined with the original 8K 360° panoramic video dataset and the original self-built high-fidelity 360° video source to form a multi-scale downsampled sample pair set.
[0015] Preferably, the S3 process specifically includes:
[0016] S31, Content Segmentation and Lexicalization: Input is a pair of samples obtained from a paired supervised sample library; first, the degraded VR images in the sample pair are spatially segmented into a series of fixed-size image patches without spatial overlap; then, each image patch is transformed into a one-dimensional feature vector through a linear projection layer, which is the image lexical.
[0017] S32, Conditional Lexical Embedding: The content description vector is projected into description lexicals through an independent fully connected network; the processing policy vector is projected into instruction lexicals through another independent fully connected network; both fully connected networks consist of an input layer, a ReLU activation function layer, and an output layer with the same hidden dimension as the Transformer; the description lexicals and instruction lexicals are consistent with the generated image lexicals in terms of dimension;
[0018] S33, Sequence Construction and Transformer Encoding: The descriptive and instruction terms generated in S32 are concatenated to the front end of the image terms generated in S31; simultaneously, to preserve spatial location information, learnable position embedding vectors are added to all image terms; the complete sequence containing position embedding vectors, descriptive terms, instruction terms, and image terms is input into a multi-layer Transformer encoder, and finally outputs a high-dimensional guiding context feature sequence; each term in the guiding context feature sequence not only encodes the deep visual information of its corresponding image patch, but also contains a global context instruction on what content restoration strategy should be performed on it.
[0019] Preferably, the linear projection layer in S31 is structurally a single-layer fully connected neural network, whose input dimension is equal to the total dimension of all pixels in an image block after flattening, and whose output dimension is set as the main hidden dimension of the Transformer encoder in S33.
[0020] Preferably, the main structure of the Transformer encoder in S33 consists of N stacked encoder blocks. Each encoder block sequentially includes a multi-head self-attention module, a feedforward neural network, and residual connections and layer normalization necessary to connect them. The intrinsic correlation between any two image blocks in the content, as well as between an image block and two conditional terms, is modeled through the calculation of the multi-head self-attention mechanism.
[0021] Preferably, the virtual reality augmentation network adopts a U-Net architecture, consisting of an encoder composed of an M-layer convolutional downsampling block, a decoder composed of an M-layer deconvolutional upsampling block, and skip connections connecting the corresponding layers of the encoder and decoder. Each convolutional downsampling block includes two standard convolutional layers, two ReLU activation functions, two normalization layers, and one max pooling layer. Each deconvolutional upsampling block includes a deconvolutional upsampling layer, a feature fusion layer, two standard convolutional layers, two ReLU activation functions, and two normalization layers. Cross-attention layers are integrated in multiple layers of the decoder. These cross-attention layers are used to cross-fuse the feature values of the decoder output and the guiding context feature sequence after feature scale transformation by a multi-scale fully connected neural network, thereby injecting precise guiding information into the pixel generation process.
[0022] Preferably, the multi-scale fully connected neural network has the same number of layers as the decoder, the input dimension of each layer is the same as the guiding context feature sequence, and the output dimension corresponds to the output dimension of each layer of the encoder. The multi-scale fully connected neural network consists of M parallel fully connected sub-networks. The input of each sub-network is a guiding context feature sequence, and the output dimension is designed to be of different scales to match the output feature scales of the M layers of the decoder in the virtual reality augmentation network. This allows the guiding context feature sequence to be specifically converted into M different scales of guiding features.
[0023] Preferably, the training process of the visual content restoration module is as follows:
[0024] Define a total time step as The forward noise sequence; this forward noise sequence consists of a preset variance sequence. Composition, in which From 1 to Based on this, the noise variance is defined. for And define the cumulative noise variance ;
[0025] The training process is based on the obtained paired supervised sample library. For each sample pair in the paired supervised sample library, firstly, the original VR samples in the sample pair are trained... Perform random sampling, with a time step of 1. And add Gaussian noise with a predetermined variance to it. Obtain noisy VR samples ;
[0026] Secondly, the noisy VR samples and their corresponding time steps are embedded. The corresponding guiding contextual feature sequence for this sample is input into the visual content restoration module; furthermore, the guiding contextual feature sequence is used as a condition by the cross-attention module to... The process is performed, and a reconstructed VR sample is output. Finally, the reconstructed VR sample and the original VR sample are compared. The L2 mean squared error loss is calculated, and the parameters of the visual content restoration module are optimized through backpropagation algorithm; finally, a pre-trained visual content restoration module is obtained.
[0027] Finally, the visual content restoration module, which had been initially trained, was fine-tuned, starting from the time step. Begin by performing the following operations in a loop until time step 0: First, embed the degraded VR sample from the current sample pair and the current time step. The system inputs the generated guiding contextual feature sequence of the sample into the pre-trained visual content inpainting module; secondly, it calculates the reconstructed VR sample and the original VR sample. The L2 mean squared error loss is calculated, and the parameters of the visual content restoration module are optimized through backpropagation algorithm to finally obtain the trained visual content restoration module.
[0028] Compared with the prior art, the present invention has the following innovative features:
[0029] 1. Context-aware mechanism for multimodal adaptive fusion: This invention proposes to encode the content description text and user instruction vector into a word sequence, and achieve deep fusion of three modalities of image, text and instruction through the self-attention mechanism of adaptive Transformer. This overcomes the limitations of traditional methods, such as simple multimodal information fusion and insufficient semantic alignment, and significantly improves the content relevance and user intent conformity of the enhanced results.
[0030] 2. Refined generation architecture of multi-scale guided diffusion model: Multi-scale cross-attention layer is embedded in U-Net decoder, and multi-scale fully connected network is designed to achieve scale alignment between guided features and the output of each layer of decoder. This allows the generation process to be precisely controlled by semantic and instruction information at different resolution levels, effectively avoiding problems such as blurred details and structural distortion, and improving the generation quality of high-resolution VR content.
[0031] 3. Paired Sample Synthesis and Weakly Supervised Training Strategy: A high-quality paired sample library is constructed by simulating the degradation process of real VR content. The model is trained under weak supervision by combining content description vectors and processing strategy vectors. This reduces the dependence on large-scale manually labeled data and improves the model's generalization ability and practical value in real VR scenarios.
[0032] 4. End-to-end trainable multi-stage collaborative optimization framework: Integrating context awareness, feature extraction and content restoration into an end-to-end trainable framework ensures the overall performance of the model in terms of generation quality, semantic consistency and visual comfort.
[0033] 5. Real-time interaction and dynamic strategy adjustment capabilities: It supports users to dynamically specify enhancement strategies through command vectors and combine them with a lightweight inference engine to achieve real-time content enhancement, meeting the core requirements of low latency and high interactivity in VR applications. Attached Figure Description
[0034] Figure 1 This is a flowchart illustrating the overall implementation logic of the present invention.
[0035] Figure 2 This is a flowchart of the degradation simulation and sample pair set construction process of the present invention.
[0036] Figure 3 This is a flowchart of the content metadata extraction process for this invention.
[0037] Figure 4 This is a key architecture diagram of the context awareness and feature extraction module based on adaptive Transformer in this invention.
[0038] Figure 5 This is a diagram of the virtual reality augmented network architecture of the present invention.
[0039] Figure 6 This diagram illustrates the distribution of quality improvements after repairing degraded VR content in an embodiment of the present invention.
[0040] Figure 7 This illustrates the convergence of the L2 mean squared error loss during model training in this embodiment of the invention.
[0041] Figure 8 This is an example image of a sample before repair in an embodiment of the present invention.
[0042] Figure 9 This is an example image of a repaired sample according to an embodiment of the present invention. Detailed Implementation
[0043] This invention provides a method for restoring virtual reality visual content, such as... Figure 1 As shown, its main process is as follows:
[0044] S1, through data collection and self-built methods, constructs an original VR visual dataset containing various virtual reality content;
[0045] S2 constructs a pairwise supervised sample library by simulating various degradation methods, and provides a description vector of the labeled content and a processing strategy vector for each sample pair in the pairwise supervised sample library;
[0046] S3: Obtain sample pairs and their corresponding content description vectors and processing strategy vectors from the paired supervised sample library; construct a context awareness and feature extraction module based on adaptive Transformer, and transform the degraded VR samples, content description vectors and processing strategy vectors in the sample pairs into image words, description words and instruction words respectively; finally, use the self-attention mechanism of Transformer to deeply fuse image words, description words and instruction words, and output a high-dimensional guided context feature sequence.
[0047] S4, a visual content restoration module is built and trained based on U-Net; firstly, the degraded samples are input into the virtual reality enhancement network, and the virtual reality enhancement network encodes the degraded samples layer by layer through the encoder and then inputs them into the decoder for feature enhancement to obtain the reconstructed VR samples;
[0048] During the feature enhancement process of the decoder, a multi-scale fully connected neural network is used to transform the feature scale of the guiding context feature sequence and align it with the output feature scale of each layer in the decoder.
[0049] The invention will be further described below with reference to specific embodiments.
[0050] I. Dataset Construction for Virtual Reality Visual Content
[0051] This step aims to build a structured dataset to provide a data foundation and guiding information for subsequent adaptive repair and enhancement. First, an original VR visual dataset containing various virtual reality (VR) content was constructed through collection and self-built methods. Second, a pairwise supervised sample library was built by simulating various degradation methods for model training. Finally, each sample in the pairwise supervised sample library was labeled with a content description vector and a processing strategy vector.
[0052] VR Visual Content Acquisition: This step involves acquiring diverse, high-fidelity visual content through two methods: obtaining publicly available commercial datasets and capturing content using a self-built system. Specifically, an 8K 360° panoramic video dataset was obtained from the Google Jump academic standard dataset; a 4K+PBR texture map set was obtained from the industrial-grade PBR asset library Quixel Megascans; a high-precision 3D digital human asset set was obtained using Epic Games' MetaHuman and high-precision 3D scanning process; a multi-view light field dataset was obtained from the Stanford Light Field Archive; simultaneously, this invention obtained a self-built 8K spherical array original material set by deploying a spherical array composed of multiple synchronized RED V-Raptor 8K cameras, and stitched it using Mistika VR software to generate a self-built high-fidelity 360° video source; in addition, a self-built high-resolution PBR texture set was generated using photogrammetry technology and the 3D reconstruction software RealityCapture; the 8K 360° panoramic video dataset, the 4K+PBR texture map set, the high-precision 3D digital human asset set, the multi-view light field dataset, the self-built 8K spherical array original material set, the self-built high-fidelity 360° video source, and the self-built high-resolution PBR texture set were used as the original VR visual dataset.
[0053] II. Sample Pair Set Construction and Content Metadata Extraction
[0054] 1. Degradation Simulation and Sample Pair Set Construction: This step takes the 8K 360° panoramic video dataset, 4K+PBR texture map set, high-precision 3D digital human asset set, multi-view light field dataset, self-built 8K spherical array original material set, self-built high-fidelity 360° video source, and self-built high-resolution PBR texture set from the original VR visual dataset obtained above as input. A degraded version is generated according to a unified parameter scheme, and paired with a reference sample set. This step consists of six operations: encoding / decoding degradation, stitching seam simulation, resolution degradation processing, texture downsampling, noise injection processing, and light field link simulation, resulting in six degraded sample pair sets, as detailed below:
[0055] Encoding / decoding degradation: For the 8K 360° panoramic video dataset and the self-built high-fidelity 360° video source, JPEG / H.265 encoding / decoding with multiple QF / QP settings is used to systematically introduce block artifacts, ringing, chroma subsampling, and time-drift distortion, resulting in an encoded / decoded 8K 360° panoramic video dataset and an encoded / decoded self-built high-fidelity 360° video source. These are then combined with the original 8K 360° panoramic video dataset and the original self-built high-fidelity 360° video source to form a set of encoding / decoding degradation sample pairs.
[0056] Simulation of seams: For the overlapping areas of the original 8K spherical array source material set, controllable geometric misalignment and photometric discontinuity are injected to obtain a simulated seam original 8K spherical array source material set. This degraded source material set and the original self-built 8K spherical array source material set form a set of degraded seam samples. Then, Mistika VR software is used to stitch the materials together, and the perturbation is reflected in the stitched result to form a simulated seam self-built high-fidelity 360° video source. This degraded video source and the original self-built high-fidelity 360° video source simultaneously form a set of degraded seam samples.
[0057] Resolution degradation processing: The 8K 360° panoramic video dataset and the self-built high-fidelity 360° video source are downsampled three times to a specified level, including 4K, 2K and 1K, to obtain the downsampled 8K 360° panoramic video dataset and the downsampled self-built high-fidelity 360° video source. These are then combined with the original 8K 360° panoramic video dataset and the original self-built high-fidelity 360° video source to form a multi-scale downsampled sample pair set.
[0058] Texture downsampling: Multi-scale downsampling is performed on the 4K+PBR texture map set and the self-built high-resolution PBR texture set according to channel consistency to obtain the downsampled 4K+PBR texture map set and the downsampled self-built high-resolution PBR texture set, which are then combined with the original 4K+PBR texture map set and the original self-built high-resolution PBR texture set to form a texture downsampling sample pair set.
[0059] Noise injection processing: Gaussian or Poisson noise of various intensities is injected into the obtained 8K 360° panoramic video dataset, the self-built high-fidelity 360° video source, the downsampled 4K+PBR texture map set, and the downsampled self-built high-resolution PBR texture set to obtain noisy 8K 360° panoramic video dataset, noisy self-built high-fidelity 360° video source, noisy 4K+PBR texture map set, and noisy self-built high-resolution PBR texture set. These are then combined with the original 8K 360° panoramic video dataset, the original self-built high-fidelity 360° video source, the original 4K+PBR texture map set, and the original self-built high-resolution PBR texture set to form a noise-degraded sample pair set.
[0060] Light field link simulation: Two types of link simulations are performed on the multi-view light field dataset: view-by-view independent encoding and group joint encoding. The resulting encoded and decoded multi-view light field dataset and the resampled joint encoded multi-view light field dataset are then combined with the original multi-view light field dataset to form a set of light field link degradation sample pairs.
[0061] This step ultimately outputs a paired supervised sample library containing diverse degradation types and intensity distributions, including sets of encoding / decoding degradation samples, seam degradation samples, multi-scale downsampling samples, texture downsampling samples, noise degradation samples, and optical field link degradation samples. This paired supervised sample library provides a complete data foundation for subsequent metadata extraction and adaptive conditional vector generation. The flowchart for this step is shown below. Figure 2 As shown.
[0062] 2. Content metadata extraction:
[0063] This step obtains a content description vector and a processing strategy vector for each sample pair in the paired supervised sample library. The content description vector is a high-dimensional numerical vector that encodes the static metadata of the content, specifically the content type and original resolution, as well as the degradation parameters of the record, specifically the encoding type and noise intensity, and the scene complexity quantified through image gradient and entropy analysis. The processing strategy vector is a three-dimensional numerical vector, with the first dimension being the peak signal-to-noise ratio of the image, the second dimension being the image processing latency in milliseconds, and the third dimension being the one-hot encoding of the visual content restoration strategy, which is annotated by experts. Finally, each sample pair in the paired supervised sample library has a corresponding content description vector and processing strategy vector. The flowchart of this step is as follows: Figure 3 As shown.
[0064] III. Design of Context Awareness and Feature Extraction Module Based on Adaptive Transformer
[0065] This step involves obtaining sample pairs and their corresponding content description vectors and processing policy vectors from the acquired paired supervised sample library. Next, a context-aware and feature extraction module based on adaptive Transformer is constructed to transform the degraded VR samples, content description vectors, and processing policy vectors in the sample pairs into image terms, description terms, and instruction terms, respectively. Finally, the image terms, description terms, and instruction terms are deeply fused using the Transformer's self-attention mechanism, ultimately outputting a high-dimensional guided context feature sequence. The flowchart and key model architecture diagram for this step are shown below. Figure 4 As shown.
[0066] Content Segmentation and Lexicalization: The input for this step is a pair of samples obtained from the paired supervised sample library. First, the degraded VR images in the sample pairs are spatially segmented into a series of fixed-size image patches without spatial overlap. Then, each image patch is transformed into a one-dimensional feature vector through a linear projection layer, which is the image lexical. The linear projection layer is structurally a single-layer fully connected neural network. Its input dimension is equal to the total dimension of all pixels in an image patch after flattening, and its output dimension is set as the main hidden dimension of the Transformer encoder in the subsequent process.
[0067] Conditional lexical embedding: This step projects the content description vector into description lexicals through a separate fully connected network; and projects the processing policy vector into instruction lexicals through another separate fully connected network. Both fully connected networks consist of an input layer, a ReLU activation function layer, and an output layer with the same hidden dimensions as the Transformer. The description and instruction lexicals maintain the same dimensionality as the image lexicals generated above.
[0068] Sequence Construction and Transformer Encoding: The generated descriptive and instruction terms are concatenated to the front of the generated image terms. Simultaneously, to preserve spatial location information, learnable positional embedding vectors are added to all image terms. The complete sequence containing positional embedding vectors, descriptive terms, instruction terms, and image terms is input into a multi-layer Transformer encoder. The main structure of this Transformer encoder consists of N stacked encoder blocks. Each encoder block sequentially contains a multi-head self-attention module, a feedforward neural network, and the necessary residual connections and layer normalization. The multi-head self-attention mechanism models the intrinsic relationships between any two image blocks in the content, and between an image block and two conditional terms. The final output of this step is a high-dimensional guiding context feature sequence. Each term in this guiding context feature sequence not only encodes the deep visual information of its corresponding image block but also contains global contextual instructions on which content restoration strategy should be executed, serving as the core guiding information for the visual content restoration module.
[0069] IV. Visual Content Repair Module
[0070] The visual content restoration module is based on U-Net and includes a virtual reality enhancement network and a multi-scale fully connected neural network. Degraded samples from sample pairs obtained from a paired supervised sample library are input into the virtual reality enhancement network. The virtual reality enhancement network performs layer-by-layer feature encoding on the degraded samples through an encoder and then inputs them into a decoder for feature enhancement to obtain reconstructed VR samples. The reconstructed VR samples are the restored VR samples. At the same time, during the feature enhancement process of the decoder, a multi-scale fully connected neural network is used to transform the feature scale of the guiding context feature sequence and align it with the feature scale of each layer output in the decoder.
[0071] 1. Visual content restoration module construction:
[0072] The virtual reality augmented network adopts the U-Net architecture, such as Figure 5As shown, the encoder consists of an M-layer convolutional downsampling block, a decoder consisting of an M-layer deconvolutional upsampling block, and skip connections connecting the corresponding layers of the encoder and decoder. Each convolutional downsampling block contains two standard convolutional layers, two ReLU activation functions, two normalization layers, and one max-pooling layer; each deconvolutional upsampling block contains one deconvolutional upsampling layer, one feature fusion layer, two standard convolutional layers, two ReLU activation functions, and two normalization layers. Cross-attention layers are integrated into multiple layers of the decoder. These cross-attention layers are used to cross-fuse feature values between the decoder output and the guiding context feature sequence after feature scale transformation by the multi-scale fully connected neural network, thereby injecting precise guiding information into the pixel generation process. The multi-scale fully connected neural network has the same number of layers as the decoder, with the input dimension of each layer being the same as the guiding context feature sequence, and the output dimension corresponding to the output dimension of each layer of the encoder. The multi-scale fully connected neural network consists of M parallel fully connected sub-networks. Each subnetwork takes a sequence of guiding contextual features as input, and their output dimensions are designed to match the output feature scales of the M-layer decoder in the virtual reality augmented network. This structure allows the guiding contextual feature sequence to be specifically converted into M different scales of guiding features, thereby providing accurate and scale-aligned guiding information for the decoder's pixel generation process at different feature levels in subsequent cross-attention layers, thus optimizing the reconstruction and restoration effect of VR samples.
[0073] 2. Visual Content Restoration Module Training
[0074] This step first defines a total time step as... The forward noise sequence. This forward noise sequence consists of a pre-defined variance sequence. Composition, in which From 1 to Based on this, noise variance is defined. for And define the cumulative noise variance as follows:
[0075]
[0076] The training process in this step is based on the obtained paired supervised sample library. For each sample pair in the paired supervised sample library, firstly, the original VR samples in the sample pair are trained... Perform random sampling, with a time step of 1. And add Gaussian noise with a predetermined variance to it. Obtain noisy VR samples The calculation formula for this process is:
[0077]
[0078] Secondly, embed the noisy VR sample and its corresponding time step. The network inputs the corresponding guiding contextual feature sequence for the sample pair into the visual content restoration module; furthermore, the network uses a cross-attention module to take the guiding contextual feature sequence as a condition for... The process is performed, and a reconstructed VR sample is output. Finally, the reconstructed VR sample and the original VR sample are compared. The L2 mean squared error loss is calculated, and the parameters of the visual content restoration module are optimized through backpropagation algorithm; finally, a pre-trained visual content restoration module is obtained; through this step, the model's feature extraction ability, denoising ability and generalization ability are enhanced;
[0079] Finally, the present invention performs fine-tuning training on the initially trained visual content restoration module to obtain a fully trained visual content restoration module; this process starts from the time step Begin by performing the following operations in a loop until time step 0: First, embed the degraded VR sample from the current sample pair and the current time step. The system inputs the generated guiding contextual feature sequence of the sample into the pre-trained visual content inpainting module; secondly, it calculates the reconstructed VR sample and the original VR sample. The L2 mean squared error loss is calculated, and the parameters of the visual content restoration module are optimized through backpropagation algorithm to finally obtain the trained visual content restoration module; through this step, the model has the ability to restore the six types of degraded samples.
[0080] V. Model Deployment for VR Applications
[0081] The purpose of this step is to deploy the adaptive Transformer-based context-aware and feature extraction module and the trained visual content restoration module to an industrial server to enable rapid restoration of VR content.
[0082] Model solidification: This step solidifies the parameters of the context-aware and feature extraction module based on the adaptive Transformer and the trained visual content restoration module.
[0083] Distributed system architecture deployment: This step deploys the solidified model to a distributed architecture; specifically, in the backend cloud data center, the solidified model is deployed as an elastically scalable microservice cluster using Kubernetes containerized orchestration technology.
[0084] Task Routing and Scheduling Implementation: This step describes the system's workflow when it receives a new VR content processing request. When a VR content processing request arrives at the system, it must contain the degraded VR content to be processed and a user-specified repair strategy. First, the degraded VR content is sent to the deployed microservice cluster, where a content description vector and a processing strategy vector are calculated based on the degraded VR content and the user-specified repair strategy. Second, the degraded VR content, content description vector, and processing strategy vector are used by an adaptive Transformer-based context awareness and feature extraction module to generate a guiding contextual feature sequence. Third, the degraded VR content and the generated guiding contextual feature sequence are input into the trained visual content repair module to obtain the repaired VR visual content.
[0085] VI. Experimental Comparison
[0086] This experiment aims to verify the effectiveness of the method of the present invention and provide technical verification for the adaptive processing and quality improvement of high-fidelity VR content. This experiment uses a combined dataset of the 360° panoramic video dataset, PBR texture map set, and multi-view light field dataset as the training and testing dataset. The preprocessing steps described above are performed on this dataset to obtain paired supervised sample libraries and corresponding content description vectors and processing strategy vectors. Training is performed using the L2 mean squared error loss defined above.
[0087] Figure 6 The graph clearly shows the distribution of quality improvement after restoring degraded VR content using the method of this invention. Warmer colors in the graph indicate a greater improvement in the Structure Metric Index (SSIM) value of the restored image compared to the original high-fidelity image. As can be seen from the graph, in areas where there was significant codec block artifacts, the heatmap shows a prominent bright red, indicating that the structural information in these areas has been restored.
[0088] Depend on Figure 7 As can be seen, the L2 mean squared error loss exhibits typical convergence characteristics during model training. In the early stages of training, the loss value decreases rapidly, indicating that the defined U-Net noise prediction network has learned the basic pattern of predicting noise from noisy content. In the middle stages of training, the loss decreases more slowly, and the model begins to refine its learning of the conditional information in the guided contextual feature sequences provided by the context awareness and feature extraction modules, distinguishing and processing differentiated restoration instructions for different degradation types through the cross-attention module. In the later stages of training, the loss curve tends to stabilize, indicating that the training process defined for the visual content restoration module is stable and effective.
[0089] Figure 8 For a sample in the dataset, the image before restoration. Figure 9The repaired image of this sample shows that the method of the present invention can effectively improve the resolution of the image and add more detailed features to the image.
[0090] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.
[0091] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.
Claims
1. A virtual reality data acquisition and repair method, characterized by, Includes the following processes: S1, through data collection and self-built methods, constructs an original VR visual dataset containing various virtual reality content; S2 constructs a pairwise supervised sample library by simulating various degradation methods, and provides a description vector of the labeled content and a processing strategy vector for each sample pair in the pairwise supervised sample library; S3: Obtain sample pairs and their corresponding content description vectors and processing strategy vectors from the paired supervised sample library; construct a context awareness and feature extraction module based on adaptive Transformer, and transform the degraded VR samples, content description vectors and processing strategy vectors in the sample pairs into image words, description words and instruction words respectively; finally, use the self-attention mechanism of Transformer to deeply fuse image words, description words and instruction words, and output a high-dimensional guided context feature sequence. The context awareness and feature extraction module, specifically the data processing procedure includes: S31, Content Segmentation and Lexicalization: Input is a pair of samples obtained from a paired supervised sample library; first, the degraded VR images in the sample pair are spatially segmented into a series of fixed-size image patches without spatial overlap; then, each image patch is transformed into a one-dimensional feature vector through a linear projection layer, which is the image lexical. S32, Conditional Lexical Embedding: The content description vector is projected into description lexicals through an independent fully connected network; the processing policy vector is projected into instruction lexicals through another independent fully connected network; both fully connected networks consist of an input layer, a ReLU activation function layer, and an output layer with the same hidden dimension as the Transformer; the description lexicals and instruction lexicals are consistent with the generated image lexicals in terms of dimension; S33, Sequence Construction and Transformer Encoding: The descriptive and instruction terms generated in S32 are concatenated to the front end of the image terms generated in S31; simultaneously, to preserve spatial location information, learnable position embedding vectors are added to all image terms; the complete sequence containing position embedding vectors, descriptive terms, instruction terms, and image terms is input into a multi-layer Transformer encoder, and finally outputs a high-dimensional guided context feature sequence; each term in the guided context feature sequence not only encodes the deep visual information of its corresponding image patch, but also contains a global context instruction on what content restoration strategy should be performed on it; S4 is a visual content restoration module built and trained based on U-Net. The module includes a virtual reality enhancement network and a multi-scale fully connected neural network. First, the degraded samples are input into the virtual reality enhancement network. The virtual reality enhancement network encodes the degraded samples layer by layer through the encoder and then inputs them into the decoder for feature enhancement to obtain the reconstructed VR samples. During the feature enhancement process of the decoder, a multi-scale fully connected neural network is used to transform the feature scale of the guiding context feature sequence and align it with the output feature scale of each layer in the decoder. The virtual reality augmentation network adopts a U-Net architecture, consisting of an encoder composed of an M-layer convolutional downsampling block, a decoder composed of an M-layer deconvolutional upsampling block, and skip connections connecting the corresponding layers of the encoder and decoder. Each convolutional downsampling block contains two standard convolutional layers, two ReLU activation functions, two normalization layers, and one max pooling layer. Each deconvolutional upsampling block contains one deconvolutional upsampling layer, one feature fusion layer, two standard convolutional layers, two ReLU activation functions, and two normalization layers. Cross-attention layers are integrated in multiple layers of the decoder. These cross-attention layers are used to cross-fuse the feature values of the decoder output and the guiding context feature sequence after feature scale transformation by a multi-scale fully connected neural network, thereby injecting precise guiding information into the pixel generation process.
2. The virtual reality data acquisition and repair method as described in claim 1, characterized in that: In step S2, a paired supervised sample library is constructed by simulating various degradation methods. It takes the 8K 360° panoramic video dataset, 4K+PBR texture map set, high-precision 3D digital human asset set, multi-view light field dataset, self-built 8K spherical array original material set, self-built high-fidelity 360° video source, and self-built high-resolution PBR texture set from the original VR visual dataset obtained in step S1 as inputs. A degraded version is generated according to a unified parameter scheme and paired with the reference to form a sample library. The process includes six operations: encoding and decoding degradation, stitching seam simulation, resolution degradation processing, texture downsampling, noise injection processing, and light field link simulation, which respectively yield six sets of degraded sample pairs.
3. The virtual reality data acquisition and repair method as described in claim 2, characterized in that: The stitching seam simulation involves injecting controllable geometric misalignment and photometric discontinuity into the overlapping areas of the original 8K spherical array source material set to obtain the simulated seam. The degraded source material set and the original 8K spherical array source material set form a set of degraded sample pairs for the stitching seam. Then, Mistika VR software is used to stitch the materials together, incorporating the disturbance into the stitched result to form a simulated seam high-fidelity 360° video source. The degraded video source and the original high-fidelity 360° video source simultaneously form a set of degraded sample pairs for the stitching seam.
4. The virtual reality data acquisition and repair method as described in claim 2, characterized in that: The resolution degradation process involves downsampling the 8K 360° panoramic video dataset and the self-built high-fidelity 360° video source three times to a specified level, including 4K, 2K, and 1K, to obtain a downsampled 8K 360° panoramic video dataset and a downsampled self-built high-fidelity 360° video source. These are then combined with the original 8K 360° panoramic video dataset and the original self-built high-fidelity 360° video source to form a multi-scale downsampled sample pair set.
5. The virtual reality data acquisition and repair method as described in claim 1, characterized in that: The linear projection layer in S31 is structurally a single-layer fully connected neural network. Its input dimension is equal to the total dimension of all pixels in an image patch after flattening, and its output dimension is set as the main hidden dimension of the Transformer encoder in S33.
6. The virtual reality data acquisition and repair method as described in claim 1, characterized in that: The main structure of the Transformer encoder in S33 consists of N stacked encoder blocks. Each encoder block contains a multi-head self-attention module, a feedforward neural network, and residual connections and layer normalization necessary to connect them. The intrinsic correlation between any two image blocks in the content, as well as between an image block and two conditional terms, is modeled through the calculation of the multi-head self-attention mechanism.
7. The virtual reality data acquisition and repair method as described in claim 1, characterized in that: The multi-scale fully connected neural network has the same number of layers as the decoder. The input dimension of each layer is the same as the guiding context feature sequence, and the output dimension corresponds to the output dimension of each layer of the encoder. The multi-scale fully connected neural network consists of M parallel fully connected sub-networks. The input of each sub-network is the guiding context feature sequence, and the output dimension is designed to be at different scales to match the output feature scales of the M layers of the decoder in the virtual reality augmented network. This allows the guiding context feature sequence to be specifically converted into guiding features of M different scales.
8. The virtual reality data acquisition and repair method as described in claim 1, characterized in that: The training process of the visual content restoration module is as follows: Define a total time step as The forward noise sequence; this forward noise sequence consists of a preset variance sequence. Composition, in which From 1 to Based on this, the noise variance is defined. for And define the cumulative noise variance ; The training process is based on the obtained paired supervised sample library. For each sample pair in the paired supervised sample library, firstly, the original VR samples in the sample pair are trained... Perform random sampling, with a time step of 1. And add Gaussian noise with a predetermined variance to it. Obtain noisy VR samples ; Secondly, the noisy VR samples and their corresponding time steps are embedded. The corresponding guiding contextual feature sequence for this sample is also input into the visual content restoration module; Furthermore, by using a cross-attention module, the guiding contextual feature sequence is taken as a condition for... The process is performed, and a reconstructed VR sample is output. Finally, the reconstructed VR sample and the original VR sample are compared. The L2 mean squared error loss is calculated, and the parameters of the visual content restoration module are optimized through backpropagation algorithm; finally, a pre-trained visual content restoration module is obtained. Finally, the visual content restoration module, which had been initially trained, was fine-tuned, starting from the time step. Begin by performing the following operations in a loop until time step 0: First, embed the degraded VR sample from the current sample pair and the current time step. The system inputs the generated guiding contextual feature sequence of the sample into the pre-trained visual content inpainting module; secondly, it calculates the reconstructed VR sample and the original VR sample. The L2 mean squared error loss is calculated, and the parameters of the visual content restoration module are optimized through backpropagation algorithm to finally obtain the trained visual content restoration module.