A virtual reality scene dynamic adjustment method and system based on deep learning

Abstract

The application provides a kind of virtual reality scene dynamic adjustment method and system based on deep learning, it is related to virtual reality scene dynamic adjustment field, method includes obtaining interactive data and obtaining the initial fusion feature of visual element according to interactive data;According to the weight allocation processing of initial fusion feature to light and shadow effect and space structure to obtain element correlation matrix;Element correlation matrix is detected to the region of light and shadow effect and space structure mismatch to obtain uncoordinated area;Judge whether the proportion of uncoordinated area exceeds preset threshold;If yes, then generate alternative light and shadow effect variant for uncoordinated area, and obtain realistic evaluation value, and then obtain optimized element fusion scheme, again to dynamic adaptation input sequence Time series modeling processing, obtain real-time updated scene generation parameter, to dynamically adjust virtual reality scene.The application can dynamically and accurately optimize virtual reality scene visual element, meet the needs of users to immersive and personalized experience.

Description

Technical Field

[0001] This invention relates to the field of virtual reality scene dynamic adjustment technology, and in particular to a method and system for virtual reality scene dynamic adjustment based on deep learning. Background Technology

[0002] With the rapid development of virtual reality technology, virtual reality scene generation technology has emerged, which brings users an immersive experience by constructing realistic virtual environments.

[0003] Virtual reality (VR) scenes are interactive virtual environments constructed through technology, enabling users to have an immersive sensory experience. The core of VR is simulating visual, auditory, and even tactile perceptions, blurring the boundaries between virtual and reality. In the field of virtual reality, the visual effects of the scene are crucial; the harmony between lighting effects and spatial structure directly affects the user's immersion and experience quality.

[0004] In existing technologies, the lighting effects and spatial structure of virtual reality scenes are usually pre-set, lacking the ability to adjust in real time based on user interaction data. For example, in some virtual reality games, the lighting and spatial layout of the scene are fixed and do not change regardless of how the user interacts with the scene. However, this pre-set, fixed mode of virtual reality scenes has obvious limitations: on the one hand, it cannot meet the personalized preferences of different users, because each user may have different preferences for lighting effects and spatial structure; on the other hand, when the lighting effects and spatial structure in the scene do not match, it will lead to visual disharmony and affect the user's immersion. This disharmony may manifest as a mismatch between lighting and object shapes, or inconsistencies between lighting changes and spatial layout, thereby reducing the overall quality of the virtual reality scene. Summary of the Invention

[0005] Based on this, the purpose of this invention is to provide a method and system for dynamic adjustment of virtual reality scenes based on deep learning, which solves the technical problems in the prior art where virtual reality scenes are pre-set in a fixed pattern and lack the ability to adjust in real time according to user interaction data. This results in the inability to meet the personalized preferences of different users, and when the lighting effects and spatial structure in the scene do not match, it leads to visual disharmony and affects the user's immersion.

[0006] This invention provides a method for dynamically adjusting virtual reality scenes based on deep learning, comprising:

[0007] The system acquires user interaction data in a virtual reality scene and combines it with a feature extraction algorithm to extract user preference information and environmental change information from the interaction data to obtain the initial fusion features of visual elements.

[0008] The initial fusion features are used to perform weighted processing on the lighting effects and spatial structure to obtain an element correlation matrix; the element correlation matrix is ​​then used to detect regions where the lighting effects and spatial structure do not match to obtain incompatible regions;

[0009] Determine whether the proportion of incompatible areas exceeds a preset threshold;

[0010] If so, then a pre-trained conditional generative adversarial network is used to generate alternative lighting effect variants for the incongruent region, and the realism of the alternative lighting effect variants is evaluated to obtain a realism evaluation value.

[0011] Based on the realism assessment value, user preference information and spatial structure are fused to obtain an optimized element fusion scheme. Based on the optimized element fusion scheme, the dynamically adaptive input sequence is subjected to temporal modeling processing to obtain real-time updated scene generation parameters, so as to dynamically adjust the presentation effect of the virtual reality scene according to the real-time updated scene generation parameters.

[0012] The aforementioned deep learning-based method for dynamically adjusting virtual reality scenes, by extracting user preference and environmental change information from interaction data, can overcome the shortcomings of existing technologies that do not consider user preferences and environmental dynamics, providing a precise basis for visual element optimization. Secondly, weight allocation processing can establish a correlation between lighting effects and spatial structure. Then, inconsistency region detection can proactively identify mismatches between the two, avoiding the neglect of inconsistencies in existing technologies. Furthermore, generating alternative lighting effect variants and evaluating their realism solves the problem of existing technologies lacking effective optimization methods. Moreover, by integrating user preference information and spatial structure to obtain optimization schemes and time-series modeling, scene parameters are updated in real time, overcoming the shortcomings of insufficient dynamic adjustment in existing technologies. This allows for dynamic and precise optimization of visual elements in virtual reality scenes, effectively solving the problem of poor scene presentation in existing technologies and meeting users' needs for immersion and personalized experiences.

[0013] In addition, the deep learning-based virtual reality scene dynamic adjustment method of the present invention may also have the following additional technical features:

[0014] Furthermore, the step of generating alternative lighting effect variants for incongruous regions using a pre-trained conditional generative adversarial network includes:

[0015] The spatial coordinates and lighting parameters of the incongruent area are obtained, and an initial lighting feature vector is constructed based on the spatial coordinates and lighting parameters of the incongruent area. The initial lighting feature vector includes the lighting direction, intensity, and color attributes.

[0016] The initial light and shadow feature vector is input into a pre-trained conditional generative adversarial network to reconstruct the light and shadow effect, thereby obtaining a preliminary light and shadow variant; the preliminary light and shadow variant is then feathered at the edges to obtain a processed smooth light and shadow variant.

[0017] By combining the three-dimensional normal information of the spatial structure, the conflict between light and shadow and geometric surface in the smooth light and shadow variant is eliminated to obtain a geometrically adapted light and shadow variant; the geometrically adapted light and shadow variant is subjected to real-time rendering test to generate the alternative light and shadow effect variant.

[0018] Furthermore, methods for generating pre-trained conditional generative adversarial networks include:

[0019] A training dataset is obtained by labeling light and shadow sample data from historical virtual reality scenes. The training dataset includes lighting direction, intensity, and color attributes.

[0020] The training dataset is input into the generative adversarial network to train the generator and discriminator adversarially, resulting in an initial generative adversarial network with the ability to generate light and shadow.

[0021] The pixel-level difference loss between the generator's output in the initial generative adversarial network and the real light and shadow sample is calculated to obtain the loss function value;

[0022] Based on the loss function value, the parameters of the generator and discriminator networks are updated by backpropagation to obtain the updated generative adversarial network.

[0023] Determine whether the loss function value of the generative adversarial network after parameter update has converged to the convergence threshold;

[0024] If not, the initial generative adversarial network is updated according to the parameter-updated generative adversarial network, and the process returns to the step of calculating the pixel-level difference loss between the generator output in the initial generative adversarial network and the real light and shadow sample, until the loss function value converges to the convergence threshold, thus obtaining the pre-trained conditional generative adversarial network.

[0025] The formula for calculating the loss function is as follows:

[0026] ;

[0027] In the formula, Let G( be the loss function value). x () represents the lighting and shadow effects output by the generator. y As a real light and shadow sample, N For the total number of pixels, |G( x ) i - y i |No.i L1 difference of 1 pixel, SSIM(G( x ),y) is the structural similarity index between the generated result and the real sample. f (·) represents a pre-trained visual feature extraction network. a、b、c These are the weighting coefficients.

[0028] Further, the step of evaluating the realism of the candidate lighting effect variants to obtain a realism evaluation value includes:

[0029] Based on the candidate lighting effect variants, calculate the average brightness of all pixels in the variants to obtain the overall average brightness.

[0030] Based on the candidate light and shadow effect variants, identify the highlight areas in the variants and calculate the area ratio of the highlight areas to obtain the highlight area ratio;

[0031] Based on the candidate lighting effect variants, pixel gradient analysis is performed on the edges of the shadow areas in the variants to obtain the shadow edge sharpness value;

[0032] Based on the candidate lighting effect variants, calculate the chromaticity difference between adjacent pixels in the variants, and sum the difference values ​​to obtain the total chromaticity jump;

[0033] The overall brightness average, the proportion of the highlight area, the sharpness value of the shadow edge, and the sum of the chromaticity jumps are weighted and summed to obtain the realism evaluation value.

[0034] Furthermore, the step of fusing user preference information with spatial structure based on the realism assessment value to obtain an optimized element fusion scheme includes:

[0035] Based on the realism assessment value, the user preference information is weighted to obtain a preference weight matrix;

[0036] For the spatial structure, extract the structural feature vector to obtain the spatial structure feature matrix;

[0037] Based on the preference weight matrix and the spatial structure feature matrix, a weighted fusion scheme for initial elements is generated.

[0038] The initial element fusion scheme is subjected to multi-objective optimization to obtain a set of candidate fusion schemes;

[0039] The candidate fusion scheme set is dynamically simulated and verified to obtain a simulation verification result dataset;

[0040] Feasibility scores are calculated based on the simulation verification result dataset to obtain the feasibility score of the scheme;

[0041] The feasibility scores of the proposed schemes are ranked from highest to lowest, and the candidate fusion scheme with the highest feasibility score is selected as the optimized element fusion scheme.

[0042] Furthermore, the step of performing dynamic simulation verification on the candidate fusion scheme set to obtain the simulation verification result dataset includes:

[0043] A simulation scene model is constructed based on the candidate fusion scheme set. The simulation scene model includes geometric structure, material properties, and lighting parameters. The simulation scene model is then used to perform dynamic simulations through a physics engine to obtain object motion trajectory data.

[0044] Based on the object's motion trajectory data, the light and shadow parameters are tracked and calculated in real time to generate a dynamic light and shadow change sequence; inter-frame difference analysis is performed on the dynamic light and shadow change sequence to obtain a light and shadow coherence index.

[0045] The performance of each scheme in the candidate fusion scheme set is scored based on the light and shadow coherence index, and a simulation verification result dataset is generated.

[0046] Furthermore, the step of performing time-series modeling on the dynamically adaptive input sequence to obtain real-time updated scene generation parameters, based on the optimized element fusion scheme, includes:

[0047] Based on the optimized element fusion scheme, the continuous change features of user interaction behavior in the time dimension are extracted to generate a dynamically adaptive input sequence; based on the dynamically adaptive input sequence, time windows are divided to obtain time-series segment data of fixed length;

[0048] State dependencies are calculated on the time-series data to obtain the internal state variables at the current time; feature weighted fusion is performed on the internal state variables at the current time to obtain the feature representation enhanced for key time steps.

[0049] Based on the enhanced feature representation of the key time step, a nonlinear mapping transformation is performed to generate a scene parameter prediction vector; the scene parameter prediction vector is then subjected to numerical range constraint processing to obtain the scene generation parameters that are updated in real time.

[0050] Another aspect of the present invention provides a deep learning-based virtual reality scene dynamic adjustment system, the system comprising:

[0051] The acquisition module is used to acquire user interaction data in a virtual reality scene, and combine it with a feature extraction algorithm to extract user preference information and environmental change information from the interaction data to obtain the initial fusion features of visual elements.

[0052] The detection module is used to perform weight allocation processing on the lighting effects and spatial structure according to the initial fusion features to obtain an element correlation matrix; and to detect regions where the lighting effects and spatial structure do not match the element correlation matrix to obtain incompatible regions.

[0053] The judgment module is used to determine whether the proportion of incompatible areas exceeds a preset threshold.

[0054] The first execution module is used to generate alternative lighting effect variants for the incongruent regions through a pre-trained conditional generative adversarial network if the proportion of incongruent regions exceeds a preset threshold, and to evaluate the realism of the alternative lighting effect variants to obtain a realism evaluation value.

[0055] The dynamic adjustment module is used to fuse user preference information with spatial structure based on the realism evaluation value to obtain an optimized element fusion scheme. Based on the optimized element fusion scheme, the module performs temporal modeling processing on the dynamically adaptive input sequence to obtain real-time updated scene generation parameters, so as to dynamically adjust the presentation effect of the virtual reality scene according to the real-time updated scene generation parameters.

[0056] In another aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the deep learning-based virtual reality scene dynamic adjustment method described above.

[0057] In another aspect, the present invention provides a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the deep learning-based virtual reality scene dynamic adjustment method described above. Attached Figure Description

[0058] Figure 1 This is a flowchart of the method for dynamically adjusting virtual reality scenes based on deep learning in the first embodiment of the present invention;

[0059] Figure 2 This is a flowchart of the deep learning-based virtual reality scene dynamic adjustment method in the second embodiment of the present invention.

[0060] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0061] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0063] To address the limitations of existing technologies that pre-set fixed patterns for virtual reality scenes, lacking the ability to adjust in real-time based on user interaction data, thus failing to meet the personalized preferences of different users and causing visual disharmony when lighting effects and spatial structures in the scene are mismatched, affecting user immersion, this application provides a deep learning-based method and system for dynamic adjustment of virtual reality scenes. Specifically: First, by extracting user preference information and environmental change information from interaction data, the shortcomings of existing technologies in not considering user preferences and environmental dynamics can be overcome, providing a precise basis for visual element optimization. Second, weight allocation processing can establish a correlation between lighting effects and spatial structure. Third, inconsistency area detection can proactively identify mismatches between the two, avoiding the neglect of inconsistencies in existing technologies. Fourth, generating alternative lighting effect variants and evaluating their realism solves the problem of existing technologies lacking effective optimization methods. Fifth, by integrating user preference information and spatial structure to obtain optimization schemes and time-series modeling, scene parameters are updated in real-time, overcoming the shortcomings of insufficient dynamic adjustment in existing technologies. This enables dynamic and precise optimization of visual elements in virtual reality scenes, effectively solving the problem of poor scene presentation in existing technologies and meeting users' needs for immersion and personalized experiences.

[0064] To facilitate understanding of the present invention, several embodiments are given below. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that the disclosure of the present invention will be more thorough and complete.

[0065] Example 1

[0066] Please see Figure 1 The figure shows a method for dynamically adjusting a virtual reality scene based on deep learning in the first embodiment of the present invention, the method comprising steps S101 to S105:

[0067] S101. Acquire user interaction data in the virtual reality scene, and combine it with feature extraction algorithm to extract user preference information and environmental change information from the interaction data to obtain the initial fusion features of visual elements.

[0068] In virtual reality scenarios, user interaction data contains rich information reflecting user preferences and environmental changes. Specifically, by analyzing user movement trajectories, dwell time, and interaction frequency, user preference information can be extracted, such as whether the user prefers bright or soft lighting effects, or open or compact spatial structures. Simultaneously, by monitoring and analyzing changes in the position, shape, and lighting of objects within the scene, environmental change features can be extracted. These feature data are then processed using algorithms, such as feature extraction algorithms, to transform the raw interaction data into initial fused features that characterize visual elements. This process involves multiple stages, including data collection, preprocessing, and feature extraction. Data collection can be achieved through sensors in the virtual reality device; preprocessing includes data cleaning and normalization; and feature extraction can employ common methods such as principal component analysis and linear discriminant analysis to ensure that the resulting initial fused features accurately reflect user preferences and environmental changes.

[0069] S102. Based on the initial fusion features, weight allocation processing is performed on the lighting effects and spatial structure to obtain the element correlation matrix; the element correlation matrix is ​​then used to detect regions where the lighting effects and spatial structure do not match to obtain the incompatible regions.

[0070] Weight allocation involves assigning different weights to lighting effects and spatial structure based on the importance of each feature in the initial fusion feature set. This process can be implemented by constructing a weight allocation model, which can be either rule-based or machine learning-based. A rule-based model can assign higher weights to lighting effects based on predefined rules, such as prioritizing lighting effects over spatial structure in user preference information. A machine learning-based model can learn the mapping relationship between features and weights through training data, thus automatically assigning weights to lighting effects and spatial structure. Through weight allocation, an element-valued correlation matrix is ​​obtained, reflecting the strength of the correlation between lighting effects and spatial structure. This correlation matrix can be constructed using matrix operations, such as multiplying the feature vectors of lighting effects and spatial structure to obtain a matrix representing their degree of correlation.

[0071] Specifically, incongruent regions refer to areas in a virtual reality scene that are numerically abnormal and require optimization, identified by analyzing an element association matrix containing information about the relationship between lighting effects and spatial structure using region detection algorithms in image processing techniques such as edge detection and region growing. The element association matrix contains information about the relationship between lighting effects and spatial structure; by analyzing this matrix, areas where lighting effects and spatial structure do not match can be detected. Specifically, region detection algorithms in image processing techniques, such as edge detection and region growing, can be used to identify regions in the matrix that represent mismatches. These algorithms can detect areas with numerically abnormal values, which typically correspond to parts where lighting effects and spatial structure are incongruous. For example, edge detection algorithms can detect edges by calculating the differences between adjacent elements in the matrix, while region growing algorithms can start from a seed point and gradually expand to regions similar to the seed point, thereby identifying mismatched regions. Detected incongruent regions can be processed by labeling or extraction. The above process involves the technical principles of image processing and pattern recognition; through corresponding operational techniques, the parts of a virtual reality scene that need optimization can be effectively identified.

[0072] S103. Determine whether the proportion of the incoherent area exceeds the preset threshold.

[0073] If the proportion of the non-coordinated area exceeds the preset threshold, then step S104 is executed;

[0074] When the proportion of detected inconsistencies exceeds a preset threshold, it indicates a severe mismatch between the lighting effects and spatial structure in the scene, requiring optimization. In this case, a pre-trained Conditional Generative Adversarial Network (CGAN) can be used to generate alternative lighting effect variants. A CGAN is a deep learning model consisting of a generator and a discriminator. The generator is responsible for generating new lighting effect variants, while the discriminator determines whether the generated variants are realistic. During training, the generator and discriminator compete against each other. The generator continuously learns how to generate more realistic lighting effect variants, while the discriminator continuously learns how to distinguish between generated variants and real lighting effects. Through this method, the generator can learn various features and patterns of lighting effects, thus generating high-quality alternative lighting effect variants. The pre-trained CGAN is trained using a large amount of lighting sample data from historical virtual reality scenes. This sample data contains information on lighting effects and spatial structure in different scenes, and through data annotation, it provides the generator and discriminator with the necessary supervision information for training. When generating alternative lighting effect variants, the spatial coordinates and lighting parameters of the incompatible areas can be input into the generator as conditions. The generator will then generate matching lighting effect variants based on the corresponding conditions, thus generating alternative lighting effect variants.

[0075] S104. Using a pre-trained conditional generative adversarial network, alternative lighting effect variants are generated for the incongruous regions, and the realism of the alternative lighting effect variants is evaluated to obtain a realism evaluation value.

[0076] For example, after generating candidate lighting effect variants, a realism evaluation is required to determine their similarity to realistic lighting effects. This realism evaluation can be achieved through a series of image analysis methods, including calculating the average brightness of all pixels in the variant, identifying highlight areas and calculating their area proportions, performing pixel gradient analysis on the edges of shadow areas, and calculating chromaticity differences between adjacent pixels. The average brightness reflects the overall brightness of the lighting effect, the highlight area proportion reflects the naturalness of the highlights, the shadow edge sharpness assesses the clarity of shadow edges, and the total chromaticity jump measures the reasonableness of color changes in the lighting effect. Through comprehensive analysis of these features, a realism evaluation value is obtained, which measures the degree of realism of the candidate lighting effect variants. The realism evaluation value can be calculated using a weighted summation, where the weights of each feature are allocated according to their influence on realism.

[0077] S105. Based on the realism assessment value, user preference information and spatial structure are fused to obtain an optimized element fusion scheme. Based on the optimized element fusion scheme, the dynamic adaptive input sequence is processed by temporal modeling to obtain real-time updated scene generation parameters, so as to dynamically adjust the presentation effect of the virtual reality scene according to the real-time updated scene generation parameters.

[0078] Optionally, based on the realism assessment value, user preference information and spatial structure can be fused to obtain an optimized element fusion scheme. First, user preference information is weighted according to the realism assessment value to obtain a preference weight matrix. This process can be implemented by constructing a weight allocation model, which dynamically adjusts the weights of each feature in the user preference information based on the realism assessment value. For example, if a candidate lighting effect variant has a high realism assessment value, it indicates that the variant is closer to a realistic lighting effect. In this case, the weights of features in the user preference information that have a smaller impact on the variant can be appropriately reduced to better fuse user preference information and spatial structure. Next, structural feature vectors are extracted from the spatial structure to obtain a spatial structure feature matrix. The structural feature vectors can contain various feature information of the spatial structure, such as the shape, size, and position of objects. Then, the preference weight matrix and the spatial structure feature matrix are weighted and fused to generate an initial element fusion scheme. The fusion process can be implemented through matrix operations, such as element-wise multiplication or addition of the two matrices, to obtain a fused matrix that represents the preliminary fusion result of user preference information and spatial structure. Finally, the initial element fusion scheme is subjected to multi-objective optimization to obtain a set of candidate fusion schemes. Multi-objective optimization can consider multiple optimization objectives, such as user satisfaction, scene realism, and computational efficiency. By using optimization algorithms to weigh these objectives, a set of candidate fusion schemes, i.e., the optimized element fusion scheme, is obtained.

[0079] Specifically, firstly, based on the optimized element fusion scheme, the continuous change features of user interaction behavior over time are extracted to generate a dynamically adaptive input sequence. This process can be achieved through time series analysis of user interaction data, for example, using a sliding window method. The user's interaction behavior over a period of time is treated as a window of data, and the window is gradually moved to obtain a series of window data, which constitute the dynamically adaptive input sequence. Next, time windows are divided based on the dynamically adaptive input sequence to obtain fixed-length time-series segments. The purpose of time window division is to segment the continuous dynamically adaptive input sequence into multiple fixed-length segments. Then, state dependency calculation is performed on the time-series segment data to obtain the internal state variables at the current time step. State dependency calculation can be achieved by constructing a state transition model, which can calculate the state at the next time step based on the current state and the dynamically adaptive input sequence, thus obtaining the internal state variables. Feature weighted fusion processing is then performed on the internal state variables at the current time step to obtain enhanced feature representations for key time steps. Feature weighted fusion processing can weight the features in the dynamically adaptive input sequence according to the importance of the internal state variables, thereby highlighting the features of key time steps. Finally, a nonlinear mapping transformation is performed on the enhanced feature representation based on key time steps to generate scene parameter prediction vectors. This nonlinear mapping transformation can be achieved by constructing a nonlinear model that outputs corresponding scene parameter prediction values ​​based on the input feature representation. Numerical range constraints are applied to the scene parameter prediction vectors to obtain real-time updated scene generation parameters. These constraints ensure that the generated scene parameters remain within a reasonable range, thus guaranteeing the stability and plausibility of the virtual reality scene. Through these technical processes, dynamic adjustments to the virtual reality scene can be achieved, allowing the scene's presentation to adapt to changes in user interaction behavior in real time.

[0080] In summary, the deep learning-based virtual reality scene dynamic adjustment method in the above embodiments of the present invention, by extracting user preference information and environmental change information from interaction data, can overcome the shortcomings of existing technologies that do not consider user preferences and environmental dynamics, providing a precise basis for visual element optimization. Secondly, weight allocation processing can establish a correlation between lighting effects and spatial structure. Then, incoordination region detection can proactively discover mismatches between the two, avoiding the neglect of incoordination in existing technologies. Furthermore, generating alternative lighting effect variants and evaluating realism solves the problem of existing technologies lacking effective optimization methods. Moreover, by integrating user preference information and spatial structure to obtain optimization schemes and time-series modeling, scene parameters are updated in real time, overcoming the shortcomings of insufficient dynamic adjustment in existing technologies. This enables dynamic and precise optimization of virtual reality scene visual elements, effectively solving the problem of poor scene presentation in existing technologies and meeting users' needs for immersion and personalized experiences.

[0081] Example 2

[0082] Please see Figure 2 The figure shows a method for dynamically adjusting a virtual reality scene based on deep learning according to a second embodiment of the present invention. The method includes steps S201 to S207:

[0083] S201. Obtain user interaction data in the virtual reality scene, and combine it with feature extraction algorithm to extract user preference information and environmental change information from the interaction data to obtain the initial fusion features of visual elements.

[0084] S202. Based on the initial fusion features, weight allocation processing is performed on the lighting effects and spatial structure to obtain the element correlation matrix; the element correlation matrix is ​​then used to detect regions where the lighting effects and spatial structure do not match to obtain the incompatible regions.

[0085] S203. Determine whether the proportion of the incoherent area exceeds the preset threshold.

[0086] If the proportion of the uncoordinated area exceeds the preset threshold, then proceed to step S204;

[0087] When the proportion of incongruent areas does not exceed a preset threshold, it indicates that the matching degree between the lighting effects and spatial structure in the current virtual reality scene is within an acceptable range and will not significantly affect the user's immersion and visual experience. Therefore, there is no need to initiate a complex optimization process to regenerate lighting effect variants. At this time, the current core lighting parameters (such as lighting direction, intensity, color, etc.) will remain unchanged to maintain the stability and consistency of the scene's visual presentation and avoid unnecessary parameter adjustments that may interfere with the user's experience. At the same time, a dynamic monitoring mechanism will be continuously activated to track subtle changes in user interaction behavior (such as user movement in the scene, manipulation of scene objects, etc.) and environmental information in real time, and continuously update the association state between lighting effects and spatial structure. Once it is detected that the proportion of incongruent areas exceeds the preset threshold due to dynamic changes, the aforementioned optimization process of calling the pre-trained CGAN to generate alternative lighting variants will be immediately triggered.

[0088] S204. Generate alternative lighting effect variants for incongruous regions using a pre-trained conditional generative adversarial network.

[0089] In this embodiment, step S204 specifically includes steps S2041a to S2045a:

[0090] S2041a. Obtain the spatial coordinates and lighting parameters of the incongruent area. Construct an initial lighting feature vector based on the spatial coordinates and lighting parameters of the incongruent area. The initial lighting feature vector includes the lighting direction, intensity, and color attributes.

[0091] In virtual reality scenes, the spatial coordinates and lighting parameters of incongruous areas are crucial information for constructing the initial lighting feature vector. Spatial coordinates determine the specific location of the incongruous area within the scene, while lighting parameters include illumination direction, intensity, and color attributes. These parameters can be obtained through the rendering engine of the virtual reality scene. For example, illumination direction can be calculated from the position and direction of light sources in the scene, illumination intensity can be determined from the brightness parameters of the light sources, and color attributes can be obtained from the color values ​​of the light sources. Combining these parameters into a vector yields the initial lighting feature vector. This vector provides the foundational data for subsequent lighting effect reconstruction, ensuring that the generated lighting effects match the original lighting features of the incongruous area.

[0092] S2042a. Input the initial light and shadow feature vector into the pre-trained conditional generative adversarial network to reconstruct the light and shadow effect, thereby obtaining the initial light and shadow variant.

[0093] The initial lighting and shadow feature vectors are input into a pre-trained Conditional Generative Adversarial Network (CGN) to reconstruct the lighting and shadow effects. A CGN is a deep learning model consisting of a generator and a discriminator. The generator is responsible for generating new lighting and shadow effects based on the input initial feature vectors, while the discriminator is responsible for evaluating the realism of the generated effects. During training, the generator and discriminator compete against each other. The generator continuously learns how to generate more realistic lighting and shadow effects, while the discriminator continuously learns how to distinguish between generated and real lighting and shadow effects. In this way, the generator can generate initial variations of lighting and shadow, providing a variety of possible lighting and shadow effects for subsequent processing.

[0094] S2043a. Perform edge feathering on the initial light and shadow variant to obtain the processed smooth light and shadow variant.

[0095] Edge feathering is applied to the initial light and shadow variant to obtain a smoothed variant. Edge feathering is an image processing technique used to reduce the sharpness of image edges, making them smoother and more natural. This process can be achieved by applying Gaussian blur or other smoothing filters. Specifically, a Gaussian blur algorithm can be applied to the edge regions of the initial light and shadow variant, and the smoothness of the edges can be controlled by adjusting the blur radius. Edge feathering effectively reduces abrupt changes in the light and shadow variant at the edges, making the lighting effects more natural and avoiding visual disharmony caused by overly sharp edges.

[0096] S2044a. Combine the three-dimensional normal information of the spatial structure to eliminate the conflict between light and shadow and geometric surface in the smooth light and shadow variant, so as to obtain a geometrically adapted light and shadow variant.

[0097] Based on the 3D normal information of the spatial structure, this paper addresses the conflict between lighting and geometric surfaces in smoothed lighting variants. 3D normal information reflects the orientation of geometric surfaces in a scene; analyzing this information determines the relationship between lighting variants and the geometric surfaces. If the direction of the lighting variant is inconsistent with the normal direction of the geometric surface, it may lead to visual conflicts. For example, if the lighting direction is perpendicular to the normal direction of the object's surface, the lighting may look unnatural. By adjusting the direction and intensity of the lighting variant to match the normal information of the geometric surface, a geometrically adapted lighting variant can be obtained. This process can be achieved by calculating the angle between the lighting direction and the normal direction and adjusting the lighting parameters according to the angle, thus ensuring the harmony between the lighting effects and the geometric surface.

[0098] S2045a: Perform real-time rendering tests on geometrically adapted lighting variants and generate alternative lighting effect variants.

[0099] Real-time rendering tests are performed on geometrically adapted lighting variants to generate alternative lighting effect variants. Real-time rendering tests involve actually rendering the lighting variants using the rendering engine of the virtual reality scene to evaluate their visual effects within the scene. This process simulates the performance of the lighting variants under different lighting conditions and viewpoints, ensuring that the generated lighting effects have good visual appeal in a real-world scene. Real-time rendering tests generate multiple alternative lighting effect variants, providing various options for subsequent realism evaluation. This process involves rendering techniques in computer graphics; real-time rendering allows for rapid evaluation of the feasibility and aesthetics of lighting effects.

[0100] The method for generating the pre-trained conditional generative adversarial network specifically includes steps S2041b to S2046b:

[0101] S2041b: Data annotation is performed based on light and shadow sample data in historical virtual reality scenes to obtain a training dataset, which includes lighting direction, intensity, and color attributes.

[0102] For example, in generating lighting and shadow effects in virtual reality scenes, the quality of the training dataset is crucial to model performance. First, it's necessary to annotate the lighting and shadow sample data from historical virtual reality scenes. Specifically, the annotation process involves accurately labeling the lighting direction, intensity, and color attributes in the sample data. These attributes are key features of the lighting and shadow effects; the lighting direction determines the angle of light and shadow projection, intensity affects the brightness of the light and shadow, and color attributes determine the hue and saturation of the light and shadow. Annotation work is typically done by professional data annotators who, according to predefined standards and rules, use annotation tools to label the corresponding attributes on the sample data. After annotation, the resulting training dataset contains rich lighting and shadow feature information.

[0103] S2042b: Input the training dataset into the generative adversarial network to train the generator and discriminator adversarially, and obtain an initial generative adversarial network with the ability to generate light and shadow.

[0104] Optionally, a Generative Adversarial Network (GAN) is a powerful generative model consisting of a generator and a discriminator. In this step, the labeled training dataset is input into the GAN to begin adversarial training. The generator aims to generate realistic lighting effects based on the input noise vector, while the discriminator aims to distinguish the generated lighting effects from real lighting samples. During training, the generator continuously attempts to generate samples that are closer to realistic lighting effects, while the discriminator continuously learns how to more accurately identify the differences between real and generated samples. Through this adversarial process, the generator gradually learns how to generate high-quality lighting effects, while the discriminator continuously improves its discrimination ability. After multiple rounds of adversarial training, the performance of both the generator and the discriminator gradually improves, ultimately resulting in an initial GAN ​​capable of generating lighting effects. This process involves neural network training techniques in deep learning, using optimization algorithms such as stochastic gradient descent to adjust network parameters, enabling the network to better fit the training data.

[0105] S2043b: Calculate the pixel-level difference loss between the generator's output in the initial generative adversarial network and the real light and shadow samples to obtain the loss function value.

[0106] The formula for calculating the loss function is as follows:

[0107] ;

[0108] In the formula, Let G( be the loss function value). x () represents the lighting and shadow effects output by the generator. y As a real light and shadow sample, N For the total number of pixels, |G( x ) i - y i |No. i L1 difference of 1 pixel, SSIM(G( x ),y) is the structural similarity index between the generated result and the real sample. f (·) represents a pre-trained visual feature extraction network. a、b、c These are the weighting coefficients.

[0109] Specifically, to evaluate the difference between the lighting effects generated by the generator and real lighting samples, a pixel-level difference loss needs to be calculated. Specifically, the loss function value is calculated by comparing the pixel-level differences between the generator's output lighting effects and real lighting samples. The loss function comprehensively considers the pixel-level L1 difference, the structural similarity index between the generated result and the real sample, and the output of the pre-trained visual feature extraction network. The L1 difference measures the direct difference in pixel values ​​between the generated lighting effects and real lighting samples; the structural similarity index (SSIM) is used to evaluate the structural similarity between the generated lighting effects and real lighting samples; and the pre-trained visual feature extraction network (such as a convolutional neural network) is used to extract high-level visual features of the lighting effects, further evaluating the realism of the generated effect.

[0110] S2044b: Based on the loss function value, perform backpropagation to update the parameters of the generator and discriminator networks, and obtain the updated generative adversarial network.

[0111] For example, backpropagation is a commonly used optimization algorithm in deep learning. It adjusts network parameters to minimize the loss function value by calculating the gradient of the loss function with respect to the network parameters. Specifically, it first calculates the gradient of the loss function value with respect to the generator and discriminator network parameters, and then updates the network parameters based on the corresponding gradient values. This process involves two main steps: gradient calculation and parameter updating. Gradient calculation can be achieved using automatic differentiation techniques. Automatic differentiation tools, such as TensorFlow or PyTorch, can automatically calculate the gradient of the loss function value with respect to the network parameters. Parameter updating is achieved using optimization algorithms, such as the Adam (Adaptive Moment Estimation) optimization algorithm or the RMSprop (Root Mean Square Propagation) optimization algorithm. These optimization algorithms adjust the network parameters based on the gradient values, allowing the network to gradually converge to the optimal solution during training. Through backpropagation updates, the generator can generate samples that more closely resemble realistic lighting effects, while the discriminator can more accurately identify the differences between real and generated samples, thereby improving the overall performance of the generative adversarial network.

[0112] S2045b: Determine whether the loss function value of the generative adversarial network after parameter update has converged to the convergence threshold.

[0113] If not, update the initial generative adversarial network according to the updated generative adversarial network and return to step S2043b until the loss function value converges to the convergence threshold, and then execute step S2046b.

[0114] S2046b, Obtain the pre-trained conditional generative adversarial network.

[0115] To further improve the performance of the generative adversarial network (GAN), steps S2043b to S2044b are repeated until the loss function converges to a preset convergence threshold. This process is called iterative training. Through multiple iterations, the performance of the generator and discriminator gradually improves, and the generated lighting effects gradually approach realistic lighting samples. In each iteration, the generator generates new lighting effects based on the current network parameters, the discriminator evaluates the generated effects, calculates the loss function value, and then updates the network parameters through backpropagation. This process is repeated until the loss function value no longer decreases significantly, reaching the convergence threshold. The convergence threshold is a predefined value used to determine whether the network has converged to a stable solution. Through multiple iterative training steps, a pre-trained conditional generative adversarial network is finally obtained, which has the ability to generate high-quality lighting effects and can generate realistic lighting effects for virtual reality scenes.

[0116] A high-quality training dataset is obtained through data annotation. Generative adversarial networks (GANs) are then used for adversarial training. Pixel-level difference loss is calculated, and network parameters are updated via backpropagation. Finally, a pre-trained conditional GAN ​​is obtained through multiple iterations of training. This method not only generates high-quality lighting effects but also ensures that the generated lighting effects are highly consistent with real-world lighting samples, significantly improving the realism and visual quality of lighting effects in virtual reality scenes.

[0117] S205. Perform a realism evaluation on the candidate lighting effect variants to obtain a realism evaluation value.

[0118] Specifically, step S205 includes steps S2051 to S2055:

[0119] S2051. Based on the alternative lighting effect variants, calculate the average brightness of all pixels in the variants to obtain the overall average brightness.

[0120] Optionally, to evaluate the overall brightness characteristics of the candidate lighting effect variants, it is first necessary to calculate the average brightness of all pixels in the variant. This process involves iterating through each pixel of the candidate lighting effect variant image and extracting its brightness value. The brightness value can be obtained by converting the RGB components of the pixel to grayscale values; a common conversion formula is:

[0121] Y = 0.299R + 0.587G + 0.114B;

[0122] Where Y is the luminance value, and R, G, and B represent the red, green, and blue components of the pixel, respectively. The overall average luminance is obtained by averaging the luminance values ​​of all pixels. This average reflects the overall brightness and darkness of the candidate lighting effect variants and is one of the important indicators for evaluating the realism of lighting effects.

[0123] S2052. Based on the candidate light and shadow effect variants, identify the highlight areas in the variants and calculate the area ratio of the highlight areas to obtain the highlight area ratio.

[0124] In evaluating the realism of alternative lighting effect variations, the identification and analysis of highlight regions is a crucial step. Highlight regions typically refer to areas in an image with brightness values ​​exceeding a certain threshold; these areas often have a significant visual impact on realistic lighting effects. Identifying highlight regions can be achieved by setting a brightness threshold and comparing the brightness values ​​of all pixels to this threshold; pixels with brightness values ​​exceeding the threshold are considered part of the highlight region. The area ratio of highlight regions is calculated by counting the number of pixels in the highlight region and dividing it by the total number of pixels. This ratio reflects the distribution of highlight regions in the overall image and helps assess the naturalness of the lighting effect.

[0125] S2053. Based on the candidate light and shadow effect variants, perform pixel gradient analysis on the edges of the shadow areas in the variants to obtain the shadow edge sharpness value.

[0126] Specifically, the edge sharpness of shadow areas is another important indicator for evaluating the realism of lighting effects. The sharpness value of shadow edges can be obtained by performing pixel gradient analysis on the edges of shadow areas in candidate lighting effect variations. Pixel gradient analysis is typically achieved by calculating the brightness difference between adjacent pixels in an image; commonly used gradient calculation methods include the Sobel operator or the Canny edge detection algorithm. These methods can effectively detect edge information in an image, thereby quantifying the sharpness of shadow edges. A higher sharpness value indicates a clearer shadow edge, which is a common characteristic in realistic lighting effects.

[0127] S2054. Based on the candidate light and shadow effect variants, calculate the chromaticity difference between adjacent pixels in the variants, and sum the difference values ​​to obtain the total chromaticity jump.

[0128] For example, the calculation of the sum of chromaticity jumps is used to evaluate the plausibility of color changes in alternative lighting effect variants. A chromaticity jump refers to a sudden change in color between adjacent pixels; such a change is usually smoother in realistic lighting effects. Calculating the sum of chromaticity jumps first requires calculating the chromaticity difference between adjacent pixels, which can be achieved by calculating the Euclidean distance between the RGB components, i.e.:

[0129] ;

[0130] Here, △C represents the total chromaticity jump, and (R1, G1, B1) and (R2, G2, B2) are the RGB components of two adjacent pixels, respectively. By summing the chromaticity differences of all adjacent pixel pairs, the total chromaticity jump is obtained. This metric helps to evaluate the naturalness of color transitions in lighting effects.

[0131] S2055. The overall brightness average, highlight area ratio, shadow edge sharpness value, and chromatic abrupt change are weighted and summed to obtain the realism evaluation value.

[0132] Specifically, the realism assessment value is calculated by weighted summation of the above indicators. This process involves weighting the overall average brightness, highlight area proportion, shadow edge sharpness value, and total chromatic abrupt changes. The weighting can be determined based on the degree of influence of each indicator on realism; for example, shadow edge sharpness may have a greater impact on realism and therefore can be assigned a higher weight. The weighted summation formula can be expressed as:

[0133] S=w1﹒ I+w2﹒ H+w3﹒ S+w4﹒ C;

[0134] Where S is the realism assessment value, I is the overall average brightness, H is the proportion of the highlight area, S is the shadow edge sharpness value, C is the color gradation, and w1, w2, w3, and w4 are the weighting coefficients of each indicator. Using the above method, a comprehensive assessment value reflecting the realism of the candidate lighting and shadow effect variations can be obtained.

[0135] By conducting a multi-dimensional analysis of candidate lighting effect variants, including calculating the overall average brightness, highlight area proportion, shadow edge sharpness value, and total chromatic abrupt changes, and then weighting and summing these indicators, a comprehensive and quantitative realism assessment value was obtained. This assessment value accurately reflects the degree of similarity between the candidate lighting effect variants and real lighting effects.

[0136] S206. Based on the realism assessment value, user preference information is integrated with the spatial structure to obtain an optimized element integration scheme.

[0137] In this embodiment, step S206 specifically includes steps S2061 to S2067:

[0138] S2061. Based on the realism assessment value, the user preference information is weighted to obtain the preference weight matrix.

[0139] For example, in the optimization of virtual reality scenes, user preference information is one of the key factors affecting user experience. Weighting user preference information based on realism assessment values ​​ensures that the generated fusion scheme better meets the user's personalized needs. Specifically, the realism assessment value reflects the similarity between candidate lighting effect variants and real lighting effects, and this value can be used as the basis for weight allocation. For instance, if a user preference performs well in the realism assessment, it can be given a higher weight. Weight allocation can be achieved by constructing a weight allocation model that dynamically adjusts the weights of each feature in the user preference information based on the realism assessment value. The final preference weight matrix combines each feature of the user preference information with its corresponding weight.

[0140] S2062. For the spatial structure, extract the structural feature vector to obtain the spatial structure feature matrix.

[0141] Optionally, spatial structure is a crucial component of virtual reality scenes, and its feature extraction is essential for generating high-quality fusion solutions. Extracting structural feature vectors of spatial structure is achieved by analyzing the geometry, position, and layout of objects in the scene. Specifically, geometric analysis methods from computer graphics, such as bounding box detection and polygon mesh analysis, can be used to extract the shape features of objects. Simultaneously, spatial analysis algorithms, such as spatial partitioning and neighborhood analysis, can be used to obtain the relative positions and layout information between objects. This feature information is organized into structural feature vectors, further forming a spatial structure feature matrix. This matrix not only contains the geometric features of each object in the scene but also reflects the spatial relationships between objects.

[0142] S2063. Based on the preference weight matrix and the spatial structure feature matrix, a weighted fusion is performed to generate an initial element fusion scheme.

[0143] Specifically, the weights in the preference weight matrix can be combined with the eigenvectors in the spatial structure feature matrix through operations such as matrix multiplication or weighted summation. For example, for each eigenvector in the spatial structure feature matrix, a weighted sum is applied according to the corresponding weight in the preference weight matrix to obtain the fused eigenvector. The fused eigenvectors together constitute the initial element fusion scheme, which integrates information from user preferences and spatial structure.

[0144] S2064. Perform multi-objective optimization on the initial element fusion scheme to obtain a set of candidate fusion schemes.

[0145] Specifically, multi-objective optimization of the initial element fusion scheme aims to find a balance among multiple optimization objectives, resulting in a set of candidate fusion schemes. Multi-objective optimization typically involves multiple objective functions, such as user satisfaction, scene realism, and computational efficiency. These objective functions may conflict, thus requiring optimization algorithms to balance them. For example, evolutionary algorithms such as genetic algorithms and particle swarm optimization, or gradient-based optimization methods such as multi-objective gradient descent, can be used. These algorithms can search among multiple objectives to find a set of candidate fusion schemes that satisfy all objectives. Through multi-objective optimization, a set of candidate fusion schemes that achieves a balance among different objectives can be obtained.

[0146] S2065. Perform dynamic simulation verification on the candidate fusion scheme set to obtain the simulation verification result dataset.

[0147] The dynamic simulation verification of the candidate fusion schemes aims to evaluate their performance in real-world virtual reality scenarios. Dynamic simulation verification constructs a virtual simulation environment to simulate user interactions and scene changes within the virtual reality environment. Specifically, a physics engine can be used to simulate object movement and interactions, combined with real-time rendering of lighting and shadow effects, to evaluate the visual effects and performance of candidate fusion schemes under different conditions. Through dynamic simulation verification, performance data for each candidate fusion scheme can be collected, such as the coherence of lighting and shadow effects and the naturalness of object movement. This performance data constitutes the simulation verification result dataset.

[0148] In this embodiment, step S2065 specifically includes steps S2065a to S2065e:

[0149] S2065a. Construct a simulation scenario model based on the candidate fusion scheme set.

[0150] Optionally, the simulation scene model includes geometric structure, material properties, and lighting parameters. Constructing a simulation scene model is fundamental to dynamic simulation verification; it requires comprehensive consideration of geometric structure, material properties, and lighting parameters. Geometric structure defines the shape and layout of objects in the scene, material properties determine the visual characteristics of object surfaces, such as reflectivity, refractive index, and texture, while lighting parameters control the direction, intensity, and color of illumination. These elements together constitute a complete virtual scene model capable of simulating physical and visual phenomena in the real world. When constructing the model, modeling techniques from computer graphics, such as 3D modeling software or programming interfaces, are typically used to integrate the corresponding attributes and parameters into a unified scene description.

[0151] S2065b: Perform dynamic simulation of the simulation scene model using a physics engine to obtain object motion trajectory data.

[0152] For example, dynamics simulation is the process of calculating the motion of objects in a simulated scene model using a physics engine. A physics engine is a software component that simulates the motion and interactions of objects based on Newtonian mechanics principles. During the simulation, the physics engine considers the object's mass, inertia, external forces (such as gravity and friction), and collision detection and response. Through these calculations, the position and velocity of the object at different points in time can be obtained, i.e., the object's trajectory data.

[0153] S2065c: Real-time tracking and calculation of light and shadow parameters based on object motion trajectory data to generate dynamic light and shadow change sequences.

[0154] Real-time tracking and calculation of lighting parameters involves dynamically adjusting lighting effects based on object motion trajectory data. As an object moves within the scene, its relative position and angle with the light source change, leading to alterations in lighting effects. By tracking the object's position and orientation in real time, combined with the light source parameters, the lighting effects at each moment can be calculated. This process typically involves calculating lighting models, such as the Phong lighting model or more complex global illumination models. These models can calculate visual effects such as reflected light, refracted light, and shadows based on the object's material properties and light source parameters, generating a dynamic sequence of lighting changes that reflects the evolution of lighting effects over time.

[0155] S2065d: Frame-to-frame difference analysis was performed on the dynamic light and shadow change sequence to obtain the light and shadow coherence index.

[0156] Specifically, inter-frame difference analysis is a method for evaluating the coherence of dynamic lighting and shadow changes. By comparing lighting and shadow changes between consecutive frames, the smoothness of the lighting and shadow effects can be quantified. This analysis typically involves image processing techniques, such as calculating pixel differences between adjacent frames or using more advanced feature matching algorithms to detect the continuity of lighting and shadow changes. The lighting and shadow coherence metric can be a comprehensive score that reflects the stability and naturalness of lighting and shadow effects over time. This metric is crucial for evaluating the realism and visual comfort of lighting and shadow effects in virtual reality scenes.

[0157] S2065e: Based on the light and shadow coherence index, the performance of each scheme in the candidate fusion scheme set is scored, and a simulation verification result dataset is generated.

[0158] Optionally, performance scoring of candidate fusion schemes based on the light and shadow coherence index is a process of evaluating the performance of each scheme in a dynamic scene. The performance scoring model can be obtained according to predefined rules or trained through machine learning algorithms, and it converts the light and shadow coherence index into a quantitative score. This score reflects the visual quality and user experience of each candidate scheme under dynamic light and shadow changes. By collecting the corresponding scores, a simulation verification result dataset is generated.

[0159] By constructing a simulation scene model that includes geometric structure, material properties, and lighting parameters, and using a physics engine to perform dynamic simulation, the lighting parameters are tracked and calculated in real time to generate a dynamic lighting change sequence. Inter-frame difference analysis is performed to obtain a lighting coherence index. Finally, the candidate fusion schemes are evaluated based on the above indexes. This method can comprehensively evaluate the performance of candidate fusion schemes in dynamic virtual reality scenes, ensuring the dynamic coherence and visual quality of lighting effects.

[0160] S2066. Calculate the feasibility score based on the simulation verification result dataset to obtain the feasibility score of the scheme.

[0161] Optionally, calculating a feasibility score based on the simulation verification result dataset is for the quantitative evaluation of candidate fusion schemes. The feasibility score calculation can be achieved by constructing a scoring model that scores each candidate fusion scheme based on various performance indicators in the simulation verification result dataset. The scoring model can be a machine learning-based model, such as a support vector machine or neural network, which can learn the mapping relationship between various performance indicators and the feasibility score based on predefined rules or training data. Through feasibility score calculation, a feasibility score can be obtained for each candidate fusion scheme.

[0162] S2067. Sort the feasibility scores of the schemes from high to low, and select the candidate fusion scheme with the highest feasibility score as the optimized element fusion scheme.

[0163] For example, ranking the feasibility scores of the solutions from highest to lowest and selecting the candidate fusion solution with the highest score as the optimized element fusion solution is a key step in determining the optimal solution. The ranking process can be implemented using simple sorting algorithms, such as quicksort or mergesort. Selecting the solution with the highest score means that it performs best among all candidate solutions and better meets the requirements of user preferences and spatial structure.

[0164] This method weights user preference information based on realism assessment values, extracts spatial structure features, performs weighted fusion to generate an initial fusion scheme, conducts multi-objective optimization to obtain a candidate scheme set, evaluates scheme performance through dynamic simulation, calculates feasibility scores, and selects the optimal scheme. This completes the process from comprehensively considering user preferences and spatial structure information to generating, verifying, and optimizing candidate schemes. This approach not only ensures that the generated fusion scheme meets users' personalized needs but also ensures the scheme's performance in real-world virtual reality scenarios through multi-objective optimization and dynamic simulation verification, thereby significantly improving the user experience and visual quality of virtual reality scenes.

[0165] S207. Based on the optimized element fusion scheme, perform temporal modeling processing on the dynamically adaptive input sequence to obtain real-time updated scene generation parameters, so as to dynamically adjust the presentation effect of the virtual reality scene according to the real-time updated scene generation parameters.

[0166] As a specific example, step S207 specifically includes steps S2071 to S2076:

[0167] S2071. Extract the continuous change features of user interaction behavior in the time dimension based on the optimized element fusion scheme, and generate a dynamically adaptive input sequence.

[0168] For example, extracting the continuous change characteristics of user interaction behavior over time is key to achieving dynamic adaptive adjustment. User interaction behavior includes user movement, operations, and interaction events, all of which are continuous in time. By analyzing the time-series data of these behaviors, features reflecting user interaction patterns can be extracted. For example, the rate of change of user movement speed and fluctuations in operation frequency can be calculated. These features can capture the dynamic changes in user interaction behavior. The dynamic adaptation input sequence is a time series containing user interaction behavior features, reflecting the user's real-time interaction state in a virtual reality scene.

[0169] S2072. Divide the time window based on the dynamically adaptive input sequence to obtain time segment data of fixed length.

[0170] Specifically, time window partitioning based on dynamically adaptive input sequences aims to divide continuous time-series data into fixed-length time-series segments. Time window partitioning involves sliding a fixed-length window across the time series, dividing the data into multiple segments. Each segment contains user interaction behavior characteristics over a given period, and these segments can be analyzed and processed independently. The length of the time window can be adjusted according to the needs of the application; for example, shorter time windows can capture rapidly changing interaction behaviors, while longer time windows can smooth out short-term fluctuations. Through time window partitioning, dynamically adaptive input sequences can be transformed into a series of fixed-length time-series segments.

[0171] S2073. Perform state dependency calculation on the time series data to obtain the internal state variables at the current time.

[0172] State dependency computation is a state estimation method based on time-series data. It estimates the internal state at the current moment by analyzing features in time-series data segments. Internal state variables can include user interaction intentions, dynamic changes in the scene, etc., reflecting the user's real-time state in the virtual reality scene. State dependency computation can be implemented by constructing state transition models, such as using Markov models or Hidden Markov Models to describe the transition relationships between states. Using these models, the internal state variables at the current moment can be calculated based on the characteristics of the current moment and the historical states.

[0173] S2074. By performing feature weighted fusion processing on the internal state variables at the current moment, the feature representation of the key time step is obtained.

[0174] Specifically, feature weighted fusion is a feature enhancement method based on state information. It weights features in time-series data according to the internal state variables at the current moment. Internal state variables reflect the importance of features in the current state, and weighting enhances the representation of key features. For example, if the current state variables indicate that the user is moving quickly, features related to movement speed can be assigned higher weights. Feature weighted fusion can be implemented by constructing a weighted model, such as using a linear or non-linear weighted model to calculate the weighted feature representation. Through feature weighted fusion, enhanced feature representations at key time steps can be obtained.

[0175] S2075. Perform nonlinear mapping transformation based on the feature representation enhanced by key time steps to generate scene parameter prediction vectors.

[0176] For example, a nonlinear mapping transformation is a nonlinear function that maps input features to output features, which can be implemented using neural networks or other nonlinear models. In virtual reality scenes, a nonlinear mapping transformation can map enhanced feature representations to predicted values ​​of scene parameters. For instance, a multilayer perceptron (MLP) or a convolutional neural network (CNN) can be used to implement a nonlinear mapping transformation. These models can learn the nonlinear relationship between feature representations and scene parameters through training, thereby generating accurate scene parameter prediction vectors.

[0177] S2076. Perform numerical range constraint processing on the scene parameter prediction vector to obtain the scene generation parameters that are updated in real time.

[0178] Optionally, numerical range constraint processing is a method to ensure that predicted values ​​are within a reasonable range. It constrains the predicted values ​​of each scene parameter in the scene parameter prediction vector by setting upper and lower limits. In virtual reality scenes, the range of scene parameter values ​​is usually determined by the physics engine and rendering engine. For example, the range of light intensity might be [0,1], and the range of object position might be the scene boundary. Numerical range constraint processing ensures that the generated scene parameters are within a reasonable range, avoiding unreasonable parameter values. Numerical range constraint processing can be implemented through simple conditional judgments or more complex constraint optimization algorithms. Through numerical range constraint processing, real-time updated scene generation parameters can be obtained, which can be directly used for dynamic adjustment of the virtual reality scene.

[0179] By extracting continuous change features of user interaction behavior to generate a dynamically adaptive input sequence, dividing the time window, calculating state dependencies to obtain internal state variables, performing feature weighted fusion processing to obtain enhanced feature representations, performing nonlinear mapping transformation to generate scene parameter prediction vectors, and performing numerical range constraint processing to obtain real-time updated scene generation parameters, this method realizes the dynamic adaptive adjustment of virtual reality scenes and ensures that the generated scene parameters are within a reasonable range, thereby significantly improving the dynamic adaptability of virtual reality scenes and user experience.

[0180] In summary, the deep learning-based virtual reality scene dynamic adjustment method in the above embodiments of the present invention, by extracting user preference information and environmental change information from interaction data, can overcome the shortcomings of existing technologies that do not consider user preferences and environmental dynamics, providing a precise basis for visual element optimization. Secondly, weight allocation processing can establish a correlation between lighting effects and spatial structure. Then, incoordination region detection can proactively discover mismatches between the two, avoiding the neglect of incoordination in existing technologies. Furthermore, generating alternative lighting effect variants and evaluating realism solves the problem of existing technologies lacking effective optimization methods. Moreover, by integrating user preference information and spatial structure to obtain optimization schemes and time-series modeling, scene parameters are updated in real time, overcoming the shortcomings of insufficient dynamic adjustment in existing technologies. This enables dynamic and precise optimization of virtual reality scene visual elements, effectively solving the problem of poor scene presentation in existing technologies and meeting users' needs for immersion and personalized experiences.

[0181] Example 3

[0182] The third embodiment of the present invention provides a deep learning-based virtual reality scene dynamic adjustment system, comprising:

[0183] The acquisition module is used to acquire user interaction data in a virtual reality scene, and combine it with a feature extraction algorithm to extract user preference information and environmental change information from the interaction data to obtain the initial fusion features of visual elements.

[0184] The detection module is used to perform weight allocation processing on the lighting effects and spatial structure according to the initial fusion features to obtain an element correlation matrix; and to detect regions where the lighting effects and spatial structure do not match the element correlation matrix to obtain incompatible regions.

[0185] The judgment module is used to determine whether the proportion of incompatible areas exceeds a preset threshold.

[0186] The first execution module is used to generate alternative lighting effect variants for the incongruent regions through a pre-trained conditional generative adversarial network if the proportion of incongruent regions exceeds a preset threshold, and to evaluate the realism of the alternative lighting effect variants to obtain a realism evaluation value.

[0187] The dynamic adjustment module is used to fuse user preference information with spatial structure based on the realism evaluation value to obtain an optimized element fusion scheme. Based on the optimized element fusion scheme, the module performs temporal modeling processing on the dynamically adaptive input sequence to obtain real-time updated scene generation parameters, so as to dynamically adjust the presentation effect of the virtual reality scene according to the real-time updated scene generation parameters.

[0188] In summary, the deep learning-based virtual reality scene dynamic adjustment system in the above embodiments of the present invention, by extracting user preference information and environmental change information from interaction data, can overcome the shortcomings of existing technologies that do not consider user preferences and environmental dynamics, providing a precise basis for visual element optimization. Secondly, weight allocation processing can establish a correlation between lighting effects and spatial structure. Then, incoordination region detection can proactively discover mismatches between the two, avoiding the neglect of incoordination in existing technologies. Furthermore, generating alternative lighting effect variants and evaluating realism solves the problem of existing technologies lacking effective optimization methods. Moreover, by integrating user preference information and spatial structure to obtain optimization schemes and time-series modeling, scene parameters are updated in real time, overcoming the shortcomings of insufficient dynamic adjustment in existing technologies. This enables dynamic and precise optimization of virtual reality scene visual elements, effectively solving the problem of poor scene presentation in existing technologies and meeting users' needs for immersion and personalized experiences.

[0189] Furthermore, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the methods described above.

[0190] Furthermore, embodiments of the present invention also propose a data processing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the methods described above.

[0191] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0192] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0193] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0194] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0195] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A deep learning based dynamic adjustment method for virtual reality scenes, characterized in that, include: The system acquires user interaction data in a virtual reality scene and combines it with a feature extraction algorithm to extract user preference information and environmental change information from the interaction data to obtain the initial fusion features of visual elements. The initial fusion features are used to perform weighted processing on the lighting effects and spatial structure to obtain an element correlation matrix; the element correlation matrix is ​​then used to detect regions where the lighting effects and spatial structure do not match to obtain incompatible regions; Determine whether the proportion of incompatible areas exceeds a preset threshold; If so, then a pre-trained conditional generative adversarial network is used to generate alternative lighting effect variants for the incongruent region, and the realism of the alternative lighting effect variants is evaluated to obtain a realism evaluation value. Based on the realism assessment value, user preference information and spatial structure are fused to obtain an optimized element fusion scheme. Based on the optimized element fusion scheme, the dynamic adaptive input sequence is subjected to temporal modeling processing to obtain real-time updated scene generation parameters, so as to dynamically adjust the presentation effect of the virtual reality scene according to the real-time updated scene generation parameters. The step of generating alternative lighting effect variants for incongruous regions using a pre-trained conditional generative adversarial network includes: The spatial coordinates and lighting parameters of the incongruent area are obtained, and an initial lighting feature vector is constructed based on the spatial coordinates and lighting parameters of the incongruent area. The initial lighting feature vector includes the lighting direction, intensity, and color attributes. The initial light and shadow feature vector is input into a pre-trained conditional generative adversarial network to reconstruct the light and shadow effect, thereby obtaining a preliminary light and shadow variant; the preliminary light and shadow variant is then feathered at the edges to obtain a processed smooth light and shadow variant. By combining the three-dimensional normal information of the spatial structure, the conflict between light and shadow and geometric surface in the smooth light and shadow variant is eliminated to obtain a geometrically adapted light and shadow variant; the geometrically adapted light and shadow variant is subjected to real-time rendering test to generate the alternative light and shadow effect variant. 2.The deep learning based dynamic adjustment method of virtual reality scene according to claim 1, characterized in that, Methods for generating pre-trained conditional generative adversarial networks include: A training dataset is obtained by labeling light and shadow sample data from historical virtual reality scenes. The training dataset includes lighting direction, intensity, and color attributes. The training dataset is input into the generative adversarial network to train the generator and discriminator adversarially, resulting in an initial generative adversarial network with the ability to generate light and shadow. The pixel-level difference loss between the generator's output in the initial generative adversarial network and the real light and shadow sample is calculated to obtain the loss function value; Based on the loss function value, the parameters of the generator and discriminator networks are updated by backpropagation to obtain the updated generative adversarial network. Determine whether the loss function value of the generative adversarial network after parameter update has converged to the convergence threshold; If not, the initial generative adversarial network is updated according to the parameter-updated generative adversarial network, and the process returns to the step of calculating the pixel-level difference loss between the generator output in the initial generative adversarial network and the real light and shadow sample, until the loss function value converges to the convergence threshold, thus obtaining the pre-trained conditional generative adversarial network. The formula for calculating the loss function is as follows: ； In the formula, G is the loss function value, G( x ) is the light and shadow effect output by the generator, y G is the real light and shadow sample, N G is the total number of pixels, |G( x ) i - y i |The L1 difference of the i th pixel, SSIM(G( x ),y) is the structural similarity index of the generated result and the real sample, φ (·) is a pre-trained visual feature extraction network, α, β, γ G is a weight coefficient. 3.The deep learning based dynamic adjustment method of virtual reality scene according to claim 1, wherein, The steps for evaluating the realism of the candidate lighting effect variants to obtain a realism evaluation value include: Based on the candidate lighting effect variants, calculate the average brightness of all pixels in the variants to obtain the overall average brightness. Based on the candidate light and shadow effect variants, identify the highlight areas in the variants and calculate the area ratio of the highlight areas to obtain the highlight area ratio; Based on the candidate lighting effect variants, pixel gradient analysis is performed on the edges of the shadow areas in the variants to obtain the shadow edge sharpness value; Based on the candidate lighting effect variants, calculate the chromaticity difference between adjacent pixels in the variants, and sum the difference values ​​to obtain the total chromaticity jump; The overall brightness average, the highlight area ratio, the shadow edge sharpness value, and the sum of the chromaticity jumps are weighted and summed to obtain the realism evaluation value. 4.The deep learning based dynamic adjustment method of virtual reality scenes according to claim 1, characterized in that, The steps of fusing user preference information with spatial structure based on the realism assessment value to obtain an optimized element fusion scheme include: Based on the realism assessment value, the user preference information is weighted to obtain a preference weight matrix; For the spatial structure, extract the structural feature vector to obtain the spatial structure feature matrix; Based on the weighted fusion of the preference weight matrix and the spatial structure feature matrix, an initial element fusion scheme is generated. The initial element fusion scheme is subjected to multi-objective optimization to obtain a set of candidate fusion schemes; The candidate fusion scheme set is dynamically simulated and verified to obtain a simulation verification result dataset; Feasibility scores are calculated based on the simulation verification result dataset to obtain the feasibility score of the scheme; The feasibility scores of the proposed schemes are ranked from highest to lowest, and the candidate fusion scheme with the highest feasibility score is selected as the optimized element fusion scheme.

5. The method of claim 4, wherein the method further comprises: The steps for performing dynamic simulation verification on the candidate fusion scheme set to obtain the simulation verification result dataset include: A simulation scene model is constructed based on the candidate fusion scheme set. The simulation scene model includes geometric structure, material properties, and lighting parameters. The simulation scene model is then used to perform dynamic simulations through a physics engine to obtain object motion trajectory data. Based on the object's motion trajectory data, the light and shadow parameters are tracked and calculated in real time to generate a dynamic light and shadow change sequence; inter-frame difference analysis is performed on the dynamic light and shadow change sequence to obtain a light and shadow coherence index. The performance of each scheme in the candidate fusion scheme set is scored based on the light and shadow coherence index, and a simulation verification result dataset is generated. 6.The deep learning based dynamic adjustment method of virtual reality scenes according to claim 1, wherein, The steps for performing time-series modeling on the dynamically adaptable input sequence to obtain real-time updated scene generation parameters, based on the optimized element fusion scheme, include: Based on the optimized element fusion scheme, the continuous change features of user interaction behavior in the time dimension are extracted to generate a dynamically adaptive input sequence; based on the dynamically adaptive input sequence, time windows are divided to obtain time-series segment data of fixed length; State dependencies are calculated on the time-series data to obtain the internal state variables at the current time; feature weighted fusion is performed on the internal state variables at the current time to obtain the feature representation enhanced for key time steps. Based on the enhanced feature representation of the key time step, a nonlinear mapping transformation is performed to generate a scene parameter prediction vector; the scene parameter prediction vector is then subjected to numerical range constraint processing to obtain the scene generation parameters that are updated in real time.

7. A deep learning-based virtual reality scene dynamic adjustment system, characterized in that, The system includes: The acquisition module is used to acquire user interaction data in a virtual reality scene, and combine it with a feature extraction algorithm to extract user preference information and environmental change information from the interaction data to obtain the initial fusion features of visual elements. The detection module is used to perform weight allocation processing on the lighting effects and spatial structure according to the initial fusion features to obtain an element correlation matrix; and to detect regions where the lighting effects and spatial structure do not match the element correlation matrix to obtain incompatible regions. The judgment module is used to determine whether the proportion of incompatible areas exceeds a preset threshold. The first execution module is used to generate alternative lighting effect variants for the incongruent regions through a pre-trained conditional generative adversarial network if the proportion of incongruent regions exceeds a preset threshold, and to evaluate the realism of the alternative lighting effect variants to obtain a realism evaluation value. The dynamic adjustment module is used to fuse user preference information with spatial structure according to the realism evaluation value to obtain an optimized element fusion scheme. According to the optimized element fusion scheme, the dynamic adaptation input sequence is subjected to temporal modeling processing to obtain real-time updated scene generation parameters, so as to dynamically adjust the presentation effect of the virtual reality scene according to the real-time updated scene generation parameters. The step of generating alternative lighting effect variants for incongruous regions using a pre-trained conditional generative adversarial network includes: The spatial coordinates and lighting parameters of the incongruent area are obtained, and an initial lighting feature vector is constructed based on the spatial coordinates and lighting parameters of the incongruent area. The initial lighting feature vector includes the lighting direction, intensity, and color attributes. The initial light and shadow feature vector is input into a pre-trained conditional generative adversarial network to reconstruct the light and shadow effect, thereby obtaining a preliminary light and shadow variant; the preliminary light and shadow variant is then feathered at the edges to obtain a processed smooth light and shadow variant. By combining the three-dimensional normal information of the spatial structure, the conflict between light and shadow and geometric surface in the smooth light and shadow variant is eliminated to obtain a geometrically adapted light and shadow variant; the geometrically adapted light and shadow variant is subjected to real-time rendering test to generate the alternative light and shadow effect variant.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the deep learning-based virtual reality scene dynamic adjustment method as described in any one of claims 1-6.

9. A data processing apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the deep learning-based virtual reality scene dynamic adjustment method as described in any one of claims 1-6.

Images