Immersive scene generation method and device and related equipment

Abstract

The invention provides an immersive scene generation method and device and related equipment, and relates to the field of computers.According to the technical scheme, static scene data of a user at a target position is obtained, and then the static scene data is processed according to a pre-trained scene generation model to generate dynamic scene data, so that the dynamic scene data is generated; and finally, fusing the static scene data and the dynamic scene data to obtain the immersive scene model of the target position, thereby improving the efficiency and flexibility of generating the immersive scene by automatically generating the immersive scene meeting the user demand.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically to a method, apparatus, and related equipment for generating immersive scenes. Background Technology

[0002] Immersive scenes refer to environments created using virtual reality (VR), augmented reality (AR), or other multimedia technologies, allowing users to deeply engage and experience a sense of presence. These scenes typically combine multiple sensory stimuli, including visual, auditory, and tactile, aiming to completely immerse users in a specific context and enhance their sense of participation and experience. Currently, immersive scenes are generally generated manually or based on scene templates. However, these methods suffer from poor flexibility and efficiency, making it difficult to generate immersive scenes that meet user needs; thus, the resulting immersive scene generation technology suffers from poor quality. Summary of the Invention

[0003] This invention provides a sensitive content recognition method, apparatus, and related equipment, which solves the problem of poor effect in generating immersive scenes in related technologies.

[0004] To solve the above problems, the present invention is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a method for generating an immersive scene, the method comprising:

[0006] Acquire static scene data corresponding to the user when the user is at the target location, the static scene data including the location information of the target location, the user's posture information and the environmental information of the target location;

[0007] The static scene data is input into a pre-trained scene generation model for processing to generate dynamic scene data, which includes object shape information, lighting effect information and texture detail information of the dynamic scene.

[0008] The dynamic scene data and the static scene data are fused to obtain an immersive scene model corresponding to the target location.

[0009] Optionally, before inputting the static scene data into a pre-trained scene generation model for processing to generate dynamic scene data, the method further includes:

[0010] Multiple training samples are generated, each training sample including static scene data and corresponding sample labels, wherein the sample labels include dynamic scene data corresponding to the static scene data.

[0011] The initial scene generation model is trained based on the multiple training samples to obtain the scene generation model, which includes a generator and a discriminator.

[0012] Optionally, the sample scene model is sampled to obtain multiple two-dimensional image data, multiple three-dimensional point cloud data and multiple environmental parameters. Different two-dimensional image data have different perspectives in the sample scene model. Different three-dimensional point cloud data correspond to different objects in the sample scene model. Different environmental parameters correspond to different environmental indicators in the sample scene.

[0013] Image preprocessing is performed on the plurality of two-dimensional image data to obtain a plurality of target two-dimensional image data. The preprocessing is used to enhance the plurality of two-dimensional image data and unify them to the same size.

[0014] The multiple 3D point cloud data are normalized to obtain multiple target 3D point cloud data. The normalization process is used to unify the multiple 3D point cloud data into the same 3D model format.

[0015] The multiple environmental parameters are standardized to obtain multiple target environmental parameters. The standardization process is used to unify the multiple environmental parameters into the same parameter format.

[0016] Multiple sample static scene data are generated based on the multiple target two-dimensional image data and the multiple target environment parameters;

[0017] Multiple sample labels are generated based on the three-dimensional point cloud data of the multiple targets.

[0018] Optionally, training the initial scene generation model based on the multiple training samples to obtain the scene generation model includes:

[0019] Multiple noise data are generated based on the multiple training samples, and the multiple training samples and the multiple noise data correspond one-to-one.

[0020] The multiple training samples and the multiple noise data are input into the initial discriminator for discrimination to obtain multiple first discrimination results. The first discrimination results are used to indicate the sample authenticity of the corresponding training sample and the sample authenticity of the corresponding noise data.

[0021] Based on the plurality of first discrimination results and the plurality of sample labels, the first loss function is calculated to obtain the first loss value;

[0022] The initial discriminator is updated based on the backpropagation algorithm and the first loss value to obtain the discriminator;

[0023] The initial generator is updated based on the discriminator and the plurality of training samples to obtain the generator.

[0024] Optionally, updating the initial generator based on the discriminator and the plurality of training samples to obtain the generator includes:

[0025] The multiple training samples are input into the discriminator for discrimination, and multiple second discrimination results are obtained. The second discrimination results are used to indicate the sample authenticity of the corresponding training sample.

[0026] Based on the plurality of second discrimination results, the first loss function is calculated to obtain a second loss value;

[0027] The initial generator is updated based on the backpropagation algorithm and the second loss value to obtain the generator.

[0028] Optionally, fusing the dynamic scene data and the static scene data to obtain an immersive scene model corresponding to the target location includes:

[0029] Feature extraction is performed on the dynamic scene data to obtain multiple first feature points, and feature extraction is performed on the static scene data to obtain multiple second feature points;

[0030] Calculate the matching relationship between the plurality of first feature points and the plurality of second feature points to obtain a matching result, the matching result being used to indicate the position and orientation of the dynamic scene data in the static scene corresponding to the static scene data;

[0031] Based on the matching results, the texture information corresponding to the dynamic scene data is fused into the static scene corresponding to the static scene data to obtain an immersive scene model corresponding to the target location.

[0032] Secondly, embodiments of this application provide an immersive scene generation apparatus, the apparatus comprising:

[0033] The acquisition module is used to acquire static scene data corresponding to the user when the user is at the target location. The static scene data includes the location information of the target location, the user's posture information, and the environmental information of the target location.

[0034] The processing module is used to input the static scene data into a pre-trained scene generation model for processing to generate dynamic scene data, wherein the dynamic scene data includes object shape information, lighting effect information and texture detail information of the dynamic scene.

[0035] The fusion module is used to fuse the dynamic scene data and the static scene data to obtain an immersive scene model corresponding to the target location.

[0036] Thirdly, this application also provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described in the first aspect above.

[0037] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0038] Fifthly, this application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the method described in the first aspect above.

[0039] This application provides a method, apparatus, and related equipment for generating immersive scenes. The method includes: acquiring static scene data corresponding to a user at a target location, the static scene data including location information of the target location, the user's posture information, and environmental information of the target location; inputting the static scene data into a pre-trained scene generation model for processing to generate dynamic scene data, the dynamic scene data including object shape information, lighting effect information, and texture detail information of the dynamic scene; and fusing the dynamic scene data and the static scene data to obtain an immersive scene model corresponding to the target location. The technical solution of this application, by acquiring static scene data of a user at a target location, processing the static scene data according to a pre-trained scene generation model to generate dynamic scene data, and finally fusing the static and dynamic scene data to obtain an immersive scene model of the target location, improves the efficiency and flexibility of generating immersive scenes by automatically generating immersive scenes that meet user needs. Attached Figure Description

[0040] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the description of the present invention will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0041] Figure 1 A flowchart illustrating a method for generating an immersive scene according to an embodiment of this application;

[0042] Figure 2 This is a schematic diagram of the overall process provided for an embodiment of this application;

[0043] Figure 3 A schematic diagram of the structure of an immersive scene generation device provided in an embodiment of this application;

[0044] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] The terms "first," "second," etc., used in the embodiments of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices. Additionally, the use of "and / or" in this application indicates at least one of the connected objects, such as A and / or B and / or C, representing seven possibilities: including A alone, B alone, C alone, and the presence of both A and B, both B and C, both A and C, and the presence of A, B, and C.

[0047] See Figure 1 , Figure 1 This is a flowchart illustrating the method for generating an immersive scene provided in an embodiment of this application. Figure 1 As shown, the method for generating an immersive scene may include the following steps:

[0048] Step 101: Obtain static scene data corresponding to the user when the user is at the target location. The static scene data includes the location information of the target location, the user's posture information, and the environmental information of the target location.

[0049] In this embodiment, an immersive scene refers to an environment created through virtual reality, augmented reality, or other multimedia technologies, enabling users to deeply participate and experience a sense of presence. When an immersive scene is generated based on user requests, static scene data corresponding to the user's target location is obtained. This static scene data may include location information, posture information, and environmental information, namely the user's position, posture information, and real-time parameters of the environment, such as the user's three-dimensional coordinates, rotation angle, current light intensity, and temperature changes.

[0050] In this embodiment, these real-time data are encoded and converted into the same format as the environmental parameter data in the noise data during training. They are then combined with randomly generated noise vectors to obtain the combined vector, which is the static scene data corresponding to the target location.

[0051] Step 102: Input the static scene data into the pre-trained scene generation model for processing to generate dynamic scene data. The dynamic scene data includes object shape information, lighting effect information and texture detail information of the dynamic scene.

[0052] In this embodiment, the pre-trained scene generation model is a model generated based on the principle of Generative Adversarial Network (GAN), which is used to process static scene data and generate dynamic scene data corresponding to the static scene data.

[0053] Specifically, the static scene data corresponding to the target location is input into the pre-trained scene generation model. The scene generation model generates dynamic scene data that conforms to the current dynamic conditions based on the learned features and patterns and combined with real-time input data. The dynamic scene data includes object shape information, lighting effect information and texture detail information of the dynamic scene. Specifically, it includes updating the object shape, lighting effect, texture detail, etc. in the scene to realize the dynamic change of the scene.

[0054] Step 103: Fuse the dynamic scene data and the static scene data to obtain an immersive scene model corresponding to the target location.

[0055] In this embodiment, dynamic scene data and static scene data are fused to generate an immersive scene model corresponding to the target location. For example, the fusion process can be implemented using a series of techniques such as feature matching, texture mapping, and scene optimization, which are not specifically limited in this embodiment. Specifically, certain algorithms can be used to fuse the two types of data. Common techniques include computer vision, machine learning, and graphics rendering. These algorithms can achieve a natural combination of dynamic elements and static environments. After data fusion, a corresponding immersive scene model needs to be established. The immersive scene model can render dynamic changes (e.g., character movement, environment changes) and seamlessly integrate with static elements (such as buildings and terrain).

[0056] The technical solution of this application obtains static scene data of the user at the target location, processes the static scene data according to the pre-trained scene generation model to generate dynamic scene data, and finally merges the static scene data and dynamic scene data to obtain an immersive scene model of the target location. This improves the efficiency and flexibility of generating immersive scenes by automatically generating immersive scenes that meet the user's needs.

[0057] In some feasible implementations, optionally, before inputting the static scene data into a pre-trained scene generation model for processing to generate dynamic scene data, the method further includes:

[0058] Multiple training samples are generated, each training sample including static scene data and corresponding sample labels, wherein the sample labels include dynamic scene data corresponding to the static scene data.

[0059] The initial scene generation model is trained based on the multiple training samples to obtain the scene generation model, which includes a generator and a discriminator.

[0060] In this embodiment, before training the scene generation model, multiple training samples are generated. Each training sample includes static scene data and corresponding sample labels, wherein the sample labels include dynamic scene data corresponding to the static scene data.

[0061] Specifically, the scene generation model consists of a generator and a discriminator. The generator employs a hybrid architecture combining a Convolutional Neural Network (CNN) and a Recurrent Neural Network (RNN), aiming to fully leverage the advantages of both networks in feature extraction. The CNN part consists of six convolutional layers stacked sequentially. The first convolutional layer uses a 5×5 kernel with a stride of 2, while the subsequent five layers all use 3×3 kernels with a uniform stride of 1. After each convolutional operation, a batch normalization (BN) layer is immediately applied to stabilize the data distribution during network training. Then, a non-linear transformation is introduced through the ReLU activation function to activate neurons, thereby effectively extracting spatial features from images and 3D models.

[0062] The RNN part uses a Long Short-Term Memory (LSTM) network, with two LSTM layers and 256 hidden units in each layer. Due to LSTM's ability to remember long sequences of data, this part can accurately process temporal data such as environmental parameters and uncover temporal dependencies. After feature extraction, the spatial features extracted by the CNN are concatenated and fused with the temporal features processed by the LSTM to form a complete feature representation. Finally, three fully connected layers are used to map and transform the fused features, outputting scene data that meets the requirements, thus realizing the process of generating target data from input features.

[0063] The discriminator employs a full CNN architecture, focusing on feature discrimination of image data. It consists of five CNN layers: the first layer uses a 5×5 convolutional kernel with a stride of 2, while the remaining four layers use 3×3 convolutional kernels with a stride of 1. Each convolutional operation is followed by a LeakyReLU activation function with a negative slope of 0.2 to broaden the function's range, avoid the vanishing gradient problem, and enhance the network's responsiveness to different features. After feature extraction from the convolutional layers, two fully connected layers integrate and map the features, ultimately outputting a probability value between 0 and 1. This value directly reflects the discriminator's probability of classifying the input data as real data, thus enabling the differentiation between generated and real data.

[0064] Optionally, generating multiple training samples includes:

[0065] The sample scene model is sampled to obtain multiple two-dimensional image data, multiple three-dimensional point cloud data and multiple environmental parameters. Different two-dimensional image data have different perspectives in the sample scene model. Different three-dimensional point cloud data correspond to different objects in the sample scene model. Different environmental parameters correspond to different environmental indices in the sample scene.

[0066] Image preprocessing is performed on the plurality of two-dimensional image data to obtain a plurality of target two-dimensional image data. The preprocessing is used to enhance the plurality of two-dimensional image data and unify them to the same size.

[0067] The multiple 3D point cloud data are normalized to obtain multiple target 3D point cloud data. The normalization process is used to unify the multiple 3D point cloud data into the same 3D model format.

[0068] The multiple environmental parameters are standardized to obtain multiple target environmental parameters. The standardization process is used to unify the multiple environmental parameters into the same parameter format.

[0069] Multiple sample static scene data are generated based on the multiple target two-dimensional image data and the multiple target environment parameters;

[0070] Multiple sample labels are generated based on the three-dimensional point cloud data of the multiple targets.

[0071] In this embodiment, multiple training samples are generated by sampling the sample scene model. Specifically, multi-source data is collected from the sample scene model. For example, multiple two-dimensional image data of the target scene are obtained by using drone aerial photography and panoramic camera photography, and multiple three-dimensional point cloud data of the scene are collected by using a three-dimensional laser scanner. At the same time, environmental parameters such as temperature, light intensity, and humidity are collected by environmental sensors.

[0072] After acquiring multiple 2D image data, multiple 3D point cloud data, and multiple environmental parameters, preprocessing, normalization, and standardization are performed on the multiple 2D image data, multiple 3D point cloud data, and multiple environmental parameters respectively to obtain multiple target 2D image data, multiple target 3D point cloud data, and multiple target environmental parameters.

[0073] Specifically, for two-dimensional image data, histogram equalization is used to enhance image contrast, and then bicubic interpolation algorithm is used to unify the image size to M*N pixels, where M is the width of the image and N is the height of the image. Finally, the pixel values ​​are mapped to the [0,1] interval by normalization as follows.

[0074]

[0075] in, This represents the pixel value in the i-th row and j-th column of the normalized image. This represents the original pixel value of the original image in the i-th row and j-th column. , ), and These represent the minimum and maximum values ​​of all pixels in the original image, respectively.

[0076] For 3D point cloud data, a point cloud simplification algorithm is used to remove redundant points, reducing the number of point clouds by 40%. Then, a surface reconstruction algorithm is used to convert the point cloud into a triangular mesh model and unify it into OBJ format. OBJ format is a file format used to store 3D models and is widely used in the fields of computer graphics and 3D modeling.

[0077] For environmental parameters, the mean is calculated. and standard deviation The data is converted to a standard normal distribution using the following formula, and the processed data is finally divided into training and test sets in a 7:3 ratio.

[0078]

[0079] in, These are the original environmental parameter sample values. The environmental parameter sample mean. For the sample standard deviation of environmental parameters, These are the standardized environmental parameter values.

[0080] Therefore, multiple static scene data of the samples are generated based on multiple target two-dimensional image data and multiple target environmental parameters, and multiple sample labels are generated based on the multiple target three-dimensional point cloud data, ultimately resulting in multiple training samples.

[0081] Optionally, training the initial scene generation model based on the multiple training samples to obtain the scene generation model includes:

[0082] Multiple noise data are generated based on the multiple training samples, and the multiple training samples and the multiple noise data correspond one-to-one.

[0083] The multiple training samples and the multiple noise data are input into the initial discriminator for discrimination to obtain multiple first discrimination results. The first discrimination results are used to indicate the sample authenticity of the corresponding training sample and the sample authenticity of the corresponding noise data.

[0084] Based on the plurality of first discrimination results and the plurality of sample labels, the first loss function is calculated to obtain the first loss value;

[0085] The initial discriminator is updated based on the backpropagation algorithm and the first loss value to obtain the discriminator;

[0086] The initial generator is updated based on the discriminator and the plurality of training samples to obtain the generator.

[0087] In this embodiment, an initial discriminator is trained to obtain a discriminator, and the initial generator is updated based on the discriminator to obtain a generator. Specifically, the Xavier initialization method is used to initialize the weight parameters of the generator and discriminator. The learning rate of the generator is set to 0.0002, the learning rate of the discriminator is 0.0001, the batch size is 32, and the number of training epochs is 150. During training, noisy data is randomly sampled from the training set and input into the generator. The noisy data contains random vectors that conform to a normal distribution and encoded environmental parameter data. After the generator outputs scene data, it is mixed with real data to form a new dataset which is then input into the discriminator.

[0088] After receiving generated and real data, the discriminator outputs the corresponding first discrimination result. Then, using the cross-entropy loss function (as shown in the formula below), it quantifies the difference between the discrimination result and the real label, deriving the discriminator loss value, i.e., the first loss value. Based on this first loss value, the error information is propagated backward from the output layer to each layer of the network through the backpropagation algorithm, updating the discriminator's weight parameters, thereby improving its ability to distinguish between real and fake data, and ultimately obtaining an updated discriminator.

[0089]

[0090] in, For real labels, The probability of discrimination is given by n, where n is the number of samples.

[0091] Optionally, updating the initial generator based on the discriminator and the plurality of training samples to obtain the generator includes:

[0092] The multiple training samples are input into the discriminator for discrimination, and multiple second discrimination results are obtained. The second discrimination results are used to indicate the sample authenticity of the corresponding training sample.

[0093] Based on the plurality of second discrimination results, the first loss function is calculated to obtain a second loss value;

[0094] The initial generator is updated based on the backpropagation algorithm and the second loss value to obtain the generator.

[0095] In this embodiment, after updating the discriminator parameters, the discriminator's weights are fixed so that their parameters do not change in this round. At this point, multiple training samples generated by the generator are input into the discriminator again to obtain the corresponding second discrimination result. The generator's loss is calculated using the cross-entropy loss function to measure the difference between the generated data and the real data in the discriminator's evaluation dimension, resulting in a second loss value. Based on this second loss value, the backpropagation algorithm is used again to update the generator's weights, prompting the generator to generate samples that more closely approximate the distribution of the real data.

[0096] Throughout the training process, the quality of the generated data was evaluated every 5 training epochs. Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) were selected as quantitative metrics. PSNR assesses image quality at the pixel level, with higher values ​​indicating smaller differences between the generated and real images at the pixel level. SSIM measures similarity at the perceptual level, such as image structure and texture, with values ​​closer to 1 indicating greater visual similarity. When the PSNR of the generated data reaches 35 dB and the SSIM reaches 0.85, the generator is considered to have high data generation quality. Alternatively, training is stopped after 150 epochs, regardless of whether the quality metrics are met, to avoid overtraining or getting trapped in local optima and to ensure a balance between efficiency and performance.

[0097] It should be noted that during GAN model training, to ensure training effectiveness and prevent overfitting, a scientific training termination mechanism needs to be constructed. This involves dynamically determining the training termination conditions by monitoring core performance indicators and training progress in real time. The specific strategy is as follows:

[0098] During training, key performance indicators (KPIs) of the generator and discriminator are continuously tracked. PSNR and SSIM of the generated data are used as core metrics to measure the quality of the generator's output. When PSNR consistently remains above 35dB and SSIM consistently remains above 0.85, it indicates that the generated data has approximated the real data distribution in terms of pixel-level differences and structural texture similarity. Simultaneously, the fluctuation range of the loss function is introduced as an evaluation criterion for training stability. If the fluctuation range of the loss function of both the generator and discriminator is controlled within 0.01 for 10 consecutive training rounds, it indicates that the network training has converged and the model performance no longer significantly improves. Meeting these performance indicator conditions indicates that the training has reached an ideal state.

[0099] Furthermore, to avoid training stagnation due to data characteristics or model structure, a maximum number of training epochs is set as a mandatory termination condition. When the number of training epochs reaches a pre-set 150, the training process terminates regardless of whether the performance metrics are met. Meeting either of the above two conditions constitutes a training termination condition, and training ends, propelling the model into the dynamic scene generation stage. If the above conditions are not met, the training process returns to its previous state, continuing iterative optimization of the generator and discriminator network parameters.

[0100] Optionally, fusing the dynamic scene data and the static scene data to obtain an immersive scene model corresponding to the target location includes:

[0101] Feature extraction is performed on the dynamic scene data to obtain multiple first feature points, and feature extraction is performed on the static scene data to obtain multiple second feature points;

[0102] Calculate the matching relationship between the plurality of first feature points and the plurality of second feature points to obtain a matching result, the matching result being used to indicate the position and orientation of the dynamic scene data in the static scene corresponding to the static scene data;

[0103] Based on the matching results, the texture information corresponding to the dynamic scene data is fused into the static scene corresponding to the static scene data to obtain an immersive scene model corresponding to the target location.

[0104] In this embodiment, during the fusion of dynamic and static scene data, feature extraction is first performed on the dynamic scene data to obtain multiple first feature points, and feature extraction is performed on the static scene data to obtain multiple second feature points. For example, feature points in the dynamic and static scene data can be extracted using a feature matching method and the Scale-Invariant Feature Transform (SIFT) algorithm.

[0105] By calculating the matching relationships between multiple first feature points and multiple second feature points, the position and pose of dynamic scene data in the static scene are determined, spatial alignment is completed, and matching results are generated. Then, texture mapping technology is used to fuse the texture information of the dynamic scene data into the static scene model. After fusion, the scene is optimized. For the image part, a non-local mean denoising algorithm is used to remove noise; for the 3D model, a Laplacian smoothing algorithm is used to optimize the model surface. Through physically based rendering (PBR) technology, combined with real-time acquired lighting information, the lighting effect of the scene is recalculated to make the scene more realistic.

[0106] After generating the immersive scene model, it can be converted to a format supported by the target device. For example, for VR devices, the 3D model data can be converted to the OpenXR standard format, and the image data to the ASTC texture format. The scene data is then transmitted to the target device via a high-speed data transfer interface, such as USB 3.2 or Wi-Fi 6. Upon receiving the data, the device uses its graphics processing unit (GPU) to perform real-time rendering. Users can then enter the dynamically constructed immersive scene by wearing VR headsets, AR glasses, or other devices, enabling them to interact with the scene, such as moving freely within it and manipulating virtual objects.

[0107] like Figure 2 As shown, Figure 2 The following is a schematic diagram of the overall process provided in this application embodiment: Data Acquisition and Preprocessing: Collect data that can be used for model training and perform preliminary processing. Preprocessing may include steps such as data cleaning, noise removal, standardization, and feature extraction to ensure the validity of the data in subsequent steps. Generator and Discriminator Structure Design: This step involves creating two main components of the generative adversarial network: a generator and a discriminator. The generator aims to generate samples that are as realistic as possible, while the discriminator is responsible for determining whether the generated samples are real data. Generator and Discriminator Training: By iteratively optimizing the parameters of the generator and discriminator, the generator can generate more realistic samples, and the discriminator can more accurately distinguish between real samples and generated samples. Training Stopping Conditions: During training, it is necessary to check whether the conditions for stopping training are met. This depends on changes in the loss function, the quality of the generated samples, or the number of training iterations. If the conditions are met, the training process ends and proceeds to the next step; otherwise, training continues. Dynamic Scene Generation: Once training is complete, the system will use the trained generator to generate dynamic scenes. The scenes can be animations or interactive environments that can provide users with an immersive experience. Scene Blending and Optimization: Based on the generated dynamic scene, scene blending and optimization are performed to ensure that the final output scene achieves the best visual and performance results. This includes effectively combining dynamic and static elements and optimizing rendering. Immersive Scene Output: Finally, the system outputs the generated scene as an immersive environment, which can be used in virtual reality, games, and other applications to provide users with an immersive experience.

[0108] The technical solution of this application obtains static scene data of the user at the target location, processes the static scene data according to the pre-trained scene generation model to generate dynamic scene data, and finally merges the static scene data and dynamic scene data to obtain an immersive scene model of the target location. This improves the efficiency and flexibility of generating immersive scenes by automatically generating immersive scenes that meet the user's needs.

[0109] See Figure 3 , Figure 3 This is a structural diagram of the immersive scene generation device provided in an embodiment of this application. For example... Figure 3 As shown, the generation of immersive scenes 300 includes:

[0110] The acquisition module 310 is used to acquire static scene data corresponding to the user when the user is at the target location. The static scene data includes the location information of the target location, the user's posture information, and the environmental information of the target location.

[0111] The processing module 320 is used to input the static scene data into a pre-trained scene generation model for processing to generate dynamic scene data, wherein the dynamic scene data includes object shape information, lighting effect information and texture detail information of the dynamic scene.

[0112] The fusion module 330 is used to fuse the dynamic scene data and the static scene data to obtain an immersive scene model corresponding to the target location.

[0113] Optional, also includes:

[0114] A generation module is used to generate multiple training samples. The training samples include static scene data of the samples and corresponding sample labels. The sample labels include dynamic scene data of the samples corresponding to the static scene data of the samples.

[0115] The training module is used to train the initial scene generation model based on the multiple training samples to obtain the scene generation model, which includes a generator and a discriminator.

[0116] Optionally, the generation module includes:

[0117] The sampling submodule is used to sample the sample scene model to obtain multiple two-dimensional image data, multiple three-dimensional point cloud data and multiple environmental parameters. Different two-dimensional image data have different perspectives in the sample scene model. Different three-dimensional point cloud data correspond to different objects in the sample scene model. Different environmental parameters correspond to different environmental indices in the sample scene.

[0118] The first processing submodule is used to perform image preprocessing on the plurality of two-dimensional image data to obtain a plurality of target two-dimensional image data. The preprocessing is used to enhance the plurality of two-dimensional image data and unify them to the same size.

[0119] The second processing submodule is used to perform normalization processing on the multiple three-dimensional point cloud data to obtain multiple target three-dimensional point cloud data. The normalization processing is used to unify the multiple three-dimensional point cloud data into the same three-dimensional model format.

[0120] The third processing submodule is used to standardize the multiple environmental parameters to obtain multiple target environmental parameters. The standardization process is used to unify the multiple environmental parameters into the same parameter format.

[0121] The first generation submodule is used to generate multiple sample static scene data based on the multiple target two-dimensional image data and the multiple target environment parameters;

[0122] The second generation submodule is used to generate multiple sample labels based on the multiple target 3D point cloud data.

[0123] Optionally, the training module includes:

[0124] The third generation submodule is used to generate multiple noise data based on the multiple training samples, wherein the multiple training samples and the multiple noise data correspond one-to-one.

[0125] The discrimination submodule is used to input the multiple training samples and the multiple noise data into the initial discriminator for discrimination, and obtain multiple first discrimination results. The first discrimination results are used to indicate the sample authenticity of the corresponding training sample and the sample authenticity of the corresponding noise data.

[0126] The first calculation submodule is used to perform loss calculation on the first loss function based on the plurality of first discrimination results and the plurality of sample labels to obtain a first loss value;

[0127] The first update submodule is used to update the initial discriminator based on the backpropagation algorithm and the first loss value to obtain the discriminator;

[0128] The second update submodule is used to update the initial generator based on the discriminator and the plurality of training samples to obtain the generator.

[0129] Optionally, the second update submodule includes:

[0130] The discrimination unit is used to input the plurality of training samples into the discriminator for discrimination and obtain a plurality of second discrimination results, wherein the second discrimination results are used to indicate the sample authenticity of the corresponding training sample;

[0131] The calculation unit is used to perform loss calculation on the first loss function based on the plurality of second discrimination results to obtain a second loss value;

[0132] The update unit is used to update the initial generator based on the backpropagation algorithm and the second loss value to obtain the generator.

[0133] Optionally, the fusion module 330 includes:

[0134] The extraction submodule is used to extract features from the dynamic scene data to obtain multiple first feature points, and to extract features from the static scene data to obtain multiple second feature points;

[0135] The second calculation submodule is used to calculate the matching relationship between the plurality of first feature points and the plurality of second feature points to obtain a matching result. The matching result is used to indicate the position and posture of the dynamic scene data in the static scene corresponding to the static scene data.

[0136] The fusion submodule is used to fuse the texture information corresponding to the dynamic scene data into the static scene corresponding to the static scene data based on the matching result, so as to obtain an immersive scene model corresponding to the target position.

[0137] The technical solution of this application obtains static scene data of the user at the target location, processes the static scene data according to the pre-trained scene generation model to generate dynamic scene data, and finally merges the static scene data and dynamic scene data to obtain an immersive scene model of the target location. This improves the efficiency and flexibility of generating immersive scenes by automatically generating immersive scenes that meet the user's needs.

[0138] This application also provides an electronic device. Please refer to [link to relevant documentation]. Figure 4 The electronic device may include a processor 401, a memory 402, and a program 4021 stored in the memory 402 and executable on the processor 401.

[0139] When program 4021 is executed by processor 401, it can achieve the following: Figure 1 Any step in the corresponding method embodiment:

[0140] Acquire static scene data corresponding to the user when the user is at the target location, the static scene data including the location information of the target location, the user's posture information and the environmental information of the target location;

[0141] The static scene data is input into a pre-trained scene generation model for processing to generate dynamic scene data, which includes object shape information, lighting effect information and texture detail information of the dynamic scene.

[0142] The dynamic scene data and the static scene data are fused to obtain an immersive scene model corresponding to the target location.

[0143] Optionally, before inputting the static scene data into a pre-trained scene generation model for processing to generate dynamic scene data, the method further includes:

[0144] Multiple training samples are generated, each training sample including static scene data and corresponding sample labels, wherein the sample labels include dynamic scene data corresponding to the static scene data.

[0145] The initial scene generation model is trained based on the multiple training samples to obtain the scene generation model, which includes a generator and a discriminator.

[0146] Optionally, the sample scene model is sampled to obtain multiple two-dimensional image data, multiple three-dimensional point cloud data and multiple environmental parameters. Different two-dimensional image data have different perspectives in the sample scene model. Different three-dimensional point cloud data correspond to different objects in the sample scene model. Different environmental parameters correspond to different environmental indicators in the sample scene.

[0147] Image preprocessing is performed on the plurality of two-dimensional image data to obtain a plurality of target two-dimensional image data. The preprocessing is used to enhance the plurality of two-dimensional image data and unify them to the same size.

[0148] The multiple 3D point cloud data are normalized to obtain multiple target 3D point cloud data. The normalization process is used to unify the multiple 3D point cloud data into the same 3D model format.

[0149] The multiple environmental parameters are standardized to obtain multiple target environmental parameters. The standardization process is used to unify the multiple environmental parameters into the same parameter format.

[0150] Multiple sample static scene data are generated based on the multiple target two-dimensional image data and the multiple target environment parameters;

[0151] Multiple sample labels are generated based on the three-dimensional point cloud data of the multiple targets.

[0152] Optionally, training the initial scene generation model based on the multiple training samples to obtain the scene generation model includes:

[0153] Multiple noise data are generated based on the multiple training samples, and the multiple training samples and the multiple noise data correspond one-to-one.

[0154] The multiple training samples and the multiple noise data are input into the initial discriminator for discrimination to obtain multiple first discrimination results. The first discrimination results are used to indicate the sample authenticity of the corresponding training sample and the sample authenticity of the corresponding noise data.

[0155] Based on the plurality of first discrimination results and the plurality of sample labels, the first loss function is calculated to obtain the first loss value;

[0156] The initial discriminator is updated based on the backpropagation algorithm and the first loss value to obtain the discriminator;

[0157] The initial generator is updated based on the discriminator and the plurality of training samples to obtain the generator.

[0158] Optionally, updating the initial generator based on the discriminator and the plurality of training samples to obtain the generator includes:

[0159] The multiple training samples are input into the discriminator for discrimination, and multiple second discrimination results are obtained. The second discrimination results are used to indicate the sample authenticity of the corresponding training sample.

[0160] Based on the plurality of second discrimination results, the first loss function is calculated to obtain a second loss value;

[0161] The initial generator is updated based on the backpropagation algorithm and the second loss value to obtain the generator.

[0162] Optionally, fusing the dynamic scene data and the static scene data to obtain an immersive scene model corresponding to the target location includes:

[0163] Feature extraction is performed on the dynamic scene data to obtain multiple first feature points, and feature extraction is performed on the static scene data to obtain multiple second feature points;

[0164] Calculate the matching relationship between the plurality of first feature points and the plurality of second feature points to obtain a matching result, the matching result being used to indicate the position and orientation of the dynamic scene data in the static scene corresponding to the static scene data;

[0165] Based on the matching results, the texture information corresponding to the dynamic scene data is fused into the static scene corresponding to the static scene data to obtain an immersive scene model corresponding to the target location.

[0166] The technical solution of this application obtains static scene data of the user at the target location, processes the static scene data according to the pre-trained scene generation model to generate dynamic scene data, and finally merges the static scene data and dynamic scene data to obtain an immersive scene model of the target location. This improves the efficiency and flexibility of generating immersive scenes by automatically generating immersive scenes that meet the user's needs.

[0167] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes of the above-described immersive scene generation embodiment and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0168] This application also provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the above-described immersive scene generation method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0169] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a communication device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0171] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for generating an immersive scene, characterized in that, The method includes: Acquire static scene data corresponding to the user when the user is at the target location, the static scene data including the location information of the target location, the user's posture information and the environmental information of the target location; The static scene data is input into a pre-trained scene generation model for processing to generate dynamic scene data, which includes object shape information, lighting effect information and texture detail information of the dynamic scene. The dynamic scene data and the static scene data are fused to obtain an immersive scene model corresponding to the target location.

2. The method according to claim 1, characterized in that, Before inputting the static scene data into a pre-trained scene generation model for processing to generate dynamic scene data, the method further includes: Multiple training samples are generated, each training sample including static scene data and corresponding sample labels, wherein the sample labels include dynamic scene data corresponding to the static scene data. The initial scene generation model is trained based on the multiple training samples to obtain the scene generation model, which includes a generator and a discriminator.

3. The method according to claim 2, characterized in that, The generation of multiple training samples includes: The sample scene model is sampled to obtain multiple two-dimensional image data, multiple three-dimensional point cloud data and multiple environmental parameters. Different two-dimensional image data have different perspectives in the sample scene model. Different three-dimensional point cloud data correspond to different objects in the sample scene model. Different environmental parameters correspond to different environmental indices in the sample scene. Image preprocessing is performed on the plurality of two-dimensional image data to obtain a plurality of target two-dimensional image data. The preprocessing is used to enhance the plurality of two-dimensional image data and unify them to the same size. The multiple 3D point cloud data are normalized to obtain multiple target 3D point cloud data. The normalization process is used to unify the multiple 3D point cloud data into the same 3D model format. The multiple environmental parameters are standardized to obtain multiple target environmental parameters. The standardization process is used to unify the multiple environmental parameters into the same parameter format. Multiple sample static scene data are generated based on the multiple target two-dimensional image data and the multiple target environment parameters; Multiple sample labels are generated based on the three-dimensional point cloud data of the multiple targets.

4. The method according to claim 2, characterized in that, The process of training the initial scene generation model based on the multiple training samples to obtain the scene generation model includes: Multiple noise data are generated based on the multiple training samples, and the multiple training samples and the multiple noise data correspond one-to-one. The multiple training samples and the multiple noise data are input into the initial discriminator for discrimination to obtain multiple first discrimination results. The first discrimination results are used to indicate the sample authenticity of the corresponding training sample and the sample authenticity of the corresponding noise data. Based on the plurality of first discrimination results and the plurality of sample labels, the first loss function is calculated to obtain the first loss value; The initial discriminator is updated based on the backpropagation algorithm and the first loss value to obtain the discriminator; The initial generator is updated based on the discriminator and the plurality of training samples to obtain the generator.

5. The method according to claim 4, characterized in that, The step of updating the initial generator based on the discriminator and the plurality of training samples to obtain the generator includes: The multiple training samples are input into the discriminator for discrimination, and multiple second discrimination results are obtained. The second discrimination results are used to indicate the sample authenticity of the corresponding training sample. Based on the plurality of second discrimination results, the first loss function is calculated to obtain a second loss value; The initial generator is updated based on the backpropagation algorithm and the second loss value to obtain the generator.

6. The method according to claim 1, characterized in that, The step of fusing the dynamic scene data and the static scene data to obtain an immersive scene model corresponding to the target location includes: Feature extraction is performed on the dynamic scene data to obtain multiple first feature points, and feature extraction is performed on the static scene data to obtain multiple second feature points; Calculate the matching relationship between the plurality of first feature points and the plurality of second feature points to obtain a matching result, the matching result being used to indicate the position and orientation of the dynamic scene data in the static scene corresponding to the static scene data; Based on the matching results, the texture information corresponding to the dynamic scene data is fused into the static scene corresponding to the static scene data to obtain an immersive scene model corresponding to the target location.

7. An immersive scene generation device, characterized in that, The device includes: The acquisition module is used to acquire static scene data corresponding to the user when the user is at the target location. The static scene data includes the location information of the target location, the user's posture information, and the environmental information of the target location. The processing module is used to input the static scene data into a pre-trained scene generation model for processing to generate dynamic scene data, wherein the dynamic scene data includes object shape information, lighting effect information and texture detail information of the dynamic scene. The fusion module is used to fuse the dynamic scene data and the static scene data to obtain an immersive scene model corresponding to the target location.

8. An electronic device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 6.

Images