A Text-Driven Immersive Open-Scene Neural Rendering and Hybrid Enhancement Approach

Abstract

This invention relates to a text-driven immersive large-scene neural rendering and blending enhancement method, comprising: 1. Creating a dataset; 2. Reconstructing the large scene based on an improved progressive neural radiation field; 3. Predicting the foreground and background matte values ​​of the rendered image based on a convolutional neural network; 4. Generating the background from text using a stable diffusion model; 5. Calculating background motion through changes in camera parameters between adjacent frames; 6. Blending the rendered foreground and background and coordinating lighting. This invention achieves real-time text-driven, editable large-scene background enhancement, allowing rendering of immersive large scenes at any observation position at a city scale, generating observation images with effects consistent with the real scene, and enabling mixed reality on this basis. It can satisfy users' personalized customization of the scene and achieve visual effects at the level of film and television special effects. This technology can be applied to fields such as 3D visualization, digital mapping, and virtual reality games.

Description

Technical Field

[0001] This invention relates to an immersive open scene and hybrid enhancement method, belonging to the field of computer vision. Background Technology

[0002] (1) Novel View Synthesis of 3D Scene refers to synthesizing scene images from previously unseen angles given multiple images and camera positions. This technique can be divided into two stages: the first stage recovers the 3D geometry from the multi-view images, and the second stage synthesizes the image based on the given viewpoint.

[0003] The main challenge in 3D scene novel perspective synthesis technology is the quality of 3D scene representation. The quality of 3D scene representation is crucial to the quality of the generated images from novel perspectives. Existing 3D scene representation methods include explicit and implicit representations. These methods have high requirements for datasets and can only be used in specific scenarios, thus limiting their application scope.

[0004] Currently, there are three main methods for achieving new perspective synthesis in 3D scenes: mesh-based new perspective synthesis, learning-based new perspective synthesis, and new perspective synthesis based on multilayer perceptron (MLP) and neural radiance field (NERF) methods.

[0005] Novel perspective synthesis based on point clouds works by recording discrete points on the surface of a 3D object in space to reconstruct its 3D model, and then using perspective transformations from different viewpoints to infer the object's image from a new perspective. Because this method records discrete 3D structures, the memory requirements increase exponentially when the reconstructed scene is large and high resolution is required.

[0006] Learning-based novel perspective synthesis works by predicting 3D feature embeddings for each voxel grid from a set of images with location information. Differentiable rendering is then used to reproduce the corresponding known view. However, these methods require mesh optimization, and obtaining the necessary initial template mesh is difficult. Furthermore, these methods often produce severe artifacts behind occluded areas.

[0007] Novel perspective synthesis based on deep learning has received widespread attention from the academic community in recent years. By using a multilayer perceptron to query the transparency and color values ​​of corresponding points in space using the input 5D vector (3D coordinates and 2D view orientation), the neural radiance field method also has great advantages in perspective synthesis. However, it still only has good training effect on foregrounds with actual depth values ​​and requires information such as the latitude and longitude of the scene center and the size of buildings in the scene.

[0008] (2) Mixed reality is a technology that cleverly integrates virtual information with the real world, and it has been widely used in film production, virtual reality games, and other fields. Background enhancement technology is mainly used for sky background enhancement. Currently, there are two main ways to achieve sky background enhancement: methods based on manual post-production and methods based on vision-based image segmentation and matching.

[0009] Manual post-production methods are primarily used in professional film and game special effects production. However, manually replacing the sky area in a video is often time-consuming and labor-intensive, sometimes requiring specialized post-processing techniques. The process typically involves frame-by-frame blue screen matching and background motion capture. Even with the aid of professional software, users may spend hours editing a few seconds of video. Mobile devices can perform real-time sky and background segmentation and image fusion, but this method requires specific cameras and gyroscopes and cannot effectively handle background enhancement during camera panning.

[0010] A vision-based image segmentation and matching method first uses a convolutional neural network to smoothly match foreground and background parts of the image by predicting the matte sky at the pixel level. Then, an iterative Lucas method with a pyramid is used to calculate optical flow, estimating background motion by tracking feature points on the sky background frame by frame. Finally, the affine background image is fused with the foreground. However, due to the method's dependence on background feature points, it cannot enhance scenes with solid-color backgrounds (such as sunny days or nighttime).

[0011] This invention proposes a text-driven immersive open-scene neural rendering and hybrid enhancement method, solving the problems of 3D scene representation quality in new perspective synthesis and low universality of background enhancement techniques. It requires only a textual description of background features, and the method automatically generates an immersive enhanced scene based on the user's description, allowing for user-editable scenes. This invention utilizes a progressive neural radiation field model for 3D scene reconstruction and new perspective synthesis, resulting in reconstructed images with clear edges and rich texture details, thus improving the quality of 3D scene representation. This invention enhances the reconstructed scene background using camera pose coordinates. This method can estimate the affine transformation of the background image in real time to match it with each frame of the rendered foreground image, and can enhance solid-color backgrounds, exhibiting high universality and good background enhancement effects. This invention uses a stable diffusion model to generate the background, enabling the generation of immersive scenes based on user-input background feature descriptions, meeting users' personalized needs. Summary of the Invention

[0012] 1. Objective: This invention aims to provide a text-driven, immersive open-scene neural rendering and hybrid enhancement method to achieve real-time text-driven scene background enhancement. It allows rendering from any observation position within a city-scale scene. Furthermore, by integrating mixed reality, it provides users with personalized scene customization, achieving cinematic visual effects. This invention not only significantly reduces the workload in traditional 3D modeling and film special effects production but also enables non-professionals to easily edit and create personalized mixed reality scenes. This invention enriches and improves deep learning-based scene rendering and enhancement methods, and can be applied to fields such as 3D visualization, digital mapping, and virtual reality games.

[0013] 2. Technical Solution: This invention is achieved through the following technical solution:

[0014] This invention is a technique for hybrid enhancement based on progressive neural radiation field for scene neural rendering and text-driven rendering. It consists of five steps: reconstructing the scene based on the improved progressive neural radiation field, predicting the foreground and background matte values ​​of the rendered image based on the convolutional neural network, generating the background from the text using a stable diffusion model, calculating the background motion through changes in camera parameters between adjacent frames, and blending the rendered foreground and background and coordinating the lighting.

[0015] Step 1: Reconstructing the scene based on an improved progressive neural radiation field

[0016] This invention establishes and trains a progressive neural radiation field model in a progressive manner, starting with large-scale data and gradually training it to enable different network layers of the neural radiation field (specific structures as follows) to be trained. Figure 1There is a clear division of labor between the two, and the dimension of spatial location coding is gradually increased as the training phase increases, so as to make full use of the information of the entire frequency band in the location coding with a smaller number of input parameters.

[0017] This invention divides each training scenario into... At each scale, the distance between adjacent pixels in world coordinates is approximately twice the distance between adjacent pixels. During training, the input parameters of the neural radiation field start from the farthest viewpoint (...). The image data begins in [the context of the image]. Distant views typically present relatively simple texture details, so a shallow multilayer perceptron can be used as the base block. As training progresses, each new training phase incorporates a closer viewpoint. Meanwhile, the data from the previous stage continues to be used for training. As the training set expands, the neural radiation field model introduces residual blocks to capture the increasing complexity and detail in scene components. The design of these residual blocks allows the intermediate layers to focus on learning details missing from the larger-scale training phase and utilizes skip connections to provide high-frequency spatial features. Throughout the training process, the base blocks and all added residual blocks in the network remain trainable.

[0018] This invention employs a multi-level supervision strategy to ensure consistent rendering quality across all scales. During the training phase... In this stage, the output after superimposing the base block and each residual block is... Joint supervision of images at all scales, i.e. .stage The losses are summarized from arrive All output headers:

[0019]

[0020] In the formula, For the first Scale datasets, For pixel coordinates in the dataset, These are the ground truth value and the corresponding predicted pixel value (RGB), respectively.

[0021] The multi-level supervision strategy adopted in this invention unifies different levels of detail into a single model, which can be controlled by L.

[0022] Step 2: Predict the foreground and background matte values ​​of the rendered image using a convolutional neural network.

[0023] In step one, the present invention completes the reconstruction of the scene, and by pre-setting the trajectory of the observation camera, a rendered image from a new perspective can be obtained.

[0024] This invention utilizes a convolutional neural network (CNN) to accurately extract the rendered image from step one, predicting sky matte values ​​within a pixel-level regression framework. This separates the background region (the sky, etc., can be considered as an infinitely distant region) and the foreground region (the region excluding the background). This framework generates coarse-scale sky matte, and then guided filtering is used for fine-scale matte prediction. The convolutional neural network consists of a segmentation encoder E, a mask prediction decoder D, and a guided filtering module.

[0025] The segmentation encoder uses a convolutional architecture similar to that of a residual network (ResNet) to learn intermediate feature representations from the downsampled input image. The mask prediction decoder uses an upsampled network with several convolutional layers to predict a coarse sky matte. The guided filtering module uses the coarse sky matte and the original input image to generate a refined sky matte. Since the sky region typically appears at the top of the image, this invention uses coordinate convolutional layers in the encoder's input layer and all upsampled layers of the decoder.

[0026] Assumption and The network represents an input image at full resolution and its downsampled version. The input is The output is the same as Matte sky images of the same size. Assuming... and This represents the predicted and ground truth values ​​of sky matteness. This invention trains a network to minimize the error between the predicted and ground truth values ​​of sky matteness, defining the loss function as follows:

[0027]

[0028] In the formula, where It is a 2-norm. It is the number of pixels in the image. It is a downsampled image dataset.

[0029] In the guided filtering module, this invention uses a full-resolution image I as the guiding image. This image is stored in the computer using three RGB channels (red, green, and blue). To obtain better color contrast for the sky, this invention only retains the blue channel. The filtering transfers the structure of the guiding image to the low-resolution matte sky and produces a more accurate result than the output of a convolutional neural network. The refined predicted full-resolution matte sky... It can be represented as:

[0030]

[0031] In the formula, where and It involves guided filtering and bilinear upsampling operations. and These are the predefined radius and regularization coefficients of the guided filter.

[0032] Step 3: Generate background from text using a stable diffusion model

[0033] The development of generative models has made it possible to directly generate images. Among many models, the probability diffusion model has stronger generative capabilities and is suitable for inductive reasoning of image-like data biases by using the UNet network skeleton.

[0034] This invention utilizes the publicly available pre-trained probability diffusion model SD2.1-v to accurately generate background images from input prompt labels. To ensure users can consistently generate high-quality background enhancement images, this invention adds "background," "distant scene," and "no ground" to the user-input prompt labels, guiding the model to generate better background images.

[0035] Step 4: Calculate background motion by analyzing changes in camera parameters between adjacent frames.

[0036] In step four, the present invention completes the generation of the background image. Considering that the scene is rendered and generated after the camera pose is input, the present invention estimates the motion of the background by using the changes in camera pose in each frame of multiple rendered images to match the changes in the viewpoint of the foreground.

[0037] Assuming the background is at infinity, it has only 3 degrees of freedom in rigid planar motion; therefore, its affine matrix can be derived from... To indicate:

[0038]

[0039] in These are the x and y coordinates of the current frame's pixels; These are the x and y coordinates of the pixels in the previous frame.

[0040] Assume that the transformation matrices of two adjacent frames from the camera coordinate system to the world coordinate system are respectively Then the transformation matrix from the previous frame to the next frame can be calculated:

[0041]

[0042] In the formula, Let be the rotation matrix for the transformation between the two cameras. This is the translation vector between the two cameras. Therefore, the transformation relationship of pixel coordinates between two adjacent frames can be calculated:

[0043]

[0044] Where K is the camera's intrinsic parameter matrix, n is the unit vector along the z-axis, and z is the distance from the background plane to the camera. When the transformation relationship is such that:

[0045]

[0046] Therefore, the solution can be found as follows:

[0047]

[0048] Obtain the affine matrix of each adjacent frame. After that, render the initial frame and the first frame in the image. Affine matrix between frames This can be written in the following matrix multiplication form:

[0049]

[0050] in This is the transformation matrix cropped from the center of the original background image in the first frame. Therefore, the pixel coordinates of the background image are multiplied by the affine matrix on the left. Then, bilinear interpolation can be used to obtain the first... The background image to be merged in the frame.

[0051] Step 5: Blend the rendered foreground and background and adjust the lighting.

[0052] Through the above steps, this invention completes the extraction of the foreground and background regions of the rendered image, the generation of the background image, and the prediction of the sky matte value. This invention fuses the foreground and background images based on an image matching equation and a correction.

[0053] Using the image matching equation, the higher the output pixel value of the sky matte value, the higher the probability that the pixel belongs to the sky background. Therefore, the fused image Y(t) of frame t can be expressed as:

[0054]

[0055] In the formula, I(t) is the foreground image of the t-th frame, A(t) is the predicted sky matte value, and B(t) is the background image to be fused.

[0056] This invention utilizes the concept of lighting coordination to assign the hue and brightness of the background to the foreground, thereby ensuring that the hue and brightness of the foreground and background images are consistent. The correction formula is as follows:

[0057]

[0058] in They are respectively and The average brightness These represent the average brightness of the background at a matte prediction value of 1 in the original image and the average brightness of the foreground in the original image, respectively. These are two adjustable parameters, whose physical meanings are the proportion of the original image's foreground brightness and the proportion of the background brightness's influence on the foreground, respectively. These represent the intermediate variables and the final output image in the illumination coordination process, respectively.

[0059] 3. Advantages and effects:

[0060] This invention is a text-driven immersive scene rendering and hybrid enhancement method. Its advantages include: Firstly, it optimizes the data processing and lighting calculations in progressive neural radiation fields. Only multiple images of the same scene and their corresponding camera parameters are required as input, allowing the network to generate a new foreground image from any position within the scene. This method boasts wide applicability and low model training complexity. Secondly, it directly estimates background motion using camera pose information, enabling enhancement of scenes with solid-color backgrounds and demonstrating high method versatility. Thirdly, it coordinates the lighting of the foreground and background images, ensuring a harmonious match in hue and brightness, resulting in realistic generated effects. Attached Figure Description

[0061] Figure 1 The structural diagram of the multilayer perceptron designed in this invention.

[0062] Figure 2 A schematic diagram of multi-scale joint progressive training of the dataset.

[0063] Figure 3 Background enhancement technology flowchart.

[0064] Figure 4 Text-driven image generation. Detailed Implementation

[0065] To better understand the technical solution of the present invention, the embodiments of the present invention will be further described below with reference to the accompanying drawings:

[0066] This invention is implemented using Python within the PyTorch framework. First, a progressive neural radiation field model and a background segmentation convolutional network are constructed, and relevant hyperparameters are set. Then, a preprocessed dataset is read in to iteratively optimize the neural radiation field and the convolutional neural network.

[0067] The computer is equipped with an Intel(R) Core(TM) i5-10300H processor with a clock speed of 2.50GHz, 16GB of memory, and an NVIDIA GeForce GTX 1650 graphics card with 4.0GB of video memory.

[0068] This text-driven immersive scene rendering and blending enhancement method includes the following steps:

[0069] Step 1: Creating a dataset

[0070] The dataset of this invention includes 2D images of classic domestic scenes and corresponding camera intrinsic and extrinsic parameter sets. Some possible scene examples in the dataset are shown in Table 1. The specific camera parameter storage format is as follows: for each frame image, take... The first three rows of the extrinsic parameter matrix are processed, and the image width, image height, and camera focal length are added to the end of each row. Then, the 15 values ​​are stored in a .json file.

[0071] This invention constructs datasets from various data sources. For international scenarios, Google Earth Studio is used to acquire 3D images and virtual camera coordinates; for domestic scenarios, drone aerial videos are obtained from YouTube and Bilibili streaming platforms, and multi-view reconstruction is performed by capturing feature points from different frames to estimate the extrinsic parameters of each camera.

[0072] Table 1: Dataset Scenarios and Reconstruction Metrics

[0073]

[0074] Step 2: Reconstructing the scene based on the improved progressive neural radiation field

[0075] This invention establishes and trains the model in a progressive manner, starting with large-scale data and gradually training it to ensure clear division of labor between different network layers. As the training phase increases, the dimension of spatial location coding is gradually increased to fully utilize the information of the entire frequency band in the location coding with a smaller input.

[0076] This invention divides each scene to be trained into L scales, where the distance between adjacent pixels in world coordinates between two scales is a factor of 2. At the start of training, the model begins with the furthest viewpoint (L = 1). Far-distance views typically exhibit relatively simple texture details; therefore, a shallow multilayer perceptron can be used as the base block, with four hidden layers, each containing... W = 256 hidden units to accommodate the furthest scale I l= 1. As training progresses, each new training phase incorporates a closer perspective (L + 1), while data from the previous phase continues to be used for training. As the training set expands, the model introduces residual blocks to capture increasingly complex details in scene components. The design of these residual blocks allows intermediate layers to focus on learning details missing from larger-scale training phases and utilizes skip connections to provide high-frequency spatial features. Throughout the training process, all layers in the network remain trainable.

[0077] The multilayer perceptron network model with L = 2 as an example is as follows: Figure 1 As shown, in the input vector, Spatial location encoding for three-dimensional position coordinates, This encodes the spatial location of two-dimensional directional coordinates. In the hidden layers, numbers represent the dimension of the output vector after passing through that layer. The final output prediction is the sum of the basic output trained by the L1 network and the residual network obtained from the L2 network.

[0078] This invention employs multi-level supervision to ensure consistent rendering quality across all scales. In the L-th stage of training, the output after superimposing the base block and each residual block is jointly supervised by images from all scales up to L, i.e., I... l <=L. The sum of the losses in stage L is from H1 to H. L All output headers:

[0079]

[0080] In the formula, C(r) and Cˆ(r) are the true values ​​of the image pixels and the corresponding predicted RGB values.

[0081] The multi-level supervision strategy employed in this invention unifies different levels of detail into a single model, which can be controlled using L. A scale diagram is shown below. Figure 2 As shown.

[0082] In step two, the present invention completes the reconstruction of the scene. By setting the trajectory of the observation camera, the present invention can obtain mixed reality images from a new perspective. The flowchart is as follows. Figure 3 As shown, the entire process can be divided into a foreground extraction module, a background motion calculation module, a text-driven generation module, and a real-time rendering enhancement module.

[0083] Step 3: Predict the foreground and background matte values ​​of the rendered image using a convolutional neural network.

[0084] In the foreground extraction module, a convolutional neural network is used to accurately extract the foreground, and the sky matte value is predicted under a pixel-level regression framework. This framework generates coarse-scale sky matte, and then guided filtering is used for fine-scale matte prediction. This method consists of a segmentation encoder network E, a mask prediction decoder network D, and a guided filtering module.

[0085] Table 2 shows the structure of the segmentation encoder network and the mask prediction decoder network D. The segmentation encoder uses the same convolutional architecture as the residual network to learn the intermediate feature representation of the downsampled input image. The mask prediction decoder uses an upsampling network with convolutional layers to predict a coarse sky matte. The guided filtering module uses the coarse sky matte and the original input image to generate a refined sky matte. Since the sky region usually appears at the top of the image, this invention uses coordinate convolutional layers in the input layer of the segmentation encoder and all upsampling layers of the mask prediction decoder, and adds the horizontal and vertical coordinates of the image to the last two channels of the original convolutional layer.

[0086] Table 2: Network configuration parameters, where CoordConv represents coordinate convolutional layer with ReLU activation function; BN represents batch normalization; UP represents bilinear upsampling; and Pool represents max pooling.

[0087]

[0088]

[0089] Assumption and This represents an input image at full resolution and its image downsampled by 64 times. The network... The input is The output is Matte sky images of the same size. Assuming... and This represents the predicted and ground truth values ​​of sky matteness. This invention trains a network to minimize the error between the predicted and ground truth values ​​of sky matteness, defining the loss function as follows:

[0090]

[0091] In the formula, where The square of the L2 norm. It is the number of pixels in the image. It is a downsampled image dataset.

[0092] In the guided filtering module, this invention uses a full-resolution image. As a guide image, only the blue channel of the RGB tri-channel is retained to obtain better color contrast for the sky. Filtering transfers the structure of the guide image to a low-resolution matte sky, producing a more accurate result than the CNN output. The refined, predicted full-resolution matte sky... It can be represented as:

[0093]

[0094] In the formula, where and It involves guided filtering and bilinear upsampling operations. and These are the predefined radius and regularization coefficients of the guided filter.

[0095] Step 4: Generate background from text using a stable diffusion model

[0096] In the text-driven generation module of this invention, a pre-trained stable diffusion model SD2.1-v (a publicly disclosed stable model capable of generating accurate images from text) is used to generate a background image from the input text. The input image size is specified (1080 pixels long, 1920 pixels wide), and a prompt label is entered. Specific examples are provided below. Figure 4 As shown in (a) to (f), the captions represent the input prompt tags. This invention adds three prompts by default after each input prompt tag: "background," "distant scene," and "no ground," guiding the model to generate a better background image.

[0097] Step 5: Calculate background motion by analyzing changes in camera parameters between adjacent frames.

[0098] In step four, the present invention completes the generation of the background image. In the background motion calculation module, the present invention first assumes that the background is at infinity, and then uses the change of camera pose to calculate the motion of the background to match the change of the foreground viewpoint.

[0099] Assuming the background is at infinity, it has only 3 degrees of freedom in rigid planar motion; therefore, its affine matrix can be derived from... To indicate:

[0100]

[0101] Assume that the transformation matrices of two adjacent frames from the camera coordinate system to the world coordinate system are respectively Then the transformation matrix from the previous frame to the next frame can be calculated:

[0102]

[0103] In the formula, Let be the rotation matrix for the transformation between the two cameras. This is the translation vector between the two cameras. Therefore, the transformation relationship of pixel coordinates between two adjacent frames can be calculated:

[0104]

[0105] Where K is the camera's intrinsic parameter matrix, n is the unit vector along the z-axis, and z is the distance from the background plane to the camera. When the transformation relationship is such that:

[0106]

[0107] Therefore, the solution can be found as follows:

[0108]

[0109] Obtain the affine matrix of each adjacent frame. Then, the affine matrix between the initial frame and the t-th frame in the video. This can be written in the following matrix multiplication form:

[0110]

[0111] Where Mc is the transformation matrix cropped from the center of the original background image in the first frame. Therefore, the pixel coordinates of the background image are multiplied by the affine matrix on the left. Then, bilinear interpolation can be used to obtain the background image to be fused in frame t.

[0112] Step Six: Blend the rendered foreground and background and adjust the lighting.

[0113] Through the above steps, this invention completes neural rendering of the foreground image, text-driven generation of the background image, and predicts the matte value of the sky based on a convolutional neural network containing spatial coordinate encoding. In the real-time rendering enhancement module, this invention fuses the foreground and background images based on an image matching equation, achieving a mixed reality effect.

[0114] Using the image matching equation, the higher the output pixel value of the sky matte value, the higher the probability that the pixel belongs to the sky background. Therefore, the fused image Y(t) of frame t can be expressed as:

[0115]

[0116] In the formula, I(t) is the foreground image of the t-th frame, A(t) is the predicted sky matte value, and B(t) is the background image to be fused.

[0117] This invention utilizes the concept of lighting coordination to assign the hue and brightness of the background to the foreground, thereby ensuring that the hue and brightness of the foreground and background images are consistent. The correction formula is as follows:

[0118]

[0119] in Let I(t) and I'(t) be the mean brightness values, respectively. These represent the average brightness of the background at a matte prediction value of 1 in the original image and the average brightness of the foreground in the original image, respectively. These are two adjustable parameters, physically representing the proportion of foreground brightness in the original image and the proportion of background brightness's influence on the foreground, respectively. When the foreground and background brightness are similar, You can take 0.5 from each.

Claims

1. A text-driven immersive open-scene neural rendering and blending enhancement method, characterized in that: Includes the following steps: Step 1: Reconstructing the scene based on an improved progressive neural radiation field The progressive neural radiation field model is built and trained in a progressive manner, starting with large-scale data and gradually training it. This allows for clear division of labor between different network layers of the neural radiation field, and gradually increases the dimension of spatial location encoding as the training stage increases. The input parameters fully utilize the information of the entire frequency band in the location encoding. Step 2: Predict the foreground and background matte values ​​of the rendered image based on a convolutional neural network. In step one, the scene is reconstructed, and a rendered image from a new perspective is obtained by pre-setting the trajectory of the observation camera. The image rendered in step one is extracted using a convolutional neural network. The sky matte value is predicted under a pixel-level regression framework, thereby separating the background region (the region at infinity) and the foreground region (the region excluding the background). The framework generates a coarse-scale sky matte, and then guided filtering is used for fine matte prediction. The convolutional neural network consists of a segmentation encoder E, a mask prediction decoder D, and a guided filtering module. Step 3: Generate background from text using a stable diffusion model Using the publicly available pre-trained probability diffusion model SD2.1-v, we can accurately generate background images from input prompt labels. We can also add "background", "distant scene" and "no ground" to the user-input prompt labels to guide the model to generate better background images. Step 4: Calculate background motion by analyzing changes in camera parameters between adjacent frames. In step four, the background image is generated. Considering that the scene is rendered after the camera pose is input, the motion of the background is estimated by using the changes in the camera pose in each frame of multiple rendered images to match the changes in the perspective of the foreground. Step 5: Blend the rendered foreground and background and adjust the lighting. After the above steps 1-4, the foreground and background regions of the rendered image are extracted, the background image is generated, and the sky matte value is predicted; based on the image matching equation and based on the correction, the foreground and background images are fused.

2. The text-driven immersive open-scene neural rendering and blending enhancement method according to claim 1, characterized in that: In step one, each training scenario is divided into... In two scales, the distance between adjacent pixels in world coordinates is twice that of each other. During training, the input parameters of the neural radiation field are from the furthest viewpoint. The training begins with image data; distant views present relatively simple texture details, and a shallow multilayer perceptron is used as the base block. As training progresses, a closer viewpoint is added in each new training phase. Meanwhile, the data from the previous stage will continue to be used for training. As the training set expands, the neural radiation field model introduces residual blocks to capture the increasing complexity of details in scene components. The design of the residual blocks allows the intermediate layers to focus on learning details missing in the large-scale training phase and utilizes skip connections to provide high-frequency spatial features. Throughout the training process, the base blocks and all added residual blocks in the network remain trainable.

3. A text-driven immersive open-scene neural rendering and hybrid enhancement method according to claim 1 or 2, characterized in that: A multi-level supervision strategy is employed to ensure consistent rendering quality across all scales; during the training phase... In this stage, the output after superimposing the base block and each residual block is... Joint supervision of images at all scales, i.e. ;stage The losses are summarized from arrive All output headers: ； In the formula, For the first Scale datasets, For pixel coordinates in the dataset, These are the ground truth value and the corresponding predicted pixel value (RGB), respectively.

4. The text-driven immersive open-scene neural rendering and blending enhancement method according to claim 3, characterized in that: The multi-level supervision strategy adopted unifies different levels of detail into a single model, which is controlled by L.

5. The text-driven immersive open-scene neural rendering and hybrid enhancement method according to claim 1, characterized in that: In step two, the segmentation encoder uses the same convolutional architecture as the residual network to learn intermediate feature representations of the downsampled input image; the mask prediction decoder uses an upsampled network with several convolutional layers to predict a coarse sky matte; the guided filtering module uses the coarse sky matte and the original input image to generate a refined sky matte; coordinate convolutional layers are used in the input layer of the encoder and all upsampled layers of the decoder.

6. A text-driven immersive open-scene neural rendering and blending enhancement method according to claim 1 or 5, characterized in that: set up and The network represents an input image at full resolution and its downsampled version. The input is The output is the same as Matte sky images of the same size; assuming and Let represent the predicted and ground truth values ​​of the matte sky. The network is trained to minimize the error between the predicted and ground truth values ​​of the matte sky, and the loss function is defined as follows: ； In the formula, where It is a norm 2. It is the number of pixels in the image. It is a downsampled image dataset.

7. The text-driven immersive open-scene neural rendering and blending enhancement method according to claim 6, characterized in that: In the guided filtering module, the full-resolution image I is used as the guide image. This image is stored in the computer using three channels: red, green, and blue (RGB). Here, only the blue channel is retained. Filtering transfers the structure of the guiding image to the low-resolution sky matte, while thinning predicts the full-resolution sky matte. Represented as: ； In the formula, where and It involves guided filtering and bilinear upsampling operations; and These are the predefined radius and regularization coefficients of the guided filter.

8. The text-driven immersive open-scene neural rendering and blending enhancement method according to claim 1, characterized in that: In step four, assuming the background is at infinity, the background has only 3 degrees of freedom in planar rigid motion, therefore its affine matrix is... To indicate: ； in These are the x and y coordinates of the current frame's pixels; The x and y coordinates of the pixels in the previous frame; Let the transformation matrices of two adjacent frames from the camera coordinate system to the world coordinate system be respectively Then, the transformation matrix from the previous frame to the next frame can be calculated: ； In the formula, Let be the rotation matrix for the transformation between the two cameras. Let be the translation vector between the two cameras; then the transformation relationship of pixel coordinates between two adjacent frames can be calculated: ； Where K is the camera's intrinsic parameter matrix, n is the unit vector along the z-axis, and z is the distance from the background plane to the camera; when When the transformation relationship is simplified to: ； Therefore, the solution is: ； Obtain the affine matrix of each adjacent frame. After that, render the initial frame and the first frame in the image. Affine matrix between frames Write it in the following matrix multiplication form: ； in, This is the transformation matrix cropped from the center of the original background image in the first frame; therefore, the pixel coordinates of the background image are multiplied by the affine matrix on the left. Then, bilinear interpolation is used to obtain the first... The background image to be merged in the frame.

9. The text-driven immersive open-scene neural rendering and blending enhancement method according to claim 1, characterized in that: In step five, using the image matching equation, the higher the output pixel value of the sky matte value, the higher the probability that the pixel belongs to the sky background. Therefore, the fused image Y(t) of frame t is represented as: ； In the formula, I(t) is the foreground image of the t-th frame, A(t) is the predicted sky matte value, and B(t) is the background image to be fused.

10. The text-driven immersive open-scene neural rendering and blending enhancement method according to claim 9, characterized in that: By applying the concept of lighting coordination, the foreground's hue and brightness are assigned to the background, ensuring that the foreground and background images have consistent hue and brightness. The correction formula is as follows: ； in, They are respectively and The average brightness These represent the average brightness of the background at a matte prediction value of 1 in the original image and the average brightness of the foreground in the original image, respectively. These are two adjustable parameters, whose physical meanings are the proportion of the original image's foreground brightness and the proportion of the background brightness's influence on the foreground, respectively. These represent the intermediate variables and the final output image in the illumination coordination process, respectively.

Images