A Method and System for Generating 3D Scene Styles Based on Weighted Style Neural Radiation Fields

Abstract

The present application relates to the field of image processing, and proposes a kind of three-dimensional scene style generation method and system based on weighted style neural radiance field, including weighted fusion network, through latent style encoding and pyramid weighted fusion feature decoding, realize the effective capture of latent encoding of style and the fusion weighting of multi-scale feature, adaptively optimize style transfer effect, enhance the multi-aspect performance of generated image in object structure feature preservation, color restoration, style fusion naturalness and multi-view coherence, texture style hidden variable neural radiance field is processed by the network structure combination of basic neural radiance field network, implicit encoding network and merging output network, organically fuses the style features of three-dimensional scene geometry information and style image, then uses picture patch to sample and similarity measure loss function, retain scene texture details and multi-view consistency, the present application improves the quality and detail precision of three-dimensional scene style generation.

Description

Technical Field

[0001] This invention relates to the field of image processing, and in particular to a method and system for generating three-dimensional scene styles based on weighted style neural radiation fields. Background Technology

[0002] With the rapid development of the digital industry, 3D scenes have become an important part of virtual reality, augmented reality, film and television production and game development. They create an immersive experience for users, enhance the realism of interactive experiences, bring users a novel and unique visual experience, meet the growing visual aesthetic needs of the audience, and are a core element in building virtual worlds. Therefore, 3D scene style generation, as a key link in realizing applications, has gradually become an indispensable and important technology.

[0003] In existing technologies, the common approaches are 3D scene style generation based on 2D images and videos, and new perspectives based on 3D scenes. Figure 3 While 3D scene style generation is possible, existing methods struggle to combine complex scene content features with encoded style features when views are sparse. 2D stylization methods, lacking 3D information, can lead to inconsistent rendering results between adjacent views when performing style transfer on neural radiation field rendered images. Furthermore, the stylized images generated by existing methods suffer from poor texture clarity and cross-view issues in multiple perspectives. Figure 1 Insufficient consistency leads to blurry and unnatural 3D style transfer results, making it difficult to maintain cross-view consistency. Figure 1 It accurately reproduces and restores the scene's structural texture.

[0004] Therefore, how to design a 3D scene style generation method that can improve generation quality and detail accuracy while avoiding difficulties in adapting to complex scenes has become an urgent problem to be solved. Summary of the Invention

[0005] Based on this, this invention proposes a 3D scene style generation method and system based on a weighted style neural radiation field. It designs a weighted style neural radiation field based on a dual-path mutual learning multi-view texture consistency architecture, including a weighted fusion network and a texture style latent variable neural radiation field. The weighted fusion network effectively captures the latent style encoding and fuses multi-scale features through latent style encoding and pyramid weighted fusion feature decoding, adaptively optimizing the style transfer effect and enhancing the performance of the generated image in terms of object structure feature preservation, color reproduction, style fusion naturalness, and multi-view coherence. The texture style latent variable neural radiation field organically integrates the geometric information of the 3D scene with the style features of the style image through a combination of network structures including a basic neural radiation field network, an implicit encoding network, and a merged output network. Then, it utilizes image patch sampling and similarity measurement loss functions to preserve scene texture details and multi-view consistency. Figure 1In terms of consistency, this invention improves the quality and detail accuracy of 3D scene style generation.

[0006] This invention proposes a 3D scene style generation method based on weighted style neural radiation fields, comprising:

[0007] The two-dimensional image data and three-dimensional image information of the target three-dimensional scene are input into the weighted style neural radiation field. The weighted style neural radiation field is based on a dual-path mutual learning multi-view texture consistency architecture. The weighted style neural radiation field includes a weighted fusion network and a texture style latent variable neural radiation field.

[0008] The two-dimensional image data is input into the weighted fusion network to obtain stylization results and latent style encoding. The two-dimensional image data includes a rendering image and a style image. The weighted fusion network includes a broad exploration block, an adaptive instance normalization block, and a combined variational autoencoder. The weighted fusion network is based on pyramid weighted fusion feature decoding.

[0009] The three-dimensional image information and the latent style encoding are input into the texture style latent variable neural radiation field to obtain a stylized image. The three-dimensional image information includes the position coordinates and viewpoint direction of the target three-dimensional scene. The texture style latent variable neural radiation field includes a basic neural radiation field network, an implicit encoding network, and a merging output network.

[0010] A final 3D scene style image is generated based on the stylization result and the stylized image, wherein the generated final 3D scene style image is based on image patch sampling.

[0011] In summary, based on the aforementioned method for generating 3D scene styles using a weighted style neural radiation field, a weighted style neural radiation field based on a dual-path mutual learning multi-view texture consistency architecture is designed. This includes a weighted fusion network and a texture style latent variable neural radiation field. The weighted fusion network effectively captures the latent style encoding and fuses multi-scale features through latent style encoding and pyramid weighted fusion feature decoding, adaptively optimizing style transfer performance and enhancing the generated image's performance in terms of object structure feature preservation, color reproduction, style fusion naturalness, and multi-view coherence. The texture style latent variable neural radiation field organically integrates the geometric information of the 3D scene with the style features of the style image through a combination of a basic neural radiation field network, an implicit encoding network, and a merged output network. Finally, image patch sampling and similarity measurement loss functions are used to preserve scene texture details and multi-view consistency. Figure 1This invention improves the quality and detail accuracy of 3D scene style generation. Specifically, it inputs 2D image data and 3D image information of the target 3D scene into a weighted style neural radiation field. This weighted style neural radiation field is based on a dual-path mutual learning multi-view texture consistency architecture and includes a weighted fusion network and a texture style latent variable neural radiation field. The 2D image data, including a rendered image and a style map, is input into the weighted fusion network to obtain stylization results and latent style codes. The weighted fusion network includes a broad exploration block, an adaptive instance normalization block, and a combined variational autoencoder. Based on pyramid weighted fusion feature decoding, the weighted fusion network effectively captures the latent style codes and fuses and weights multi-scale features, adaptively optimizing style transfer. The effect enhances the performance of the generated image in terms of preserving object structure features, color reproduction, style fusion naturalness, and multi-view coherence. The 3D image information and the latent style encoding are input into the texture style latent variable neural radiation field to obtain a stylized image. The 3D image information includes the position coordinates and viewpoint direction of the target 3D scene. The texture style latent variable neural radiation field includes a basic neural radiation field network, an implicit encoding network, and a merging output network, organically integrating the geometric information of the 3D scene with the style features of the style image. Based on the stylization result and the stylized image, a final 3D scene style image is generated. The generation of the final 3D scene style image is based on image patch sampling, preserving scene texture details and multi-view coherence. Figure 1 In terms of consistency, this invention improves the quality and detail accuracy of 3D scene style generation.

[0012] Furthermore, the step of inputting the two-dimensional image data into the weighted fusion network to obtain stylization results and latent style encoding specifically includes:

[0013] Acquire two-dimensional image data of the target three-dimensional scene, wherein the two-dimensional image data includes a rendered image and a style image;

[0014] Feature extraction is performed on the two-dimensional image data according to the VGG model to obtain multiple rendering features and multiple style features respectively. Rendering features and style features of the same scale are grouped into feature pairs to obtain multiple sets of feature pairs of different scales and arranged in descending order of scale. Feature pairs of adjacent scales are connected through a broad exploration block.

[0015] The features are input into a broad exploration block, which includes a first non-isomorphic block, a second non-isomorphic block, and a third non-isomorphic block. The first, second, and third non-isomorphic blocks are in parallel structure. The first and second non-isomorphic blocks are two convolutional blocks of different orthogonal orders. The specific algorithms for the first and second non-isomorphic blocks are as follows:

[0016] ,

[0017] ,

[0018] in, This represents the output characteristics of the first non-isomorphic block. This represents the output characteristics of the second non-isomorphic block. This indicates batch normalization and ReLU activation function processing. and They represent sizes of and convolution, Represents the convolution kernel. This represents the input features of a broad exploration block;

[0019] The third non-isomorphic block is an inline Transformer block, and the specific algorithm for the third non-isomorphic block is as follows:

[0020] ,

[0021] ,

[0022] in, This indicates a cross-self-attention enhancement feature. This indicates cross-self-attention enhancement processing. Presentation layer normalization processing, This represents the output characteristics of the third non-isomorphic block. This indicates convolution processing. This represents the Leaky ReLU activation function;

[0023] The output features of the first non-isomorphic block, the second non-isomorphic block, and the third non-isomorphic block are merged and input into the adaptive instance normalization block and the next-scale broad exploration block, respectively.

[0024] The latent style encoding is obtained by fusing and encoding the output features of the broad exploration block based on the adaptive instance normalization block and the variational autoencoder.

[0025] Stylized results are obtained by decoding the pyramid-weighted fusion features.

[0026] Furthermore, the adaptive instance normalization block specifically includes:

[0027] The weighted fusion network includes multiple adaptive instance normalization blocks. Each single-scale feature pair has a unique corresponding adaptive instance normalization block. The transmission order of the adaptive instance normalization blocks corresponds to the scale order of the feature pair. The transmission order of the adaptive instance normalization blocks is unidirectional linear transmission. The adaptive instance normalization blocks are used to fuse the output features of the broad exploration block.

[0028] An adaptive instance normalization block is used to obtain the total fusion output features and input them into a combination of variational autoencoders to obtain latent style codes, which include the mean vector and covariance vector of a Gaussian distribution.

[0029] The loss of the latent style encoding is as follows:

[0030] ,

[0031] in, This represents the loss of latent style encoding. , Represents the position coordinates of a vector. Indicates the location Vector style encoding, , This represents the mean and variance.

[0032] Furthermore, the pyramid-weighted fusion feature decoding specifically includes:

[0033] The similarity of features at different scales is calculated based on Euclidean distance. The specific algorithm for calculating the similarity of features at different scales based on Euclidean distance is as follows:

[0034] ,

[0035] ,

[0036] in, Indicates similarity. , These represent feature maps at different scales. Indicates the number of pixels. , Represents the position coordinates of a vector. In the feature map The middle position is The vector, In the feature map Zhongyu The nearest vector, the vector position is , Indicates the ordinal number of the feature scale. This represents the nearest neighbor feature matching loss;

[0037] Weights are assigned to features at different scales and the features are decomposed hierarchically to obtain hierarchical decoding features. The specific algorithm for obtaining hierarchical decoding features is as follows:

[0038] ,

[0039] ,

[0040] in, The weights representing features at different scales. This represents the hierarchical decoding features at the current feature scale. Representing feature pairs at different scales This represents the convolution operation. Indicates upsampling, This represents the hierarchical decoding features of the previous feature scale;

[0041] Stylization results are obtained from the convolutional projection head.

[0042] Furthermore, the step of inputting the three-dimensional image information and the latent style encoding into the texture style latent variable neural radiation field to obtain a stylized image specifically includes:

[0043] Three-dimensional image information and latent style codes are input into the neural radiation field of texture style latent variables. The three-dimensional image information includes position coordinates and view direction. The position coordinates are converted into position codes.

[0044] The location encoding and the latent style encoding are input into an implicit encoding network to obtain a style vector. The implicit encoding network is constructed from multiple stacked fully connected layers, and each fully connected layer performs a linear transformation operation according to a weight matrix.

[0045] The position code and the view direction are input into a basic neural radiation field network to generate volume density information and volume density features of spatial sampling points. The basic neural radiation field network is based on a multilayer perceptron.

[0046] The style vector, volume density feature, positional encoding, and latent style encoding are input to and then output to the network to obtain color information of spatial sampling points.

[0047] A volume rendering operation is performed based on the volume density information and the color information to obtain a stylized image.

[0048] Furthermore, the merged output network specifically includes:

[0049] The merged output network includes multiple linear layers, which are unidirectional linear sequences. These linear layers perform fusion and transformation processing on style vectors, volume density features, positional encoding, and latent style encoding.

[0050] Furthermore, the step of generating the final 3D scene style image based on the stylization result and the stylized image specifically includes:

[0051] Volume rendering is performed based on volume density and color information to obtain a stylized image. The specific algorithm for volume rendering is as follows:

[0052] ,

[0053] in, Representing stylized images, Indicates light, Indicates the number of spatial sampling points. Indicates the ordinal number of the spatial sampling point. Indicates the ordinal number of the previous spatial sampling point. , These represent the volume density information of the current spatial sampling point and the previous spatial sampling point, respectively. , These represent the distances between adjacent sampling points of the current spatial sampling point and the previous spatial sampling point, respectively. This indicates the color information of the current spatial sampling point;

[0054] Style transfer processing is performed on the sampled image based on image patches, wherein the image patch sampling is based on image slices of the rendered image;

[0055] The stylized image and the stylization result are mutually learned and optimized to obtain the final 3D scene style image.

[0056] This invention proposes a 3D scene style generation system based on weighted style neural radiation fields, comprising:

[0057] The input module is used to input the two-dimensional image data and three-dimensional image information of the target three-dimensional scene into the weighted style neural radiation field. The weighted style neural radiation field is based on a dual-path mutual learning multi-view texture consistency architecture. The weighted style neural radiation field includes a weighted fusion network and a texture style latent variable neural radiation field.

[0058] The weighted fusion network module is used to input the two-dimensional image data into the weighted fusion network to obtain stylization results and latent style encoding. The two-dimensional image data includes a rendering map and a style map. The weighted fusion network includes a broad exploration block, an adaptive instance normalization block, and a combined variational autoencoder. The weighted fusion network is based on pyramid weighted fusion feature decoding.

[0059] The neural radiation field network module is used to input the three-dimensional image information and the latent style encoding into the texture style latent variable neural radiation field to obtain a stylized image. The three-dimensional image information includes the position coordinates and viewpoint direction of the target three-dimensional scene. The texture style latent variable neural radiation field includes a basic neural radiation field network, an implicit encoding network, and a merging output network.

[0060] The mutual learning module is used to generate a final 3D scene style image based on the stylization result and the stylized image, wherein the generation of the final 3D scene style image is based on image patch sampling.

[0061] The present invention also provides a storage medium that stores one or more programs, which, when executed by a processor, implement the three-dimensional scene style generation method based on weighted style neural radiation fields as described above.

[0062] The present invention also provides a computer device, the computer device including a memory and a processor, wherein:

[0063] The memory is used to store computer programs;

[0064] When the processor executes the computer program stored in the memory, it implements the three-dimensional scene style generation method based on weighted style neural radiation field as described above. Attached Figure Description

[0065] Figure 1 This is a flowchart of the three-dimensional scene style generation method based on weighted style neural radiation field proposed in the first embodiment of the present invention;

[0066] Figure 2 This is a schematic diagram of the structure of the three-dimensional scene style generation system based on weighted style neural radiation field proposed in the second embodiment of the present invention;

[0067] Figure 3 This is a structural diagram of the weighted fusion network proposed in this invention;

[0068] Figure 4 This is a structural diagram of the broad exploration block proposed in this invention;

[0069] Figure 5 This is a structural diagram of the implicit coding network proposed in this invention;

[0070] Figure 6 This is a structural diagram of the merged output network proposed in this invention.

[0071] The following detailed description, in conjunction with the accompanying drawings, will further illustrate the present invention. Detailed Implementation

[0072] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0073] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0075] Please see Figure 1 The diagram shows a flowchart of a three-dimensional scene style generation method based on a weighted style neural radiation field proposed in the first embodiment of the present invention. This three-dimensional scene style generation method based on a weighted style neural radiation field includes steps S01 to S04, wherein:

[0076] Step S01: Input the two-dimensional image data and three-dimensional image information of the target three-dimensional scene into the weighted style neural radiation field;

[0077] It should be noted that in this embodiment, the weighted style neural radiation field is based on a dual-path mutual learning multi-view texture consistency architecture, and the weighted style neural radiation field includes a weighted fusion network and a texture style latent variable neural radiation field.

[0078] Step S02: Input the two-dimensional image data into the weighted fusion network to obtain stylization results and latent style encoding;

[0079] It should be noted that in this embodiment, the two-dimensional image data includes a rendered image and a style image, and the weighted fusion network includes a broad exploration block, an adaptive instance normalization block, and a combined variational autoencoder. For the specific structure of the weighted fusion network, please refer to [link to documentation]. Figure 3 Where AdaIN represents the adaptive instance normalization block, and VAE represents the combined variational autoencoder. , , , The output features represent the adaptive instance normalization blocks of feature pairs at different scales. Representing latent style encoding, the weighted fusion network is based on pyramid weighted fusion feature decoding to obtain two-dimensional image data of the target three-dimensional scene, the two-dimensional image data including rendering map and style map;

[0080] Feature extraction is performed on the two-dimensional image data according to the VGG model to obtain multiple rendering features and multiple style features respectively. Rendering features and style features of the same scale are grouped into feature pairs to obtain multiple sets of feature pairs of different scales and arranged in descending order of scale. Feature pairs of adjacent scales are connected through a broad exploration block.

[0081] The aforementioned features are applied to the input broad exploration block. For the specific structure of the broad exploration block, please refer to [link / reference needed]. Figure 4 The broad exploration block includes a first non-isomorphic block, a second non-isomorphic block, and a third non-isomorphic block. The first non-isomorphic block, the second non-isomorphic block, and the third non-isomorphic block are in parallel structure. The first non-isomorphic block and the second non-isomorphic block are two convolutional blocks of different orthogonal orders. The specific algorithms for the first non-isomorphic block and the second non-isomorphic block are as follows:

[0082] ,

[0083] ,

[0084] in, This represents the output characteristics of the first non-isomorphic block. This represents the output characteristics of the second non-isomorphic block. This indicates batch normalization and ReLU activation function processing. and They represent sizes of and convolution, Represents the convolution kernel. This represents the input features of a broad exploration block;

[0085] The third non-isomorphic block is an inline Transformer block, and the specific algorithm for the third non-isomorphic block is as follows:

[0086] ,

[0087] ,

[0088] in, This indicates a cross-self-attention enhancement feature. This indicates cross-self-attention enhancement processing. Presentation layer normalization processing, This represents the output characteristics of the third non-isomorphic block. This indicates convolution processing. This represents the Leaky ReLU activation function;

[0089] The output features of the first non-isomorphic block, the second non-isomorphic block, and the third non-isomorphic block are merged and input into the adaptive instance normalization block and the next-scale broad exploration block, respectively.

[0090] The broad exploration block incorporates a multi-layered feature coloring network architecture, which dynamically assigns weights to spatial and channel dimensions, enhancing the model's ability to recognize complex patterns. In the processing steps of the broad exploration block, convolution is used to reduce the number of channels to reduce computational complexity. Contextual semantic information is utilized to expand the receptive field analysis, process and integrate features at different levels, extract and integrate information at different perceptual levels, and improve the quality of style generation.

[0091] The latent style encoding is obtained by fusing and encoding the output features of the broad exploration block based on the adaptive instance normalization block and the variational autoencoder.

[0092] Stylization results are obtained by decoding the pyramid-weighted fusion features.

[0093] Pyramid-weighted fusion feature decoding integrates features at different resolutions, enabling the model to consider both global semantics and local details. However, in style transfer, relying solely on the highest-level features is not advisable. While it can represent the overall style, it lacks detail and cannot accurately reproduce micro-style elements. It can also cause style fusion inconsistencies, making it difficult for the model to adapt to images with different content. Therefore, this embodiment uses Euclidean distance to measure the similarity of features at different scales. Based on the estimated Euclidean distance between corresponding feature vectors, it emphasizes the reliable parts with smaller stable distances and reduces the unreliable parts with larger distances, assigning corresponding weights to features at different scales. Through this weighted fusion method, multi-scale features can be integrated more effectively, improving the effect and quality of style transfer.

[0094] The weighted fusion network includes multiple adaptive instance normalization blocks. Each single-scale feature pair has a unique corresponding adaptive instance normalization block. The transmission order of the adaptive instance normalization blocks corresponds to the scale order of the feature pair. The transmission order of the adaptive instance normalization blocks is unidirectional linear transmission. The adaptive instance normalization blocks are used to fuse the output features of the broad exploration block.

[0095] An adaptive instance normalization block is used to obtain the total fusion output features and input them into a combination of variational autoencoders to obtain latent style codes, which include the mean vector and covariance vector of a Gaussian distribution.

[0096] The loss of the latent style encoding is as follows:

[0097] ,

[0098] in, This represents the loss of latent style encoding. , Represents the position coordinates of a vector. Indicates the location Vector style encoding, , Indicates the mean and variance;

[0099] The similarity of features at different scales is calculated based on Euclidean distance. The specific algorithm for calculating the similarity of features at different scales based on Euclidean distance is as follows:

[0100] ,

[0101] ,

[0102] in, Indicates similarity. , These represent feature maps at different scales. Indicates the number of pixels. , Represents the position coordinates of a vector. In the feature map The middle position is The vector, In the feature map Zhongyu The nearest vector, the vector position is , Indicates the ordinal number of the feature scale. This represents the nearest neighbor feature matching loss;

[0103] Weights are assigned to features at different scales and the features are decomposed hierarchically to obtain hierarchical decoding features. The specific algorithm for obtaining hierarchical decoding features is as follows:

[0104] ,

[0105] ,

[0106] in, The weights representing features at different scales. This represents the hierarchical decoding features at the current feature scale. Representing feature pairs at different scales This represents the convolution operation. Indicates upsampling, This represents the hierarchical decoding features of the previous feature scale;

[0107] Stylization results are obtained from the convolutional projection head.

[0108] Step S03: Input the 3D image information and latent style encoding into the texture style latent variable neural radiation field to obtain a stylized image;

[0109] It should be noted that in this embodiment, the three-dimensional image information includes the position coordinates and viewpoint direction of the target three-dimensional scene, and the texture style latent variable neural radiation field includes a basic neural radiation field network, an implicit coding network, and a merging output network. For the specific structure of the implicit coding network, please refer to [link to documentation]. Figure 5 For the specific structure of the merged output network, please refer to [link / reference]. Figure 6 The three-dimensional image information and latent style code are input into the texture style latent variable neural radiation field. The three-dimensional image information includes position coordinates and view direction. The position coordinates are converted into position codes.

[0110] The location encoding and the latent style encoding are input into an implicit encoding network to obtain a style vector. The implicit encoding network is constructed from multiple stacked fully connected layers, and each fully connected layer performs a linear transformation operation according to a weight matrix.

[0111] This nonlinear transformation allows the network to learn more complex feature maps, thereby better capturing and representing the deep abstraction and complex patterns of style features. Through this structure, the implicit style coding network can fuse and transform multi-source information of the input to generate style vectors with specific style semantics, which play a key role in the subsequent stylization process to guide and adjust the style performance of the scene and enhance the expressiveness of the latent encoding.

[0112] The position code and the view direction are input into a basic neural radiation field network to generate volume density information and volume density features of spatial sampling points. The basic neural radiation field network is based on a multilayer perceptron.

[0113] The style vector, volume density feature, positional encoding, and latent style encoding are input to and then output to the network to obtain color information of spatial sampling points.

[0114] Perform volume rendering based on the volume density information and the color information to obtain a stylized image;

[0115] The merged output network includes multiple linear layers, which are unidirectional linear sequences. These linear layers perform fusion and transformation processing on style vectors, volume density features, positional encoding, and latent style encoding.

[0116] The modeling of the stylized scene is compensated and corrected to correct local color deviations and guide the model to learn a smoother and more natural visual presentation effect. After multi-layer processing, the output network is merged to generate color information. This color information is combined with the volume density information generated by the basic neural radiation field network and applied to the subsequent volume rendering stage to generate a 3D scene image with a specific style, thus achieving the transformation from input features to final stylized visual output.

[0117] Step S04: Generate the final 3D scene style image based on the stylization results and the stylized image;

[0118] It should be noted that in this embodiment, the generation of the final 3D scene style image is based on image patch sampling, and volume rendering is performed according to volume density information and color information to obtain a stylized image. The specific algorithm for the volume rendering operation is as follows:

[0119] ,

[0120] in, Representing stylized images, Indicates light, Indicates the number of spatial sampling points. Indicates the ordinal number of the spatial sampling point. Indicates the ordinal number of the previous spatial sampling point. , These represent the volume density information of the current spatial sampling point and the previous spatial sampling point, respectively. , These represent the distances between adjacent sampling points of the current spatial sampling point and the previous spatial sampling point, respectively. This indicates the color information of the current spatial sampling point;

[0121] Style transfer processing is performed on the sampled image based on image patches, wherein the image patch sampling is based on image slices of the rendered image;

[0122] The stylized image and the stylized result are mutually learned and optimized to obtain the final 3D scene style image;

[0123] The cosine similarity and structural similarity between the generated final 3D scene style image and the original rendering image are compared, and the similarity loss is calculated. The similarity loss captures the coherence and geometric similarity information between viewpoints, continuously guiding the stylized image to move closer to the texture and geometric structure of the content image. Specifically, this section calculates the cosine similarity and structural similarity of each image patch between consecutive frames of the generated final 3D scene style image and the original rendering image, and then calculates the L2 norm between the two sets of cosine similarity and structural similarity to define the similarity loss.

[0124] In summary, based on the aforementioned method for generating 3D scene styles using a weighted style neural radiation field, a weighted style neural radiation field based on a dual-path mutual learning multi-view texture consistency architecture is designed. This includes a weighted fusion network and a texture style latent variable neural radiation field. The weighted fusion network effectively captures the latent style encoding and fuses multi-scale features through latent style encoding and pyramid weighted fusion feature decoding, adaptively optimizing style transfer performance and enhancing the generated image's performance in terms of object structure feature preservation, color reproduction, style fusion naturalness, and multi-view coherence. The texture style latent variable neural radiation field organically integrates the geometric information of the 3D scene with the style features of the style image through a combination of a basic neural radiation field network, an implicit encoding network, and a merged output network. Finally, image patch sampling and similarity measurement loss functions are used to preserve scene texture details and multi-view consistency. Figure 1 This invention improves the quality and detail accuracy of 3D scene style generation. Specifically, it inputs 2D image data and 3D image information of the target 3D scene into a weighted style neural radiation field. This weighted style neural radiation field is based on a dual-path mutual learning multi-view texture consistency architecture and includes a weighted fusion network and a texture style latent variable neural radiation field. The 2D image data, including a rendered image and a style map, is input into the weighted fusion network to obtain stylization results and latent style codes. The weighted fusion network includes a broad exploration block, an adaptive instance normalization block, and a combined variational autoencoder. Based on pyramid weighted fusion feature decoding, the weighted fusion network effectively captures the latent style codes and fuses and weights multi-scale features, adaptively optimizing style transfer. The effect enhances the performance of the generated image in terms of preserving object structure features, color reproduction, style fusion naturalness, and multi-view coherence. The 3D image information and the latent style encoding are input into the texture style latent variable neural radiation field to obtain a stylized image. The 3D image information includes the position coordinates and viewpoint direction of the target 3D scene. The texture style latent variable neural radiation field includes a basic neural radiation field network, an implicit encoding network, and a merging output network, organically integrating the geometric information of the 3D scene with the style features of the style image. Based on the stylization result and the stylized image, a final 3D scene style image is generated. The generation of the final 3D scene style image is based on image patch sampling, preserving scene texture details and multi-view coherence. Figure 1 In terms of consistency, this invention improves the quality and detail accuracy of 3D scene style generation.

[0125] Please see Figure 2 The figure shows a schematic diagram of the structure of the 3D scene style generation system based on weighted style neural radiation field proposed in the second embodiment of the present invention. The system includes:

[0126] Input module 10 is used to input two-dimensional image data and three-dimensional image information of the target three-dimensional scene into a weighted style neural radiation field. The weighted style neural radiation field is based on a dual-path mutual learning multi-view texture consistency architecture. The weighted style neural radiation field includes a weighted fusion network and a texture style latent variable neural radiation field.

[0127] The weighted fusion network module 20 is used to input the two-dimensional image data into the weighted fusion network to obtain stylization results and latent style encoding. The two-dimensional image data includes a rendering map and a style map. The weighted fusion network includes a broad exploration block, an adaptive instance normalization block, and a combined variational autoencoder. The weighted fusion network is based on pyramid weighted fusion feature decoding.

[0128] The neural radiation field network module 30 is used to input the three-dimensional image information and the latent style encoding into the texture style latent variable neural radiation field to obtain a stylized image. The three-dimensional image information includes the position coordinates and view direction of the target three-dimensional scene. The texture style latent variable neural radiation field includes a basic neural radiation field network, an implicit encoding network, and a merged output network.

[0129] The mutual learning module 40 is used to generate a final 3D scene style image based on the stylization result and the stylized image, wherein the generation of the final 3D scene style image is based on image patch sampling.

[0130] The present invention also proposes a computer storage medium storing one or more programs that, when executed by a processor, implement the above-described method for generating three-dimensional scene styles based on weighted style neural radiation fields.

[0131] The present invention also proposes a computer device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to implement the above-described method for generating three-dimensional scene styles based on weighted style neural radiation fields.

[0132] Those skilled in the art will understand that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can mean any means that can contain stored, communicated, propagated, or transmitted programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0133] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0134] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0135] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0136] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A method for generating a three-dimensional scene style based on a weighted style neural radiance field, characterized in that, The method comprises the steps of: inputting two-dimensional image data and three-dimensional image information of a target three-dimensional scene into a weighted style neural radiance field, the weighted style neural radiance field being based on a two-way mutual learning multi-view texture consistency architecture, the two-way mutual learning multi-view texture consistency architecture comprising a weighted fusion network and a texture style hidden variable neural radiance field; inputting the two-dimensional image data into the weighted fusion network to obtain a stylized result and a latent style code, the two-dimensional image data comprising a rendering image and a style image, the weighted fusion network comprising a broad exploration block, an adaptive instance normalization block and a combined variational autoencoder, the weighted fusion network being based on pyramid weighted fusion feature decoding; inputting the three-dimensional image information and the latent style code into the texture style hidden variable neural radiance field to obtain a stylized image, the three-dimensional image information comprising position coordinates and a viewing direction of the target three-dimensional scene, the texture style hidden variable neural radiance field comprising a basic neural radiance field network, an implicit encoding network and a merging output network; the step of inputting the three-dimensional image information and the latent style code into the texture style hidden variable neural radiance field to obtain a stylized image specifically comprises: inputting three-dimensional image information and a latent style code into a texture style hidden variable neural radiance field, the three-dimensional image information comprising position coordinates and a viewing direction, and converting the position coordinates into position encoding; inputting the position encoding and the latent style code into an implicit encoding network to obtain a style vector, the implicit encoding network being constructed by a plurality of superimposed fully connected layers, each of the fully connected layers performing a linear conversion operation according to a weight matrix; inputting the position encoding and the viewing direction into a basic neural radiance field network to generate volume density information and volume density features of spatial sampling points, the basic neural radiance field network being based on a multi-layer perceptron; inputting the style vector, the volume density features, the position encoding and the latent style code into a merging output network to obtain color information of the spatial sampling points; performing volume rendering operation according to the volume density information and the color information to obtain a stylized image; generating a final three-dimensional scene style image according to the stylized result and the stylized image, the generation of the final three-dimensional scene style image being based on picture patch pair sampling.

2. The method of claim 1, wherein, the step of inputting the two-dimensional image data into the weighted fusion network to obtain a stylized result and a latent style code specifically comprises: obtaining two-dimensional image data of a target three-dimensional scene, the two-dimensional image data comprising a rendering image and a style image; performing feature extraction on the two-dimensional image data according to a VGG model to obtain a plurality of rendering features and a plurality of style features respectively, grouping rendering features and style features of the same scale into a feature pair, obtaining a plurality of feature pairs of different scales and arranging them in descending order of scale, and connecting feature pairs of adjacent scales through a broad exploration block; The feature pair is input into a wide exploration block, the wide exploration block includes a first non-homogeneous block, a second non-homogeneous block and a third non-homogeneous block, the first non-homogeneous block, the second non-homogeneous block and the third non-homogeneous block are parallel structures, the first non-homogeneous block and the second non-homogeneous block are two different orthogonal order convolution blocks, and the specific algorithm of the first non-homogeneous block and the second non-homogeneous block is as follows: ， ， wherein, represents the output feature of the first non-homogeneous block, represents the output feature of the second non-homogeneous block, represents the batch normalization and ReLU activation function processing, and respectively represent a convolution of size and , represents a convolution kernel, represents the input feature of the broad exploration block; The third non-homogeneous block is an inline Transformer block, and the specific algorithm of the third non-homogeneous block is as follows: ， ， wherein, denotes cross self-attention enhanced features, denotes cross self-attention enhanced processing, denotes layer normalization processing, denotes output features of the third non-isomorphic block, denotes convolution processing, denotes a Leaky ReLU activation function; The output features of the first non-homogeneous block, the output features of the second non-homogeneous block and the output features of the third non-homogeneous block are merged to be input into an adaptive instance normalization block and a wide exploration block of the next scale respectively; The output features of the wide exploration block are fused and encoded according to the adaptive instance normalization block and a combined variational autoencoder to obtain a latent style code; The style result is obtained by decoding the pyramid weighted fusion feature.

3. The method of claim 2, wherein, The adaptive instance normalization block specifically includes: The weighted fusion network includes a plurality of adaptive instance normalization blocks, each single-scale feature pair has a unique corresponding adaptive instance normalization block, the transmission order of the adaptive instance normalization block corresponds to the scale order of the feature pair, the transmission order of the adaptive instance normalization block is a one-way linear transmission, and the adaptive instance normalization block is used for fusing the output features of the wide exploration block; The adaptive instance normalization block obtains a total fusion output feature and inputs the total fusion output feature into the combined variational autoencoder to obtain a latent style code, and the latent style code includes a mean vector and a covariance vector of a Gaussian distribution; The loss of the latent style code is specifically as follows: ， wherein, denotes the loss of the latent style encoding, , denotes the vector position coordinate, denotes the style encoding of the vector at position , , denotes the mean and variance.

4. The method of claim 2, wherein, The pyramid weighted fusion feature decoding specifically includes: The similarity of different scale features is calculated according to the Euclidean distance, and the specific algorithm of calculating the similarity of different scale features according to the Euclidean distance is as follows: ， ， wherein, denotes similarity, , denote feature maps of different scales respectively, denotes the number of pixels, , denotes the vector position coordinates, denotes the vector in the feature map with position , denotes the vector in the feature map closest to with vector position , denotes the feature scale ordinal, denotes the nearest neighbor feature matching loss; The different scale features are assigned weights and hierarchically decomposed to obtain hierarchical decoding features, and the specific algorithm of obtaining the hierarchical decoding features is as follows: ， ， wherein, denotes a weight of different scale features, denotes a hierarchically decoded feature of a current feature scale, denotes a feature pair of different scales, denotes a convolution operation, denotes an up-sampling, denotes a hierarchically decoded feature of a previous feature scale; The style result is obtained according to the convolution projection head.

5. The method of claim 1, wherein, The merging output network specifically includes: The merging output network includes a plurality of linear layers, the linear layers are a one-way linear sequence, and the linear layers fuse and change the style vector, the volume density feature, the position encoding and the latent style code.

6. The method of claim 1, wherein, The step of generating a final three-dimensional scene style image according to the style result and the stylized image specifically includes: Body rendering operation is performed according to the volume density information and the color information to obtain a stylized image, and the specific algorithm of the body rendering operation is as follows: ， wherein, represents a stylized image, represents a light ray, represents a number of spatial sample points, represents an ordinal of a spatial sample point, represents an ordinal of a previous spatial sample point, , represent volumetric density information of the current spatial sample point and the previous spatial sample point, respectively, , represent neighboring sample point distances of the current spatial sample point and the previous spatial sample point, respectively, represents color information of the current spatial sample point; The picture patch-based sampling is subjected to style transfer processing, and the picture patch-based sampling is based on a picture slice of a rendering map; The final three-dimensional scene style image is obtained through mutual learning optimization of the stylized image and the style result.

7. A three-dimensional scene style generation system based on weighted style neural radiance fields, characterized in that, It includes: An input module is configured to input two-dimensional image data and three-dimensional image information of a target three-dimensional scene into a weighted style neural radiance field, and the weighted style neural radiance field is based on a double-path mutual learning multi-view texture consistency architecture, and the double-path mutual learning multi-view texture consistency architecture includes a weighted fusion network and a texture style hidden variable neural radiance field. The weighted fusion network module is configured to input the two-dimensional image data into the weighted fusion network to obtain a stylized result and a latent style code, the two-dimensional image data including a rendering image and a style image, the weighted fusion network including a broad exploration block, an adaptive instance normalization block, and a combined variational autoencoder, and the weighted fusion network is based on pyramid weighted fusion feature decoding; The neural radiance field network module is configured to input the three-dimensional image information and the latent style code into the texture style latent variable neural radiance field to obtain a stylized image, the three-dimensional image information including position coordinates and a viewing direction of a target three-dimensional scene, and the texture style latent variable neural radiance field including a basic neural radiance field network, an implicit encoding network, and a merging output network. The step of inputting the three-dimensional image information and the latent style code into the texture style latent variable neural radiance field to obtain a stylized image specifically includes: inputting three-dimensional image information and a latent style code into a texture style latent variable neural radiance field, the three-dimensional image information including position coordinates and a viewing direction, and converting the position coordinates into position encoding; inputting the position encoding and the latent style code into an implicit encoding network to obtain a style vector, the implicit encoding network being constructed by a plurality of superimposed fully connected layers, and each of the fully connected layers performing linear conversion operation according to a weight matrix; inputting the position encoding and the viewing direction into a basic neural radiance field network to generate volume density information and volume density features of spatial sampling points, the basic neural radiance field network being based on a multi-layer perceptron; inputting the style vector, the volume density features, the position encoding, and the latent style code into a merging output network to obtain color information of the spatial sampling points; performing volume rendering operation according to the volume density information and the color information to obtain a stylized image; The mutual learning module is configured to generate a final three-dimensional scene style image according to the stylized result and the stylized image, and the generation of the final three-dimensional scene style image is based on picture patch pair sampling.

8. A storage medium, characterized by The storage medium stores one or more programs, which are executed by the processor to implement the three-dimensional scene style generation method based on a weighted style neural radiance field according to any one of claims 1-6.

9. A computer device, comprising: The computer device includes a memory and a processor, wherein: The memory is configured to store a computer program; The processor is configured to execute the computer program stored in the memory to implement the three-dimensional scene style generation method based on a weighted style neural radiance field according to any one of claims 1-6.

Images