Text-driven 3D scene stylization method, apparatus, device and storage medium

Abstract

The present application relates to the technical field of 3D scene stylization, and particularly relates to a text-driven 3D scene stylization method, device, equipment and storage medium, the method first constructs a mixed radial basis field representation architecture, then constructs an initial style radial basis function model based on the mixed radial basis field representation architecture, performs multi-round iteration training on the initial style radial basis function model, obtains a style radial basis function model, then obtains a multi-dimensional loss, fine-tunes the style radial basis function model based on the multi-dimensional loss, obtains a target style radial basis function model, finally obtains a text instruction, drives the target style radial basis function model to generate a 3D scene stylization rendering result based on the text instruction, and aims to solve the problems of blur artifacts and uneven style coverage in the existing text-guided 3D scene stylization technology, and realize consistent across perspectives, artifact-free, high-fidelity and uniform style 3D scene stylization.

Description

Technical Field

[0001] This invention relates to the field of 3D scene stylization technology, and in particular to a text-driven 3D scene stylization method, apparatus, device and storage medium. Background Technology

[0002] In the field of digital content creation, intuitive and consistent 3D content generation across different perspectives has always been a core requirement, and the emergence of NeRF (Neural Radiation Field) editing technology provides a feasible path to this goal. This technology learns the radiation field information of a scene to achieve text-guided 3D scene editing, potentially breaking through the limitations of complex and inefficient traditional 3D creation processes, and has broad application prospects in film and television production, game development, virtual simulation, and many other fields.

[0003] However, existing NeRF-based text-guided editing solutions still have significant limitations, making it difficult to balance creative flexibility with effect stability. One type of solution locks the underlying geometric structure of the scene, limiting creative adjustments during the stylization process; another type of solution, while allowing geometry to move to adapt to stylistic requirements, is prone to serious technical problems, not only producing blur artifacts and compromising visual fidelity, but also exhibiting uneven style coverage, resulting in some areas of the scene lacking or over-stylizing effects, affecting overall consistency. Summary of the Invention

[0004] In order to overcome the shortcomings of the prior art, the present invention aims to provide a text-driven 3D scene stylization method, apparatus, device and storage medium, which aims to solve the problems of blurry artifacts and uneven style coverage in the existing text-guided 3D scene stylization technology, and realize 3D scene stylization that is consistent across viewpoints, free of artifacts, high-fidelity and uniform in style.

[0005] The first aspect of this invention provides a text-driven 3D scene stylization method, comprising: constructing a hybrid radial basis field representation architecture; constructing an initial style radial basis function model based on the hybrid radial basis field representation architecture; performing multiple rounds of iterative training on the initial style radial basis function model to obtain a style radial basis function model; obtaining a multi-dimensional loss, and fine-tuning the style radial basis function model based on the multi-dimensional loss to obtain a target style radial basis function model; obtaining text instructions, and driving the target style radial basis function model to generate a 3D scene stylization rendering result based on the text instructions.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the construction of the hybrid radial basis field representation architecture includes: obtaining a set of learnable parameters; defining an adaptive radial basis function kernel set based on the set of learnable parameters; performing kernel capacity expansion processing on the adaptive radial basis function kernel set using a multi-frequency sine synthesis algorithm to obtain an optimized radial basis function kernel set; constructing a coarse-grid backbone network, wherein the coarse-grid backbone network and the optimized radial basis function kernel set form complementary feature extraction branches, and setting a neural network decoder to complete the construction of the hybrid radial basis field representation architecture.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the learnable parameter set includes position and shape; the construction of the initial style radial basis function model based on the hybrid radial basis field representation architecture includes: acquiring multi-view images of the actual scene and a preset weight allocation rule; acquiring multiple initial 3D sampling points based on the multi-view images of the actual scene, and assigning weights to each initial 3D sampling point based on the weight allocation rule to obtain weighted 3D sampling points; performing clustering processing on the weighted 3D sampling points using a clustering algorithm to obtain multiple cluster centers; initializing the position and shape of the optimized radial basis function kernel set based on the multiple cluster centers to construct the initial style radial basis function model.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the learnable parameter set further includes feature weights. The step of performing multiple rounds of iterative training on the initial style radial basis function model to obtain the style radial basis function model includes: obtaining camera lighting data based on multi-view image sampling of the actual scene, and acquiring actual color data corresponding to the camera lighting data; inputting the camera lighting data into the initial style radial basis function model for volume rendering to obtain predicted color data; calculating the model reconstruction loss based on the actual color data and the predicted color data; iteratively updating the feature weights of the optimized radial basis function kernel set of the initial style radial basis function model using a backpropagation algorithm based on the model reconstruction loss; and outputting the style radial basis function model when a preset iteration stopping condition is met.

[0009] Optionally, in the fourth implementation of the first aspect of the present invention, the step of obtaining the multi-dimensional loss and fine-tuning the style radial basis function model based on the multi-dimensional loss to obtain the target style radial basis function model includes: inputting the camera lighting data into the style radial basis function model for volume rendering to obtain stylized predicted color data, and performing stitching processing on the stylized predicted color data to obtain a stylized scene image; obtaining the target text corresponding to the stylized scene image and the fixed text corresponding to the actual scene multi-view image; calling the pre-trained CLIP (Contrastive Language-Image Pretraining) model to perform multi-dimensional loss calculation based on the stylized scene image, the actual scene multi-view image, the target text, and the fixed text to obtain the multi-dimensional loss; using the backpropagation algorithm to iteratively update the feature weights of the pre-trained radial basis function kernel group of the style radial basis function model based on the multi-dimensional loss; and outputting the target style radial basis function model when a preset iteration stopping condition is met.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the CLIP model includes a text encoder, an image encoder, and a multi-dimensional loss calculation module, wherein the text encoder and the image encoder are respectively connected to the multi-dimensional loss calculation module; the step of calling the pre-trained CLIP model to perform multi-dimensional loss calculation based on the stylized scene image, the actual scene multi-view image, the target text, and the fixed text to obtain the multi-dimensional loss includes: obtaining a pre-trained radial basis function kernel set based on the stylized radial basis function model, performing camera plane projection processing on the pre-trained radial basis function kernel set to obtain an elliptical coverage region, and generating a scene saliency map based on the elliptical coverage region; constructing a bias sampling rule based on the scene saliency map, randomly sampling the stylized scene image based on the bias sampling rule to obtain a set of stylized local image patches, and then... The process involves: acquiring a set of original local image patches corresponding to the stylized local image patch set from multi-view images of the actual scene; performing text encoding on the target text and the fixed text based on the text encoder to obtain target text embedding vectors and fixed text embedding vectors; performing image encoding on the stylized scene image, the multi-view image of the actual scene, the set of stylized local image patches, and the set of original local image patches based on the image encoder to obtain stylized image embedding vectors, original image embedding vectors, stylized image patch embedding vectors, and original image patch embedding vectors; and performing multi-dimensional loss calculation based on the target text embedding vector, the fixed text embedding vector, the stylized image embedding vector, the original image embedding vector, the stylized image patch embedding vector, and the original image patch embedding vector to obtain the multi-dimensional loss.

[0011] Optionally, in a sixth implementation of the first aspect of the present invention, the multi-dimensional loss calculation module includes a relative directional loss calculation submodule, a global contrast loss calculation submodule, a local contrast loss calculation submodule, and an auxiliary loss calculation submodule. The relative directional loss calculation submodule, the global contrast loss calculation submodule, and the local contrast loss calculation submodule are respectively connected to the text encoder and the image encoder, and the auxiliary loss calculation submodule is connected to the image encoder. The step of performing multi-dimensional loss calculation based on the target text embedding vector, the fixed text embedding vector, the stylized image embedding vector, the original image embedding vector, the stylized image patch embedding vector, and the original image patch embedding vector to obtain the multi-dimensional loss includes: based on the relative directional loss calculation... The operator module calculates the relative directionality loss based on the target text embedding vector, the fixed text embedding vector, the stylized image embedding vector, and the original image embedding vector; the global contrast loss calculation submodule calculates the global contrast loss based on the target text embedding vector, the stylized image embedding vector, and the original image embedding vector; the local contrast loss calculation submodule calculates the local contrast loss based on the stylized image patch embedding vector and the original image patch embedding vector; the auxiliary loss calculation submodule calculates the auxiliary loss based on the stylized scene image and the actual scene multi-view image; a preset weight coefficient setting rule is obtained, and the relative directionality loss, the global contrast loss, the local contrast loss, and the auxiliary loss are weighted and summed based on the weight coefficient setting rule to obtain the multi-dimensional loss.

[0012] A second aspect of the present invention provides a text-driven 3D scene stylization device, comprising: an architecture construction module for constructing a hybrid radial basis field representation architecture; a model construction module for constructing an initial style radial basis function model based on the hybrid radial basis field representation architecture; a model training module for performing multiple rounds of iterative training on the initial style radial basis function model to obtain a style radial basis function model; a model fine-tuning module for obtaining a pre-constructed multi-dimensional loss constraint system and fine-tuning the style radial basis function model based on the multi-dimensional loss constraint system to obtain a target style radial basis function model; and an instruction driving module for obtaining text instructions and driving the target style radial basis function model to generate a 3D scene stylization rendering result based on the text instructions.

[0013] A third aspect of the present invention provides a text-driven 3D scene stylization device, the text-driven 3D scene stylization device comprising: a memory and at least one processor, the memory storing instructions; the at least one processor calling the instructions in the memory to cause the text-driven 3D scene stylization device to perform the various steps of the text-driven 3D scene stylization method described in any of the preceding claims.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the steps of the text-driven 3D scene stylization method described in any of the preceding claims.

[0015] In the technical solution of this invention, a hybrid radial basis field representation architecture is first constructed. Then, an initial style radial basis function model is constructed based on the hybrid radial basis field representation architecture. The initial style radial basis function model is trained iteratively in multiple rounds to obtain the style radial basis function model. Next, a multi-dimensional loss is obtained. The style radial basis function model is fine-tuned based on the multi-dimensional loss to obtain the target style radial basis function model. Finally, text instructions are obtained. The target style radial basis function model is driven by the text instructions to generate a 3D scene stylization rendering result. This invention aims to solve the problems of blur artifacts and uneven style coverage in existing text-guided 3D scene stylization technologies, and achieves 3D scene stylization that is consistent across viewpoints, free of artifacts, highly faithful, and uniform in style. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the text-driven 3D scene stylization method provided in this embodiment of the invention.

[0017] Figure 2 A schematic diagram of the structure of the text-driven 3D scene stylization device provided in an embodiment of the present invention;

[0018] Figure 3 This is a schematic diagram of the structure of a text-driven 3D scene stylization device provided in an embodiment of the present invention. Detailed Implementation

[0019] This invention provides a text-driven 3D scene stylization method, apparatus, device, and storage medium. In this invention, the terms "first," "second," "third," "fourth," etc. (if present)," in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0020] For ease of understanding, the specific process of the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 1 One embodiment of the text-driven 3D scene stylization method in this invention includes:

[0021] 101. Construct a hybrid radial basis field representation architecture;

[0022] In this embodiment, when constructing the hybrid radial basis field representation architecture, a learnable parameter set containing the center position, anisotropic shape matrix, and feature weight vector is first obtained. Based on this parameter set, an adaptive radial basis function kernel set is defined. Each kernel can be flexibly positioned in space through its center position and anisotropic shape matrix, covering a local area with an elliptical kernel footprint, possessing morphological adaptability from near-spherical to slender elliptical. Next, a multi-frequency sine synthesis algorithm is used to expand the kernel capacity of the adaptive radial basis function kernel set: based on the original scalar output of the kernel, a multi-channel extended feature is generated by combining global scaling with multiple sets of phase offsets and frequency sine functions. Then, by combining the Hadamard product with the feature weight vector of the kernel, each kernel is expanded into a set of multi-frequency radial functions, forming an optimized radial basis function kernel set, thereby improving the high-frequency detail and texture representation capability of the kernel. Subsequently, a coarse-grid backbone network is introduced into the optimized radial basis function kernel group, and 3D query points along the camera rays are obtained. Based on these 3D query points, the contribution features of the optimized radial basis function kernel group at that point are calculated, namely the sum of the extended features of all kernels combined with the weight vector, and the interpolation features of the coarse-grid backbone network at that point. The two types of features are concatenated to obtain a hybrid feature vector that integrates adaptive high-frequency details and smooth baseline. Finally, a neural network decoder is used to decode the hybrid feature vector: the decoder contains two branches, a density network and a color network. The density network takes the hybrid feature vector as input and outputs volume density, while the color network processes the hybrid feature vector in conjunction with viewpoint encoding and outputs viewpoint-related RGB colors, thus completing the construction of the hybrid radial basis field representation architecture. By leveraging the flexible shape and spatial positioning of adaptive radial basis function kernels, the limitations of fixed meshes are overcome, allowing representation capacity to be concentrated on surface and detail regions, achieving efficient scene representation with fewer parameters. The introduction of multi-frequency sine synthesis enhances the encoding capability of high-frequency details and textures without increasing the number of kernels. The coarse-mesh backbone network provides smooth scene baseline features to ensure continuity and alleviates interpolation artifacts that may result from insufficient overlap of radial basis function kernels. Furthermore, the aggregation of hybrid feature vectors and the decoding of the dual-branch decoder can simultaneously and accurately model the volume density and view-dependent colors of the scene, balancing representation flexibility, high-frequency expressiveness, and structural stability.

[0023] 102. Construct an initial style radial basis function model based on the aforementioned hybrid radial basis field representation architecture;

[0024] In this embodiment, when constructing the initial style radial basis function model based on the hybrid radial basis field representation architecture, firstly, multi-view images of the actual scene and preset weight allocation rules are obtained. Then, based on this multi-view image set of the actual scene, initial 3D sampling points capable of representing key areas of the scene are obtained through motion structure reconstruction or training a low-resolution NeRF model. According to the weight allocation rules, each initial 3D sampling point is assigned a weight positively correlated with its importance. Specifically, sampling points located in high-gradient regions of the image, such as edges and corners, or in high-density regions of the scene, such as the surface of main objects, are assigned higher weights, while sampling points in blank areas or flat texture areas are assigned lower weights, thus obtaining weighted 3D sampling points. Next, a clustering algorithm is used to perform clustering processing on these weighted 3D sampling points in three-dimensional space. The resulting cluster centers will serve as the initial positions of each kernel in the optimized radial basis function kernel group. Simultaneously, based on the distribution characteristics of local points after clustering, the anisotropic shape matrix corresponding to each kernel is initialized, thereby completing the initialization of the position and shape of the optimized radial basis function kernel group. By combining the kernel feature weights and other parameters defined in the hybrid radial basis field (RBF) representation architecture, an initial style RBF model can be constructed. Through the synergistic effect of weight allocation rules and clustering algorithms, the position and shape of the optimized RBF kernel group can be precisely matched with the scene's geometry and texture distribution. This achieves concentrated configuration of model capacity in complex scene regions and streamlined configuration in simple, blank regions, ensuring a high degree of adaptation between model capacity and the non-uniformity of scene signals. Simultaneously, this initialization method avoids the initial configuration misconfiguration problem caused by random kernel placement, guiding the RBF to align with the core features of the scene from the early stages of training. This accelerates the convergence speed of subsequent model training and improves the reconstruction accuracy in the initial stage, achieving a higher initial peak signal-to-noise ratio compared to random or grid initialization strategies.

[0025] 103. Perform multiple rounds of iterative training on the initial style radial basis function model to obtain the style radial basis function model;

[0026] In this embodiment, the initial style radial basis function model undergoes multiple rounds of iterative training. First, based on multi-view images of the actual scene, camera ray data is sampled from all views, and the corresponding actual color data is extracted. This camera ray data is then input into the initial style radial basis function model, and volume rendering is completed by performing volume integral operations on the volume density and color output by the model to obtain the predicted color data of the corresponding ray. Subsequently, based on the actual color data and the predicted color data, the model reconstruction loss is calculated using a mean squared error combined with perceptual metrics. This loss function sums the errors corresponding to all random rays sampled in each iteration to ensure the comprehensiveness of the loss constraint. During training, a scheme combining position-level sampling and importance sampling is adopted along the rays: in the initial stage, coarse uniform ray sampling is used; as training progresses, the occupancy grid is updated every 256 steps, concentrating samples in regions with non-zero density or high-frequency details to accelerate the convergence speed. Meanwhile, the position and shape of the radial basis function kernel group are frozen during iterative optimization. The feature weights of the kernel group and the decoder network weights are updated iteratively only through the backpropagation algorithm. This ensures the stability of the scene geometry during training and avoids training instability problems that may be caused by changes in the kernel group position. When the model meets the preset iteration stopping condition, the style radial basis function model can be output.

[0027] 104. Obtain the multi-dimensional loss, and fine-tune the style radial basis function model based on the multi-dimensional loss to obtain the target style radial basis function model;

[0028] In this embodiment, camera ray data is first input into the style radial basis function model (RBF) for volume rendering to obtain stylized predicted color data for each ray. This data is then stitched together to form a complete stylized scene image. Simultaneously, target text for defining the target style and fixed text corresponding to the actual scene are acquired. Then, a pre-trained CLIP model is called to calculate multi-dimensional loss. Based on this multi-dimensional loss, the backpropagation algorithm is used to iteratively update the feature weights of the pre-trained RBF kernel group of the style RBF model, while simultaneously updating the decoder parameters. The center and shape of the kernel group are frozen to maintain multi-view consistency, and a small learning rate is used to avoid parameter mutations. When the multi-dimensional loss converges and the stylized scene image and target text subjectively match in style, the iteration stops, and the target style RBF model is output.

[0029] 105. Obtain text instructions, and drive the target style radial basis function model to generate 3D scene stylized rendering results based on the text instructions.

[0030] In this embodiment, obtaining text instructions refers to receiving natural language descriptions input by the user. These descriptions can cover dimensions such as artistic style, visual features, and texture tendencies, used to clarify the target style to be presented in the 3D scene. The process of generating stylized rendering results of the 3D scene based on the target style radial basis function model driven by these text instructions requires first determining the camera viewpoint parameters to be rendered. The camera viewpoint parameters can be determined in two ways: preset viewpoint selection and user-defined settings. If preset viewpoint selection is used, the model's pre-configured multi-viewpoint parameter set can be called. This parameter set includes combinations of camera position, orientation, and field of view angle covering all dimensions of the scene, adapting to the multi-viewpoint rendering requirements of conventional 3D scenes. If user-defined settings are used, the camera world coordinates, line-of-sight orientation vector, and field of view angle values ​​input by the user can be received to accurately match the observation viewpoint specified by the user, thus completing the determination of the camera viewpoint parameters to be rendered. The style corresponding to a text instruction refers to mapping the user-input natural language text instruction into a set of style features with clear visual orientation in the CLIP semantic space. This feature set encompasses core visual style elements such as the art style characteristics, color distribution patterns, texture attributes, and lighting effects described by the text instruction, serving as the core basis for the model to perform stylized rendering. The style corresponding to a text instruction is determined as follows: the user-input text instruction is fed into a pre-trained CLIP text encoder. The encoder's multi-layer Transformer structure encodes the text semantics layer by layer, transforming the natural language description into a high-dimensional style embedding vector. This style embedding vector is the style mapping information corresponding to the text instruction. Subsequently, these viewpoint parameters and the style mapping information corresponding to the text instruction are input into the target style radial basis function model. The model leverages the scene geometry and basic appearance representation learned during the pre-training phase, combined with the correlation between text style and scene visual features established during the fine-tuning phase, to sample the 3D space from a specified viewpoint. It obtains the fused features of each sampling point through feature aggregation operations of the mixed radial basis fields, followed by density and color branch decoding to obtain the volume density and color information of each sampling point that matches the target style. Finally, it accumulates the sampling point information into pixel color through volume rendering integration. By traversing multiple sets of different camera viewpoint parameters and repeating the above rendering process, it obtains a multi-view stylized image sequence. These sequences possess cross-viewpoint style consistency and geometric coherence, collectively constituting a multi-dimensional observable 3D scene stylization result. This achieves a direct association between natural language and 3D scene stylization content. Users do not need to master complex 3D modeling or style editing techniques; they can drive the model to generate content that meets their needs simply through intuitive text descriptions, significantly lowering the technical threshold for 3D stylization creation.Meanwhile, the target model has ensured uniform style coverage and scene structure stability during the fine-tuning stage. Therefore, the generated rendering results can accurately reproduce the style features defined by the text instructions, maintain the original geometric shape and spatial logic of the scene, and the cross-view rendering results have a high degree of style uniformity and visual coherence, avoiding style distortion or geometric disorder caused by viewpoint switching, effectively improving the efficiency and presentation quality of 3D scene stylization creation.

[0031] In this embodiment of the invention, the construction of the hybrid radial basis field representation architecture includes: obtaining a set of learnable parameters; defining an adaptive radial basis function kernel set based on the set of learnable parameters; performing kernel capacity expansion processing on the adaptive radial basis function kernel set using a multi-frequency sine synthesis algorithm to obtain an optimized radial basis function kernel set; constructing a coarse-grid backbone network, wherein the coarse-grid backbone network and the optimized radial basis function kernel set form complementary feature extraction branches, and setting a neural network decoder to complete the construction of the hybrid radial basis field representation architecture.

[0032] In this embodiment, when constructing the hybrid radial basis field representation architecture, a learnable parameter set containing the center position (i.e., position), anisotropic shape matrix (i.e., shape), and feature weight vector (i.e., feature weight) is first obtained. An adaptive radial basis function kernel set is then defined based on this parameter set. Each kernel in the adaptive radial basis function kernel set is defined by the learnable center position. Anisotropic shape matrix and feature weight vector Defined, its kernel function is ,in, Represents a three-dimensional query point. This represents the index of each kernel in the adaptive radial basis function kernel set, used to uniquely identify a specific radial basis function kernel within the kernel set. Its value range corresponds to the number of kernels. This represents the transpose of the 3D difference vector. By constraining the center position and shape matrix, this function achieves smooth attenuation coverage of local areas of the scene, and its shape can flexibly adapt to different spatial distribution requirements.

[0033] In this embodiment, a multi-frequency sine synthesis algorithm is used to expand the kernel capacity of the adaptive radial basis function kernel set. Specifically, let... The original scalar output of the kernel is transformed into a set of sine functions. ,in, Indicates the global scaling factor. This represents different phase shifts and frequencies, with the frequencies exhibiting a logarithmic linear distribution between the lowest and highest values. This represents the raw scalar output of the kernel. For the index of the sine component, This is used to distinguish sinusoidal channels of different frequencies to generate m-dimensional extended features, and its expression is: ,(in, Let represent the multi-frequency extended feature vector of the i-th adaptive radial basis function kernel. This expands the original single scalar output into m-dimensional features, thereby increasing the kernel's capacity. Here, m is the number of frequency channels (a preset hyperparameter; all adaptive radial basis function kernels share the same set of m frequency channels). Then, the extended feature vector is combined with the kernel's feature weight vector using the Hadamard product to obtain... ,in, Let i be the eigenweight vector of the i-th adaptive radial basis function kernel. Represents the Hadamard product. This indicates that the i-th optimized radial basis function kernel is at the query point. The final feature output is used to form an optimized radial basis function kernel set, which improves its high-frequency detail coding capability without increasing the number of kernels.

[0034] Subsequently, a coarse-mesh backbone network is constructed. This lightweight feature encoding network is built based on a low-resolution regular 3D mesh and can generate globally smooth spatial features through interpolation operations, providing basic geometric and layout representations for the scene. The coarse-mesh backbone network and the optimized radial basis function kernel group form complementary feature extraction branches and obtain a preset 3D query point. This point is a 3D spatial point sampled along the camera ray r(t) = o + td, where o is the camera origin, d is the ray direction vector, and t is the sampling interval parameter. Based on this query point, the contribution features of the optimized radial basis function kernel group and the interpolation features of the coarse-mesh backbone network are obtained respectively. The contribution feature is the sum of the outputs of all kernels at this point, and the specific calculation expression is as follows:

[0035] ,

[0036] in, This represents the total contribution feature of the optimized radial basis function kernel set at the 3D query point, used to encode high-frequency details and local textures of the scene; M This represents the total number of kernels in the adaptive radial basis function kernel set. It is a preset hyperparameter that determines the modeling scale of the kernel set. Indicates the first i A multi-frequency extended eigenvector of an adaptive radial basis function kernel. Indicates the first i The feature weight vector of an adaptive radial basis function kernel. Represents a three-dimensional query point. This represents the Hadamard product, an element-wise multiplication operation used to multiply the corresponding elements of the feature weight vector and the multi-frequency extended feature vector, achieving weighted fusion of multi-frequency features. The interpolated feature is the interpolation result of the coarse-grid backbone network at that point. The two types of features are concatenated to obtain a mixed feature vector. It integrates high-frequency details with smooth baseline features.

[0037] Finally, a neural network decoder is configured to complete the construction of the hybrid radial basis field representation architecture. The neural network decoder is a two-branch decoder with a fully connected multilayer perceptron (MLP) architecture. It employs a modular, stacked fully connected layer design to decode the hybrid feature vector. The decoder consists of two branches: a density network and a color network. Both branches share the input of the hybrid feature vector and use ReLU as the hidden layer activation function to achieve non-linear feature mapping. The density network is a three-layer fully connected perceptron structure, with the first fully connected layer's input dimension sharing the hybrid feature vector. The first layer has the same dimensions, with an output dimension of 256. The second fully connected layer has an input dimension of 256 and an output dimension of 128. The third fully connected layer is the output layer, with an input dimension of 128 and an output dimension of 1, and no activation function is set, directly outputting the volume density σ(x). The density network is responsible for mapping the mixed features to volume density information. Volume density represents the opacity of a point and is the core parameter for the accumulated color of light in volume rendering. The color network is a four-layer fully connected perceptron structure, combined with a 6-dimensional sine wave form of view encoding, first converting the mixed feature vectors... The 6-dimensional sinusoidal encoded features of the viewpoint direction w are concatenated to form a fused feature vector as input. The input dimension of the first fully connected layer is the sum of the dimension of the fused feature vector and the dimension of the 6-dimensional viewpoint encoding, and the output dimension is set to 256. The input dimension of the second fully connected layer is 256, and the output dimension is set to 128. The input dimension of the third fully connected layer is 128, and the output dimension is set to 64. The fourth fully connected layer is the output layer, with an input dimension of 64 and an output dimension of 3. The Sigmoid activation function is used to normalize the output value to the [0,1] interval, and finally outputs the viewpoint-related RGB color c(x,w).

[0038] In this embodiment of the invention, the learnable parameter set includes position and shape; the construction of the initial style radial basis function model based on the hybrid radial basis field representation architecture includes: acquiring multi-view images of the actual scene and a preset weight allocation rule; acquiring multiple initial 3D sampling points based on the multi-view images of the actual scene, and assigning weights to each initial 3D sampling point based on the weight allocation rule to obtain weighted 3D sampling points; performing clustering processing on the weighted 3D sampling points using a clustering algorithm to obtain multiple cluster centers; initializing the position and shape of the optimized radial basis function kernel set based on the multiple cluster centers to construct the initial style radial basis function model.

[0039] In this embodiment, when constructing the initial style radial basis function model based on the hybrid radial basis field representation architecture, the learnable parameter set includes position, shape, and feature weights. Position corresponds to the center coordinates of the radial basis function kernel, shape corresponds to the anisotropic shape matrix, and feature weights are learnable vectors that regulate the kernel feature contribution. First, multi-view images of the actual scene are obtained along with a preset weight allocation rule. The actual scene multi-view images are real images covering different observation angles of the scene. The weight allocation rule stipulates that the weight of the initial 3D sampling points is positively correlated with the importance of their respective regions; that is, sampling points located in high-gradient regions (such as edges and corners) or high-density regions (such as the surface of main objects) are assigned higher weights, while sampling points located in blank regions or flat texture regions are assigned lower weights. Next, initial 3D sampling points are obtained based on the multi-view image set of the actual scene. This can be achieved by reconstructing sparse point clouds of the scene through motion structure reconstruction, or by training a low-resolution neural radiation field model to obtain an initial density estimate and sampling from it. Subsequently, each initial 3D sampling point is assigned a corresponding weight according to the weight allocation rule, resulting in weighted 3D sampling points. The differentiated weight allocation allows points in complex areas of the scene to have a higher influence in subsequent processing.

[0040] Then, a clustering algorithm is used to perform clustering processing on the weighted 3D sampling points. This clustering algorithm adopts the Lloyd algorithm, i.e. the weighted K-Means algorithm, and obtains multiple cluster centers through iterative optimization. The position and shape of the optimized radial basis function kernel group are initialized based on multiple cluster centers. Specifically, each cluster center is used as the position of the corresponding optimized radial basis function kernel, i.e., the center coordinate of the kernel. Based on the spatial distribution characteristics of the weighted 3D sampling points contained in each cluster, the three-dimensional covariance matrix corresponding to the cluster is calculated. The calculated covariance matrix is ​​directly initialized as the anisotropic shape matrix of the corresponding optimized radial basis function kernel. At the same time, a preset scaling factor can be applied to the covariance matrix for regularization according to the scene spatial scale to adapt to the spatial coverage range of the initial stage of the model, thereby completing the initialization of the shape of the optimized radial basis function kernel group. The anisotropic shape matrix is ​​initialized based on the spatial distribution of local points after clustering. Specifically, for each cluster center, the three-dimensional coordinate data of all weighted 3D sampling points within that cluster are extracted. The spatial covariance matrix is ​​calculated for this set of three-dimensional coordinate data. This spatial covariance matrix is ​​used to characterize the spatial extension direction, density, and scale of the sampling points in the local area. The spatial covariance matrix is ​​directly used as the anisotropic shape matrix of the corresponding optimized radial basis function kernel, or the spatial covariance matrix is ​​normalized and scaled to a preset scale and then used as the anisotropic shape matrix. This matrix defines the elliptical kernel footprint of the kernel, the shape of which is determined by the direction and scale of the local point distribution, so that the shape of the kernel adapts to the spatial characteristics of the local area. The feature weights are then used as learnable parameters and updated in the subsequent training phase. This completes the initialization of the optimized radial basis function kernel set. Combined with other components of the hybrid radial basis field representation architecture, an initial style radial basis function model is constructed. By combining weight allocation and weighted clustering, the position and shape of the optimized radial basis function kernel group are precisely matched with the geometric shape and texture distribution of the scene. This enables the model capacity to be centrally configured in complex areas of the scene, allowing the model capacity to be highly adapted to the non-uniformity of the scene signal. Compared with random or grid initialization, this initialization method can guide the radial basis field to fit the core features of the scene from the early stage of training, which not only accelerates the convergence speed of subsequent model training but also improves the reconstruction accuracy in the initial stage, resulting in a higher initial peak signal-to-noise ratio. At the same time, the kernel position and shape initialized based on the clustering results have good spatial distribution rationality, providing a basis for the strategy of freezing the kernel position and shape and only updating the feature weights in the subsequent training stage. This ensures the stability of the scene geometry and avoids training instability problems that may be caused by changes in the kernel position.

[0041] In this embodiment of the invention, the learnable parameter set further includes feature weights. The step of performing multiple rounds of iterative training on the initial style radial basis function model to obtain the style radial basis function model includes: obtaining camera lighting data based on multi-view image sampling of the actual scene, and acquiring actual color data corresponding to the camera lighting data; inputting the camera lighting data into the initial style radial basis function model for volume rendering to obtain predicted color data; calculating the model reconstruction loss based on the actual color data and the predicted color data; iteratively updating the feature weights of the optimized radial basis function kernel set of the initial style radial basis function model using a backpropagation algorithm based on the model reconstruction loss; and outputting the style radial basis function model when a preset iteration stopping condition is met.

[0042] In this embodiment, the initial style radial basis function (RBF) model is trained iteratively through multiple rounds to obtain the style RBF model. First, based on multi-view images of the actual scene, camera ray data is sampled from all observation views. Camera ray data refers to the set of three-dimensional ray parameters emitted from each pixel on the camera's imaging plane along the camera's observation direction, including the three-dimensional origin coordinates and unit direction vector of each ray. At the same time, the actual color data of the corresponding pixels of these rays is extracted, which serves as the color reference for the actual observation of the scene. Subsequently, the camera ray data is input into the initial style RBF model. The model performs volume integral operations on the volume density and color information along each ray to complete the volume rendering process and outputs the predicted color data of the corresponding ray.

[0043] Next, the model reconstruction loss is calculated based on the actual color data and the predicted color data. The specific calculation expression is as follows:

[0044] ,

[0045] in, This represents the model reconstruction loss, which is the optimization objective of the first stage of training and is used to quantify the visual difference between the model rendering result and the real scene. Represents a set of light rays Summing all the light rays, This represents a single camera ray, that is, a 3D parametric ray originating from a camera pixel. During the volume rendering process, the model will follow the ray... The propagation path is sampled to obtain a series of three-dimensional spatial coordinate points. For each three-dimensional coordinate point, the model calculates its volume density and color features using a hybrid radial basis field representation architecture. Then, based on the volume rendering integral formula, the volume density and color features of all three-dimensional coordinate points along the path are integrated to finally obtain the ray. The corresponding predicted color data; This represents the predicted color data obtained by the model from rendering the volume of light r. This represents the actual color data corresponding to ray r. This represents the square of the L2 norm, i.e., the mean square error operation, used to calculate the sum of squared errors between the predicted color and the actual color.

[0046] During iterative optimization, a combination of location-based hierarchical sampling and importance sampling is employed to improve training efficiency: Initially, coarse, uniform lighting sampling is used. As training progresses, the occupancy grid is updated every 256 steps, concentrating sampling samples in areas with non-zero scene density or high-frequency details to accelerate convergence. Simultaneously, based on the model reconstruction loss, backpropagation is used to iteratively update the feature weights of the optimized radial basis function (RBF) kernel group in the initial style RBF model, while also updating the decoder network weights. During this process, the position and shape parameters of the optimized RBF kernel group are frozen, i.e., the center coordinates and anisotropic shape matrix of the kernels are fixed, ensuring the scene geometry remains stable throughout training and avoiding training instability caused by kernel position changes. The model learns the contribution of each kernel to scene color and density solely through feature weight updates. When the model meets preset iteration stopping conditions, such as the reconstruction loss converging to a preset threshold or reaching the maximum number of iterations, iteration stops and the style RBF model is output.

[0047] In this embodiment of the invention, obtaining the multi-dimensional loss and fine-tuning the style radial basis function model based on the multi-dimensional loss to obtain the target style radial basis function model includes: inputting the camera lighting data into the style radial basis function model for volume rendering to obtain stylized predicted color data, and stitching the stylized predicted color data to obtain a stylized scene image; obtaining the target text corresponding to the stylized scene image and the fixed text corresponding to the actual scene multi-view image; calling the pre-trained CLIP model to perform multi-dimensional loss calculation based on the stylized scene image, the actual scene multi-view image, the target text, and the fixed text to obtain the multi-dimensional loss; using the backpropagation algorithm to iteratively update the feature weights of the pre-trained radial basis function kernel group of the style radial basis function model based on the multi-dimensional loss; and outputting the target style radial basis function model when a preset iteration stopping condition is met.

[0048] In this embodiment, camera ray data is input into the style radial basis function model to perform volume rendering. Volume integral operations are performed on the volume density and color information output by the model along each ray to obtain the stylized predicted color data corresponding to each ray. These data are then stitched together to form a complete stylized scene image. At the same time, target text for defining the target art style and fixed text corresponding to the original scene are obtained. The target text carries the style semantics expected by the user, while the fixed text is a scene description without style bias.

[0049] Subsequently, the pre-trained CLIP model is invoked for multi-dimensional loss calculation. The pre-trained CLIP model, namely the contrastive language-image pre-trained model, is a multimodal pre-trained model for achieving visual and linguistic semantic alignment. The training process of this model is based on a large-scale image-text pair dataset and employs a contrastive learning paradigm. For all image and text samples within a batch, the model maximizes the cosine similarity between paired image and text embeddings while minimizing the cosine similarity between unpaired image and text embeddings. This contrastive loss optimizes the encoder parameters, enabling the model to learn the general association between visual content and text semantics. It possesses the ability to measure the semantic matching degree between images and text under zero-shot conditions and achieves cross-modal semantic alignment without requiring fine-tuning for specific tasks.

[0050] Based on this multidimensional loss, a backpropagation algorithm is used to iteratively update the feature weights of the pre-trained radial basis function kernel group in the style radial basis function model, while simultaneously updating the decoder parameters. During this process, the center position and anisotropic shape matrix of the kernel group are frozen to maintain multi-view consistency, and a small learning rate is used to avoid significant parameter mutations affecting rendering quality. When the multidimensional loss converges to a preset threshold, and the stylized scene image and the target text are subjectively matched in style, a preset iteration stopping condition is met, the iteration stops, and the target style radial basis function model is output. Here, the subjective style matching between the stylized scene image and the target text refers to randomly selecting several stylized scene images at preset iteration intervals (e.g., 50 times) during the iteration process, extracting their visual embedding vectors through the image encoder of the pre-trained CLIP model, and simultaneously extracting the text embedding vector of the target text, calculating the cosine similarity between the two as the style matching degree; when the style matching degree is higher than a preset threshold (e.g., 0.7), the stylized scene image and the target text are determined to be style matched. The strategy of freezing the spatial parameters of the kernel group and updating only the feature weights maintains cross-view consistency while achieving stylization adjustment. The final target model can output high-fidelity, style-uniform and structure-consistent 3D scene stylization results.

[0051] In this embodiment of the invention, the CLIP model includes a text encoder, an image encoder, and a multi-dimensional loss calculation module. The text encoder and the image encoder are respectively connected to the multi-dimensional loss calculation module. The step of calling the pre-trained CLIP model to perform multi-dimensional loss calculation based on the stylized scene image, the actual scene multi-view image, the target text, and the fixed text to obtain the multi-dimensional loss includes: obtaining a pre-trained radial basis function kernel set based on the stylized radial basis function model; performing camera plane projection processing on the pre-trained radial basis function kernel set to obtain an elliptical coverage region; and generating a scene saliency map based on the elliptical coverage region; constructing a bias sampling rule based on the scene saliency map; randomly sampling the stylized scene image based on the bias sampling rule to obtain a set of stylized local image patches; and performing multi-view sampling on the actual scene. The image acquisition process involves obtaining the original set of local image patches corresponding to the stylized local image patch set; text encoding of the target text and the fixed text based on the text encoder to obtain the target text embedding vector and the fixed text embedding vector; image encoding of the stylized scene image, the actual scene multi-view image, the stylized local image patch set, and the original set of local image patches based on the image encoder to obtain the stylized image embedding vector, the original image embedding vector, the stylized image patch embedding vector, and the original image patch embedding vector; and multi-dimensional loss calculation based on the target text embedding vector, the fixed text embedding vector, the stylized image embedding vector, the original image embedding vector, the stylized image patch embedding vector, and the original image patch embedding vector by the multi-dimensional loss calculation module to obtain the multi-dimensional loss.

[0052] In this embodiment, the pre-trained CLIP model, also known as the contrastive language-image pre-trained model, has a core architecture comprising a text encoder, an image encoder, and a multi-dimensional loss calculation module. The text encoder and image encoder are respectively connected to the multi-dimensional loss calculation module. This model is pre-trained on a large-scale image-text pair dataset and employs a contrastive learning paradigm. Specifically, the text encoder typically uses a Transformer structure to map the input text into high-dimensional text feature embeddings; the image encoder can use a visual Transformer or ResNet structure to map the input image into visual feature embeddings of the same dimension. Both types of embeddings share the same semantic feature space. During training, the model performs contrastive optimization on all samples within a batch, maximizing the cosine similarity between paired images and text embeddings while minimizing the cosine similarity between unpaired images and text embeddings. This contrastive loss continuously updates the encoder parameters, enabling the model to learn the general association between visual content and text semantics. It possesses the ability to measure cross-modal semantic matching under zero-shot conditions and achieves accurate semantic alignment between vision and language without requiring fine-tuning for specific tasks.

[0053] Subsequently, a biased sampling rule is constructed based on the scene saliency map. This rule is implemented as follows: First, the saliency value (i.e., the number of kernels covered by ellipses) of each pixel in the scene saliency map is normalized to obtain a sampling probability distribution for each pixel, giving pixels in salient regions higher sampling weights and correspondingly lower sampling weights for pixels in non-salient regions. Then, based on this probability distribution, a multinomial sampling method is used to randomly select multiple pixels as sampling centers for local image patches. Using each selected pixel as the center, a local image patch of a preset fixed size is cropped, thus obtaining a set of stylized local image patches. Simultaneously, local image patches of the same size are cropped from the corresponding viewpoints and spatial locations in the multi-view images of the actual scene, resulting in the original set of local image patches that corresponds one-to-one with the stylized local image patches. This rule, through the biased allocation of sampling probabilities, ensures that more local patches are concentrated in the core content area of ​​the scene while covering non-salient areas. This avoids the omission of important areas due to random sampling and ensures that local contrast loss applies to all areas of the scene, providing a reliable sample basis for subsequent uniform stylization constraints.

[0054] Next, the target text and fixed text are encoded separately using the CLIP model's text encoder, resulting in target text embedding vectors and fixed text embedding vectors. Then, the stylized scene image, real-world multi-view images, a set of stylized local image patches, and a set of original local image patches are encoded separately using the image encoder, resulting in stylized image embedding vectors, original image embedding vectors, stylized image patch embedding vectors, and original image patch embedding vectors. Finally, based on the multi-dimensional loss calculation module, the multi-dimensional loss is calculated using the above embedding vectors, enabling the model, after fine-tuning, to output high-fidelity, style-uniform, and structurally consistent 3D scene stylization results, while maintaining visual coherence across viewpoints.

[0055] In this embodiment of the invention, the multi-dimensional loss calculation module includes a relative directionality loss calculation submodule, a global contrast loss calculation submodule, a local contrast loss calculation submodule, and an auxiliary loss calculation submodule. The relative directionality loss calculation submodule, the global contrast loss calculation submodule, and the local contrast loss calculation submodule are respectively connected to the text encoder and the image encoder, and the auxiliary loss calculation submodule is connected to the image encoder. The multi-dimensional loss is obtained by performing multi-dimensional loss calculation based on the target text embedding vector, the fixed text embedding vector, the stylized image embedding vector, the original image embedding vector, the stylized image patch embedding vector, and the original image patch embedding vector, according to the multi-dimensional loss calculation module. This includes: calculating the relative directionality loss based on the target text embedding vector, the fixed text embedding vector, the stylized image patch embedding vector, and the original image patch embedding vector. The following steps are taken: A relative directional loss is calculated using the target text embedding vector, the fixed text embedding vector, the stylized image embedding vector, and the original image embedding vector. A global contrast loss calculation submodule calculates a global contrast loss based on the target text embedding vector, the stylized image embedding vector, and the original image embedding vector. A local contrast loss calculation submodule calculates a local contrast loss based on the stylized image patch embedding vector and the original image patch embedding vector. An auxiliary loss calculation submodule calculates an auxiliary loss based on the stylized scene image and the actual scene multi-view image. A preset weighting coefficient setting rule is obtained, and the relative directional loss, the global contrast loss, the local contrast loss, and the auxiliary loss are weighted and summed based on the weighting coefficient setting rule to obtain the multi-dimensional loss.

[0056] In this embodiment, the multi-dimensional loss calculation module includes a relative directional loss calculation submodule, a global contrast loss calculation submodule, a local contrast loss calculation submodule, and an auxiliary loss calculation submodule. The relative directional loss calculation submodule, the global contrast loss calculation submodule, and the local contrast loss calculation submodule are connected to the text encoder and the image encoder, respectively, and the auxiliary loss calculation submodule is connected to the image encoder.

[0057] The relative directionality loss calculation submodule calculates the relative directionality loss based on the target text embedding vector, the fixed text embedding vector, the stylized image embedding vector, and the original image embedding vector. The goal is to constrain the semantic direction consistency during the stylization process and avoid content drift. Specifically, the image embedding offset is first defined based on the stylized image embedding vector and the original image embedding vector. The specific calculation expression is as follows:

[0058] ,

[0059] in, Indicates the image embedding offset. Represents the stylized image embedding vector. This represents the original image embedding vector.

[0060] Simultaneously, the text embedding offset is defined based on the target text embedding vector and the fixed text embedding vector, and the specific calculation expression is as follows:

[0061] ,

[0062] in, Indicates text embedding offset, Represents the target text embedding vector. This represents a fixed text embedding vector.

[0063] Finally, relative directionality loss is calculated based on image embedding offset and text embedding offset. The optimization goal of relative directionality loss is to ensure that image embedding offset and text embedding offset are collinear and have matching magnitudes in the CLIP semantic space. That is, to constrain the semantic change direction of the stylized image relative to the original image to be consistent with the semantic style change direction of the target text relative to the fixed text, thereby preserving the core content of the scene while introducing style. The specific loss function can be expressed as:

[0064] ,

[0065] in, Indicates relative directional loss. Indicates the image embedding offset. Indicates the text embedding offset.

[0066] The global contrastive loss calculation submodule calculates a global contrastive loss based on the target text embedding vector, the stylized image embedding vector, and the original image embedding vector, to enhance the semantic alignment between the overall scene and the target style. Specifically, using the target text embedding vector as the anchor, the stylized image embedding vector as the positive example and the original image embedding vector as the negative example, a contrastive learning objective is constructed in the CLIP feature space, employing a loss function form such as infoNCE, as shown in the following formula:

[0067] ,

[0068] in, Indicates global comparison loss. Represents the cosine similarity function; This represents the temperature hyperparameter, used to adjust the smoothness of contrast loss; Represents the stylized image embedding vector. Represents the target text embedding vector. This represents the original image embedding vector.

[0069] The above formula includes two cosine similarities, corresponding to the positive and negative examples in the numerator and denominator of infoNCE, respectively. These co-values, expressed as a logarithmic ratio, together constitute the global contrastive loss. Specifically, The positive example cosine similarity is the cosine similarity between the stylized image embedding vector and the target text embedding vector. This term appears in both the numerator and denominator. The optimization goal is to make it as large as possible, driving the stylized image to converge towards the target text in the semantic space. The negative cosine similarity, i.e., the cosine similarity between the original image embedding vector and the target text embedding vector, appears in the denominator. The optimization objective is to minimize this value to prevent the original image from being semantically incorrectly pulled closer to the target style text. The goal is to maximize the cosine similarity between the stylized image embedding and the target text embedding, while minimizing the cosine similarity between the original image embedding and the target text embedding. This ensures a high degree of match between the visual features of the entire stylized scene and the semantic style of the target text, avoiding overall insufficient stylization.

[0070] The local contrast loss calculation submodule calculates local contrast loss based on the stylized image patch embedding vector and the original image patch embedding vector to ensure the uniformity of stylization across different regions of the scene. Its constraint construction relies on a bias sampling rule based on the scene saliency map. First, the saliency value of each pixel in the scene saliency map (i.e., the number of kernels covered by the radial basis function kernel ellipse) is normalized to obtain a sampling probability distribution for each pixel. This gives pixels in salient regions (such as the core content area of ​​the scene) higher sampling weights, while reducing the sampling weights of pixels in non-salient regions (such as the background area). Then, based on the above probability distribution, a multinomial sampling method is used to randomly select pixels from the stylized scene image. Each pixel is used as the sampling center of a local image patch. Local image patches of a preset fixed size are cropped with each center pixel as the base point to form a set of stylized local image patches. Then, based on the corresponding viewpoints and spatial locations of multi-view images in the actual scene, image patches of the same size as the stylized local image patches are extracted, resulting in an original set of local image patches that corresponds one-to-one with the set of stylized local image patches. By differentiating the bias sampling probability, more local blocks are concentrated in the core content area of ​​the scene, while covering non-salient areas. This avoids the omission of important areas due to random sampling and ensures that the local contrast loss can be applied to all areas of the scene, providing a reliable sample basis for subsequent uniform stylization constraints.

[0071] Based on the bias sampling rules generated by the pre-trained radial basis function kernel saliency, a set of stylized local image patches and the corresponding original set of local image patches are obtained. The set of stylized local image patches... With the original set of local image patches The images are input into the image encoder to obtain stylized image patch embedding vectors. With the original image patch embedding vector For each corresponding local patch embedding vector, the infoNCE contrastive learning logic, consistent with the global contrastive loss, is used. Stylized local image patch embeddings are used as positive examples, and the original local image patch embeddings are used as negative examples to construct the loss constraint. The local contrastive loss... The calculation formula is as follows:

[0072] ,

[0073] in, Indicates the first Embedding vectors of stylized local image patches (positive example). Indicates the first Embedding vectors of original local image patches (negative examples). Represents the target text embedding vector. This represents the total number of local image patches obtained by bias sampling. This indicates temperature hyperparameters. This represents the cosine similarity function. This loss applies the same style alignment constraint as the global loss to each local region (including small regions and background regions) in the scene, ensuring that all regions of the scene undergo style transfer. This solves the problem of missing style in some regions in traditional methods and achieves uniform style coverage across the entire scene. The difference between this and the global contrast loss is that the local contrast loss operates at the fine-grained image patch level, which can constrain the style consistency of local regions of the scene, while the global contrast loss constrains the style matching degree of the entire scene. The two work together to ensure the overall and local uniformity of 3D scene stylization.

[0074] The auxiliary loss calculation submodule calculates auxiliary losses based on stylized scene images and multi-view images of the actual scene, comprising two parts: perceptual content loss and density regularization loss. The first is perceptual content loss, which extracts visual feature maps at different levels from the stylized scene image and the multi-view image of the actual scene, calculating the squared difference loss between the corresponding feature maps. This loss constrains the spatial layout and object identification of the stylized scene to remain consistent with the original scene, preventing the loss of structural details due to over-stylization. The second is density regularization loss, which first obtains the original volume density corresponding to the multi-view image of the actual scene and the stylized volume density corresponding to the stylized scene image. Then, an L2 penalty is applied to the difference between the stylized volume density and the original volume density to obtain the density regularization loss. This loss suppresses cloud-like artifacts or noise caused by abrupt density changes, ensuring smooth changes in geometric shape without freezing the geometry to adapt to stylization requirements.

[0075] Finally, based on the preset weighting coefficient settings, the relative directional loss, global contrast loss, local contrast loss, and auxiliary loss are weighted and summed to obtain the multi-dimensional loss. The specific calculation expression is as follows:

[0076] ,

[0077] in, Indicates multi-dimensional loss. This represents the relative directional loss weight, used to adjust the alignment between the stylization direction and the target text; This represents the contrast loss weight, used to uniformly control the intensity of global and local contrast loss, balancing the intensity and uniformity of stylization; This represents the perceived content loss weight, used to constrain the degree of preservation of scene structure; This represents the density regularization loss weight, used to suppress the generation of rendering artifacts; Indicates relative directional loss. This represents the sum of global contrast loss and local contrast loss. This indicates a loss of perceived content. This represents the density regularization loss. The preset weight coefficient setting rules are mainly based on the core role level of each loss in 3D scene stylization, the complexity of the actual scene and the type difference of the target style, as well as the dynamic convergence state setting during model training. At the same time, it follows the basic principles of clear hierarchy, task adaptation and constraint balance, giving priority to ensuring semantic alignment and style uniformity.

[0078] The various sub-modules of the multi-dimensional loss calculation module work together. The relative directional loss sub-module ensures that the semantic direction is accurately aligned with the target text during the stylization process, avoiding content drift. The combination of global and local contrast loss sub-modules ensures both the overall style matching degree of the scene and achieves uniform style coverage across the entire region, solving some stylization problems. The perceptual content loss in the auxiliary loss sub-module maintains the integrity of the scene structure, while the density regularization loss suppresses rendering artifacts and ensures rendering quality. The weighted summation loss design can flexibly adjust the weights of each constraint, enabling the model to output high-fidelity, style-uniform, structurally stable, and cross-viewpoint consistent 3D scene stylization results after fine-tuning, providing a reliable loss constraint framework for text-driven 3D stylization.

[0079] The text-driven 3D scene stylization method in the embodiments of the present invention has been described above. The text-driven 3D scene stylization apparatus in the embodiments of the present invention is described below. Please refer to [link / reference]. Figure 2 One embodiment of the text-driven 3D scene stylization device of the present invention includes:

[0080] Architecture building module 201: Used to build a hybrid radial basis field representation architecture;

[0081] Model building module 202: used to build an initial style radial basis function model based on the hybrid radial basis field representation architecture;

[0082] Model training module 203: used to perform multiple rounds of iterative training on the initial style radial basis function model to obtain the style radial basis function model;

[0083] Model fine-tuning module 204: used to obtain a pre-constructed multi-dimensional loss constraint system, and fine-tune the style radial basis function model based on the multi-dimensional loss constraint system to obtain the target style radial basis function model;

[0084] Instruction-driven module 205: Used to acquire text instructions and drive the target style radial basis function model to generate 3D scene stylized rendering results based on the text instructions.

[0085] Based on the same ideas as the methods in the above embodiments, the apparatus provided in this application can implement the methods in the above embodiments.

[0086] above Figure 2 The text-driven 3D scene stylization device in this embodiment of the invention will be described in detail from the perspective of modular functional entities. The text-driven 3D scene stylization device in this embodiment of the invention will be described in detail from the perspective of hardware processing.

[0087] Figure 3 This is a schematic diagram of the structure of a text-driven 3D scene stylization device 300 provided in an embodiment of the present invention. The text-driven 3D scene stylization device 300 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the text-driven 3D scene stylization device 300. Furthermore, the processor 310 may be configured to communicate with the storage media 330, executing a series of instruction operations in the storage media 330 on the text-driven 3D scene stylization device 300 to implement the steps of the text-driven 3D scene stylization method provided in the above-described method embodiments.

[0088] The text-driven 3D scene stylization device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The illustrated structure of the text-driven 3D scene stylization device does not constitute a limitation on the text-driven 3D scene stylization device, which may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0089] The present invention also provides a computer-readable storage medium, which can be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the steps of a text-driven 3D scene stylization method.

[0090] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system, device, or unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0091] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0092] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A text-driven 3D scene stylization method, characterized in that, include: A learnable parameter set is obtained, and an adaptive radial basis function kernel set is defined based on the learnable parameter set, wherein the learnable parameter set includes position and shape; the kernel capacity of the adaptive radial basis function kernel set is expanded using a multi-frequency sine synthesis algorithm to obtain an optimized radial basis function kernel set; a coarse-grid backbone network is constructed, which forms a complementary feature extraction branch with the optimized radial basis function kernel set, and a neural network decoder is set to complete the construction of the hybrid radial basis field representation architecture; The process involves: acquiring multi-view images of a real-world scene and a preset weighting rule; obtaining multiple initial 3D sampling points based on the multi-view images, and assigning weights to each initial 3D sampling point according to the weighting rule to obtain weighted 3D sampling points; using a clustering algorithm to cluster the weighted 3D sampling points to obtain multiple cluster centers; and initializing the position and shape of the optimized radial basis function kernel group based on the multiple cluster centers to construct an initial style radial basis function model. The initial style radial basis function model is trained through multiple rounds of iteration to obtain the style radial basis function model; Obtain the multi-dimensional loss, and fine-tune the style radial basis function model based on the multi-dimensional loss to obtain the target style radial basis function model; Obtain text instructions, and based on the text instructions, drive the target style radial basis function model to generate a 3D scene stylized rendering result.

2. The text-driven 3D scene stylization method according to claim 1, characterized in that, The learnable parameter set further includes feature weights, and the step of performing multiple rounds of iterative training on the initial style radial basis function model to obtain the style radial basis function model includes: Camera light data is obtained based on multi-view image sampling of the actual scene, and actual color data corresponding to the camera light data is acquired. The camera light data is input into the initial style radial basis function model for volume rendering to obtain predicted color data; The model reconstruction loss is calculated based on the actual color data and the predicted color data. Based on the model reconstruction loss, the feature weights of the optimized radial basis function kernel set of the initial style radial basis function model are iteratively updated using the backpropagation algorithm; When the preset iteration stopping condition is met, the style radial basis function model is output.

3. The text-driven 3D scene stylization method according to claim 2, characterized in that, The step of obtaining a multi-dimensional loss and fine-tuning the style radial basis function model based on the multi-dimensional loss to obtain a target style radial basis function model includes: The camera light data is input into the style radial basis function model for volume rendering to obtain stylized predicted color data, and the stylized predicted color data is stitched together to obtain a stylized scene image. Obtain the target text corresponding to the stylized scene image and the fixed text corresponding to the multi-view image of the actual scene; The pre-trained CLIP model is invoked, and multi-dimensional loss is calculated based on the stylized scene image, the real scene multi-view image, the target text, and the fixed text to obtain the multi-dimensional loss. Based on the multi-dimensional loss, the backpropagation algorithm is used to iteratively update the feature weights of the pre-trained radial basis function kernel set of the style radial basis function model; When the preset iteration stopping condition is met, the target style radial basis function model is output.

4. The text-driven 3D scene stylization method according to claim 3, characterized in that, The CLIP model includes a text encoder, an image encoder, and a multi-dimensional loss calculation module. The text encoder and the image encoder are respectively connected to the multi-dimensional loss calculation module. The pre-trained CLIP model is invoked to perform multi-dimensional loss calculation based on the stylized scene image, the real-world multi-view image, the target text, and the fixed text to obtain the multi-dimensional loss, including: Based on the style radial basis function model, a pre-trained radial basis function kernel set is obtained. The pre-trained radial basis function kernel set is subjected to camera plane projection processing to obtain an elliptical coverage region. A scene saliency map is generated based on the elliptical coverage region. Based on the scene saliency map, a bias sampling rule is constructed. The stylized scene image is randomly sampled based on the bias sampling rule to obtain a set of stylized local image patches. The original set of local image patches corresponding to the set of stylized local image patches is obtained from the multi-view image of the actual scene. Based on the text encoder, the target text and the fixed text are respectively encoded to obtain the target text embedding vector and the fixed text embedding vector; Based on the image encoder, the stylized scene image, the real scene multi-view image, the stylized local image block set and the original local image block set are respectively image encoded to obtain the stylized image embedding vector, the original image embedding vector, the stylized image block embedding vector and the original image block embedding vector; The multi-dimensional loss is obtained by performing multi-dimensional loss calculation based on the target text embedding vector, the fixed text embedding vector, the stylized image embedding vector, the original image embedding vector, the stylized image patch embedding vector, and the original image patch embedding vector using the multi-dimensional loss calculation module.

5. The text-driven 3D scene stylization method according to claim 4, characterized in that, The multi-dimensional loss calculation module includes a relative directionality loss calculation submodule, a global contrast loss calculation submodule, a local contrast loss calculation submodule, and an auxiliary loss calculation submodule. The relative directionality loss calculation submodule, the global contrast loss calculation submodule, and the local contrast loss calculation submodule are respectively connected to the text encoder and the image encoder. The auxiliary loss calculation submodule is connected to the image encoder. The multi-dimensional loss is obtained by performing multi-dimensional loss calculation based on the target text embedding vector, the fixed text embedding vector, the stylized image embedding vector, the original image embedding vector, the stylized image patch embedding vector, and the original image patch embedding vector, including: The relative directionality loss calculation submodule calculates the relative directionality loss based on the target text embedding vector, the fixed text embedding vector, the stylized image embedding vector, and the original image embedding vector; The global contrast loss calculation submodule calculates the global contrast loss based on the target text embedding vector, the stylized image embedding vector, and the original image embedding vector; The local contrast loss calculation submodule calculates the local contrast loss based on the stylized image patch embedding vector and the original image patch embedding vector; The auxiliary loss calculation submodule calculates the auxiliary loss based on the stylized scene image and the multi-view image of the actual scene. Obtain the preset weight coefficient setting rules, and perform a weighted summation of the relative directional loss, the global contrast loss, the local contrast loss, and the auxiliary loss based on the weight coefficient setting rules to obtain the multi-dimensional loss.

6. A text-driven 3D scene stylization device, characterized in that, include: Architecture building module: used to obtain a set of learnable parameters, define an adaptive radial basis function kernel set based on the set of learnable parameters, wherein the set of learnable parameters includes position and shape; use a multi-frequency sine synthesis algorithm to expand the kernel capacity of the adaptive radial basis function kernel set to obtain an optimized radial basis function kernel set; construct a coarse mesh backbone network, wherein the coarse mesh backbone network and the optimized radial basis function kernel set form complementary feature extraction branches, and set a neural network decoder to complete the construction of the hybrid radial basis field representation architecture; Model building module: used to acquire multi-view images of the actual scene and preset weight allocation rules; acquire multiple initial 3D sampling points based on the multi-view images of the actual scene, and assign weights to each initial 3D sampling point according to the weight allocation rules to obtain weighted 3D sampling points; use a clustering algorithm to cluster the weighted 3D sampling points to obtain multiple cluster centers; initialize the position and shape of the optimized radial basis function kernel group based on the multiple cluster centers to construct an initial style radial basis function model; Model training module: used to perform multiple rounds of iterative training on the initial style radial basis function model to obtain the style radial basis function model; Model fine-tuning module: used to obtain a pre-constructed multi-dimensional loss constraint system, and fine-tune the style radial basis function model based on the multi-dimensional loss constraint system to obtain the target style radial basis function model; Instruction-driven module: used to acquire text instructions and drive the target style radial basis function model to generate 3D scene stylized rendering results based on the text instructions.

7. A text-driven 3D scene stylization device, characterized in that, The text-driven 3D scene stylization device includes: a memory and at least one processor, wherein the memory stores instructions; At least one of the processors invokes the instructions in the memory to cause the text-driven 3D scene stylization device to perform the steps of the text-driven 3D scene stylization method as claimed in any one of claims 1-5.

8. A computer-readable storage medium storing instructions thereon, characterized in that, When the instructions are executed by the processor, they implement the steps of the text-driven 3D scene stylization method as described in any one of claims 1-5.

Images