General dynamic texture generation method and system

Abstract

The invention provides a general dynamic texture generation method and system, and the method comprises the steps: separating a spatial-temporal feature from input data, constructing a spatial feature and a time feature of a texture, and carrying out the fusion of the spatial feature and the time feature, and obtaining a fusion feature; dynamically selecting a physical simulation or statistical modeling path according to a fusion feature type obtained by fusion, and generating a controllable texture evolution rule; and in combination with the generative adversarial network and the neural radiation field, processing the texture evolution rule to generate a visual consistent and physically credible dynamic texture sequence, and outputting a dynamic texture adaptive to the target scene. The method supports the generation of various dynamic textures, is high in coupling, effectively reduces the time cost of repeated modeling, solves the pain points of'fidelity-efficiency paradox 'and'physical-art balance problem' in the existing dynamic texture generation field, and has the advantages of short single-lens manufacturing period, high physical rationality standard-reaching rate and cross-scene multiplexing.

Description

Technical Field

[0001] The present invention application relates to the technical field of three-dimensional virtual animation, and particularly relates to a general dynamic texture generation method and system. Background Art

[0002] In animation production, dynamic textures are generally applied to the following things: (1) Character clothing. Through dynamic texture technology, realistic dynamic effects can be added to the character's clothing, such as fluttering hems, swaying scarves, etc., enhancing the realism and vividness of the character. (2) Character skin. Dynamic texture technology can also be applied to the detailed performance of the character's skin, such as subtle wrinkle changes, muscle tremors, etc., making the character more vivid. (3) Environmental decoration. In the scene model, dynamic textures are often used for environmental decoration elements, such as fluttering flags, swaying leaves, flowing water surfaces, burning flames, magic special effects, etc. These dynamic effects can significantly enhance the level of layering and realism of the scene.

[0003] Existing methods for generating dynamic textures include physical simulation methods, statistical model methods, and video texture synthesis methods. Among them, the physical simulation method directly simulates the process of generating natural scenes through physics, and can accurately simulate real physical phenomena (such as water flow, smoke diffusion); the statistical model method regards the dynamic texture as a stationary random process in time and uses a statistical model to describe it, which is suitable for capturing the random characteristics of the texture sequence (such as flame flicker); the video texture synthesis method can synthesize new videos of finite or infinite length through the learning and processing of a given dynamic texture video sample. This method does not require knowledge of the physical laws and interactions of specific phenomena and can generate cyclic textures (such as wave motion).

[0004] Each of the above three generation methods has its own disadvantages, which are respectively: (1) The physical simulation method has a relatively large amount of calculation and high complexity (the single-frame rendering time exceeds 500 ms). Each method has a single application, and different scenarios require different physical process simulations. Moreover, it cannot generate non-physical phenomena such as magic special effects, and has poor universality. (2) When the dynamic texture appears blurred or noisy, the robustness of the statistical model method is easily affected, and its generalization ability is poor. It needs to be retrained for a single type of texture (the training data set needs to contain at least 10^4 frame samples). (3) When the texture change is complex, it is difficult to determine the jump points in the video texture synthesis method, which affects the synthesis effect. Moreover, it depends on a high-quality input video (resolution ≥ 4K), and the texture parameters (such as the RGB values of the flame color) cannot be adjusted in real time.

[0005] Meanwhile, the coupling of the above three generation methods is poor. Especially in the production of scene models, when multiple dynamic textures are required in a scene (such as fluttering flags, flowing water surfaces, burning flames, and magic special effects), different generation method frameworks need to be switched to produce different dynamic textures. It is also impossible to adjust texture attributes (such as turbulence intensity, particle density) through a unified interface, and there may be rendering inconsistencies (such as light and shadow misalignment) when fusing dynamic textures with static scenes. Therefore, it greatly increases the production cost and time of scene models and affects the final effect of animations. Summary of the Invention

[0006] The purpose of the present invention is to address the above existing problems and deficiencies, and provide a general dynamic texture generation method and system to solve the technical problems such as poor coupling, long processing time, and poor synthesis effect of existing dynamic texture generation methods.

[0007] The technical solution of the present invention is realized as follows:

[0008] On the one hand, the present invention provides a general dynamic texture generation method, which is characterized by including: separating spatio-temporal features from the input data, constructing the spatial features and temporal features of the texture, and fusing the spatial features and temporal features to obtain a fused feature; dynamically selecting a physical simulation or statistical modeling path according to the type of the fused feature obtained by fusion to generate a controllable texture evolution rule; combining a generative adversarial network and a neural radiance field to process the texture evolution rule to generate a visually consistent and physically credible dynamic texture sequence, and outputting a dynamic texture adapted to the target scene.

[0009] In some embodiments, separating the spatio-temporal features from the input data and constructing the spatial static features and temporal dynamic features of the texture includes the following sub-steps: performing data preprocessing on the input data to form a unified data format and perform noise filtering; parsing the filtered unified data into spatio-temporally separated feature representations through a spatio-temporal separation convolutional network.

[0010] In some embodiments, fusing the spatial features and temporal features to obtain a fused feature includes the following sub-steps: obtaining the spatial gradient energy E spatial ; obtaining the temporal difference sparsity E temporal ; using quantum annealing optimization to generate a dynamic weight α, and its calculation formula is: where β is the annealing parameter and is dynamically adjusted through a simulated annealing algorithm; using the feature fusion formula F fusion =α·F spatial +(1-α)·F temporal to fuse the spatial features and the temporal features, where α is the dynamic weight, and F spatial represents the spatial feature, and F temporal represents the temporal feature.

[0011] In some embodiments, dynamically selecting a physical simulation path or a statistical modeling path according to the type of the fused feature to generate a controllable texture evolution rule includes the following sub-steps: inputting the fused feature obtained by fusion into a fully connected layer to generate a gating value g, where g ∈ [0, 1]; selecting a modeling method according to the threshold of the gating value g. Specifically, if g > 0.7, enable the physical simulation path; if g ≤ 0.7, enable the statistical modeling path; and generate a corresponding texture evolution rule according to the selected path.

[0012] In some embodiments, combining a generative adversarial network and a neural radiance field to process the texture evolution rule to generate a visually consistent and physically plausible dynamic texture sequence and output a dynamic texture adapted to the target scene includes the following sub-steps: inputting noise and the texture evolution rule into the generator of the generative adversarial network to obtain a basic texture sequence; calculating a local adversarial loss through the discriminator of the generative adversarial network to ensure the authenticity of texture details; performing multi-physical field enhancement on the basic texture sequence through the neural radiance field to synthesize a dynamic texture sequence with multi-view lighting consistency; synthesizing a dynamically textured sequence with optimized temporal consistency by constraining the frame-to-frame smoothness through an optical flow loss; and outputting a dynamic texture adapted to the target scene.

[0013] In some embodiments, after the step of combining a generative adversarial network and a neural radiance field to process the texture evolution rule to generate a visually consistent and physically plausible dynamic texture sequence, and before the step of outputting a dynamic texture adapted to the target scene, the following steps are further included: adjusting parameters in real time according to user feedback and online reinforcement learning.

[0014] Further, adjusting parameters in real time according to user feedback and online reinforcement learning includes the following sub-steps: implementing a semantic control interface for the user to adjust semantic parameters, which are mapped to underlying physical parameters for real-time modification; triggering online reinforcement learning according to the user score feedback to update and optimize the underlying physical parameters; generating a preview frame for viewing based on the adjustment result, and generating an optimized dynamic texture sequence after confirmation.

[0015] According to one aspect of the present application, a general dynamic texture generation system is further provided, which is characterized by including: a multi-modal feature extraction module for separating spatio-temporal features from the input data, constructing spatial and temporal features of the texture, and fusing the spatial and temporal features to obtain a fused feature; a hybrid dynamic texture analysis module for dynamically selecting a physical simulation or a statistical modeling path according to the type of the fused feature obtained by fusion to generate a controllable texture evolution rule; and a programmable texture synthesis module for combining a generative adversarial network and a neural radiance field to process the texture evolution rule to generate a visually consistent and physically plausible dynamic texture sequence and output a dynamic texture adapted to the target scene.

[0016] In some embodiments, an interactive optimization module is further included, which is used to adjust parameters in real time after user feedback and trigger online reinforcement learning, so as to optimize the generation of a dynamic texture sequence adapted to the target scene.

[0017] In some embodiments, the density field σ parameter in the rendering equation of the neural radiance field is parameterized, and physical quantities related to the dynamic texture type are incorporated to support various physically simulated dynamic textures.

[0018] The present invention has at least the following beneficial effects:

[0019] 1. Through the three core steps of spatio-temporal quantization feature decoupling, physical-statistical bimodal regulation, and generation-rendering joint optimization, the present invention systematically solves the pain points of the "fidelity-efficiency paradox" and the "physical-art balance problem" existing in the existing dynamic texture generation field. It has the advantages of a short single-shot production cycle, a high physical rationality compliance rate, and the generated dynamic textures can be reused across scenes. On the premise of ensuring cinema-level image quality, the production efficiency of dynamic textures is greatly improved.

[0020] 2. Since the present invention combines physical-statistical bimodal regulation and generation-rendering joint optimization, this method supports the generation of various dynamic textures (such as flames, water flows, smoke, lightning, magic particles, cloth fluttering, etc.), has strong coupling, and effectively reduces the time cost of repeated modeling.

[0021] 3. The method of the present invention also adds a step of adjusting parameters in real time according to user feedback and after online reinforcement learning. Therefore, it supports users to make real-time adjustments and ratings, and through online reinforcement learning optimization, the adjustment response strategy and feedback time can be shortened, and intuitive preview can be achieved, realizing the instant visualization of creative ideas.

[0022] The present invention will be further described below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 is a flowchart of the general dynamic texture generation method of the present invention Figure 1 ;

[0024] Figure 2 is a flowchart of the general dynamic texture generation method of the present invention Figure 2 ;

[0025] Figure 3 is a flowchart of the general dynamic texture generation method of the present invention Figure 3 ;

[0026] Figure 4 is a flowchart of the general dynamic texture generation method of the present invention Figure 4 ;

[0027] Figure 5 Flow chart of the general dynamic texture generation method of the present invention Figure 5 ;

[0028] Figure 6 Flow chart of the general dynamic texture generation method of the present invention Figure 6 ;

[0029] Figure 7 Architecture diagram of the general dynamic texture generation system of the present invention Figure 1 ;

[0030] Figure 8 Architecture diagram of the general dynamic texture generation system of the present invention Figure 2 。 Detailed implementation manners

[0031] Next, the accompanying drawings in the embodiments of the present invention will be combined to clearly and completely describe the technical solutions in the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Usually, the components of the embodiments of the present invention described and shown in the accompanying drawings here can be arranged and designed in various different configurations. Therefore, the detailed description of the following embodiments is not intended to limit the scope of the claims of the present invention, but only represents the selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts belong to the scope of protection of the present invention.

[0032] First, the technical background related to this application will be described:

[0033] Physical Constraint Embedding Layer (PCEL): A technology that embeds physical laws (such as fluid dynamics equations, heat conduction equations, etc.) into neural networks in a differentiable manner. It involves the operation of converting physical equations into neural network layers, so that the generated results not only conform to the statistical laws driven by data, but also satisfy physical rationality. For example, when generating fluid dynamic textures, PCEL may ensure that the generated fluid motion conforms to the basic physical conservation laws by embedding the Navier-Stokes equation.

[0034] Probabilistic Graphical Model (PGM): A model that represents the probabilistic relationships between variables using a graph structure. Common PGMs include Bayesian networks and Markov random fields. In dynamic texture generation, PGM may be used to model the evolution law of textures over time, such as describing the change pattern of textures through state transition probabilities.

[0035] Generative Adversarial Network (GAN): It includes a generative model G and a discriminative model D. The purpose of the generative adversarial network is to learn the distribution p of the training data g, To learn this distribution, first define an input noise variable p z (z). Next, map it to the data space G(z; θ g ). Here, G is a generative model composed of a multi-layer perceptron network parameterized by θ g . In addition, define a discriminative model D(x; θ d ) to determine whether the input data comes from the generative model or the training data. The output of D is the probability that x is training data. Finally, train D to determine the data source as accurately as possible, and train G to make the generated data conform to the distribution of the training data as much as possible.

[0036] Neural Radiance Field (NeRF): The volume rendering equation is a standard formula in the field of neural rendering. The σ in the equation is the density field, representing the density value of a point x in space at time t, which is a continuous function of space coordinates and time.

[0037] The general dynamic texture generation method provided by the present invention is mainly applied to a scene model construction device, which can be a server or a system in which a server and a terminal device cooperate with each other. Correspondingly, each part included in the scene model construction device, such as each unit, subunit, module, and sub-module, can be all set in the server or can be respectively set in the server and the terminal device.

[0038] Furthermore, the above server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers or as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules for providing a distributed server, or can be implemented as a single software or software module, which is not specifically limited herein. In some possible implementation manners, the general dynamic texture generation method of the embodiments of the present invention can be implemented by a processor calling computer-readable instructions stored in a memory.

[0039] Next, in conjunction with the accompanying drawings, the general dynamic texture generation method and system of the present invention will be described. The specific implementation method is as follows.

[0040] As Figure 1 shown, Figure 1 is a flowchart of the general dynamic texture generation method of the present invention Figure 1 . The general dynamic texture generation method may include:

[0041] S10. Separate spatio-temporal features from the input data, construct spatial features and temporal features of the texture, and fuse the spatial features and temporal features to obtain a fused feature.

[0042] Among them, the input data can be videos, simulation data, sensor signals, etc. The videos are real-shot or reference videos drawn by animators; the simulation data are physical field data (such as velocity, density, etc.) generated by fluid simulation software (such as Houdini); the sensor signals are dynamic parameters (such as wind speed, temperature, etc.) collected by motion capture devices or environmental sensors.

[0043] The present invention can adopt STS-CNN spatio-temporal separation network, 3D convolutional network, two-stream network, Transformer model, graph neural network, etc. to process spatio-temporal feature separation. Extract spatial features that do not change with time, such as shape, color, etc.; and dynamic time features, such as motion speed, periodicity, etc. from the input data. The spatio-temporal feature extraction of the present invention supports parallel extraction of physical and non-physical features, for example, simultaneously capturing the color gradient and diffusion rate of a flame.

[0044] The separated spatial features exist in a tensor structure, and the time features exist in a hidden state sequence, and then through the feature fusion formula F fusion =α·F spatial +(1-α)·F temporal for fusion, where α in the formula is a weight; F spatial represents the spatial feature, and its expression is where H′×W′ is the downsampled spatial resolution, and D s is the number of feature channels (usually 128 - 512 dimensions); F temporal represents the time feature, and its expression is where T′ is the time step, and D t is the hidden state dimension (usually 512 - 1024 dimensions). The α in this embodiment can be an adaptive weight generated by learnable parameters, rather than a fixed value (such as 0.5) in the traditional method or calculated by a simple heuristic rule. Therefore, it can optimize the learning strategy for different texture types; the α in this embodiment can also be a dynamic weight mechanism introduced with quantum annealing optimization, modeling the weight generation as an energy minimization problem, and at the same time combining the spatial feature gradient and the time feature difference sparsity constraint, so as to avoid the arbitrariness of manually setting α.

[0045] This embodiment fuses the parsed spatial features and temporal features to achieve collaborative modeling of spatiotemporal information, thereby generating more accurate and coherent dynamic textures that are both physically consistent and artistically expressive. For example, in smoke simulation, spatial features describe the shape and density distribution of smoke, and temporal features describe the diffusion speed and direction. After fusion, the movement of smoke can be simulated more realistically. The use of feature fusion can improve the model's ability to model complex dynamic textures and enhance the physical rationality and visual consistency of the generated results. Compared with the existing parallel dual-branch processing or cascade processing, the fusion processing of this embodiment not only has a fast processing speed, but also has strong spatiotemporal consistency, small physical errors, and can achieve two-way information transmission.

[0046] S20, dynamically selecting a physical simulation or statistical modeling path according to the fusion feature type obtained by fusion, and generating a controllable texture evolution rule.

[0047] The modeling method of this embodiment is selected adaptively, and the adaptive selection method can be realized by a gating mechanism, and the calculation of the gating value is realized by the input fusion feature, and the modeling method is selected according to the result of the gate control. For example, when the gating value is greater than a certain set value, the physical constraint embedding layer (PCEL) is enabled, otherwise the probabilistic graph model (PGM) is enabled, that is, the system dynamically selects whether to use a physical model or a statistical model according to certain attributes of the input fusion feature. Its advantage is that for textures that are obviously governed by physical laws (such as water flow, smoke diffusion, rain hitting the ground, etc.), PCEL can be used to ensure physical correctness; and for more abstract or artistic textures (such as irregular jumping of cartoon flames, surreal trajectories of magic particles, etc.), the flexibility provided by PGM can be relied on. Moreover, the gating mechanism can realize real-time interaction and fast calculation speed. In addition to using set values ​​to realize hard switching, soft mixing can also be realized, such as outputting g×PCEL+(1-g)×PGM, where g is the gating value. Soft mixing is suitable for dynamic types of basic physics + surreal enhancement, such as magic flames.

[0048] In addition, attention fusion, reinforcement learning strategy or multi-expert mixture (MoE) and other structures can also be used to realize the adaptive selection of modeling methods. Each method has advantages and disadvantages in the application of animation production, and can be selected according to actual needs, which is not specifically limited in this embodiment. For example, attention fusion can dynamically adjust the weights of different models instead of hard switching, which may be more flexible for processing complex animation effects, but the computational cost may be higher. For example, the reinforcement learning strategy needs to define the reward signal, which is suitable for scenarios that require multiple iterative adjustments, but the training complexity is high.

[0049] Among the texture evolution rules generated in this embodiment, the texture evolution rules generated based on the physical simulation path exist in the form of a differentiable computation graph, including: symbolic expressions of control equations, a set of physical parameters, and a user adjustment interface; the texture evolution rules generated based on the statistical modeling path exist in the form of a probability graph + conditional distribution, including: a state transition matrix, emission probabilities (implemented by a GAN decoder), and a semantic control mapping table.

[0050] S30. Combine the generative adversarial network and the neural radiance field to process the texture evolution rules to generate a visually consistent and physically plausible dynamic texture sequence, and output a dynamic texture adapted to the target scene.

[0051] This embodiment adopts a hybrid rendering pipeline that combines a generative adversarial network and a neural radiance field, realizing the enhancement of temporal consistency by inputting the basic dynamic sequence generated by the GAN into the NeRF. The GAN performs excellently in generating images with rich details but may have problems with temporal consistency, especially flickering or jitter when generating consecutive frames; while the NeRF is good at synthesizing high-fidelity 3D scenes from multiple perspectives, but may have a large computational amount and require a large amount of data in processing dynamic scenes. Therefore, combining the two can complement each other's deficiencies, thereby achieving the purpose of improving the quality and efficiency of dynamic textures.

[0052] The usage effects of the hybrid rendering pipeline are mainly reflected in three aspects: improving spatio-temporal consistency, enhancing physical rationality, and optimizing resource allocation. Specifically, the 3D radiance field modeling of the NeRF can eliminate the problem of view jumps (such as texture tearing when an object rotates). In the animation camera movement, the geometric consistency error of multi-view rendering is <0.5 pixels, achieving spatial consistency; at the same time, the frame-to-frame smoothness of adjacent frames is constrained through the optical flow loss to achieve temporal consistency and solve the problem of frame-to-frame jitter in existing GAN solutions (such as StyleGAN-V). Experiments show that through this constraint, the cumulative deformation error of a 120-frame sequence can be reduced by 82%; thus, double guarantees of spatio-temporal consistency are achieved. Input the rough motion field generated by the GAN into the NeRF, and calculate the change of the density field through a differentiable equation to realize the volume change based on physical laws and enhance physical rationality. Also, through the frequency domain analysis of the noise generated by the GAN, the NeRF anti-aliasing is only applied in the low-frequency region. At the same time, the low-resolution basic sequence (such as 512×512) generated by the GAN is super-resolved and detailed restored through the NeRF, that is, adaptive noise reduction and multi-resolution collaboration, to achieve a dynamic balance between calculation and quality and achieve the purpose of optimizing resource allocation.

[0053] Furthermore, when using the volume rendering equation (NeRF) in the present invention, the density field σ therein is not directly predicted by an MLP as in the standard NeRF, but the NeRF density field σ is dynamically adjusted to optimize the rendering efficiency. For example, the density field σ is calculated using the commonly used formula in the form of exponential decay, σ = 0.3·exp(-d 2 / 10), where d is the distance to the center of the dynamic texture. By introducing a distance attenuation term, the density field is forced to follow physical laws, thereby reducing the training time. For another example, a multi-physics coupling model of the turbulent energy dissipation rate (∈) and the temperature field (T) is introduced, and the formula for the density field σ is: where ∈ can be calculated from the vorticity field solved by the Navier-Stokes equation, σ0 is the base density coefficient, k d is the distance attenuation coefficient, λ is the weight coefficient, ∈0 is the reference energy dissipation rate, T0 is the reference temperature, and k d , λ, ∈0, and T0 are learnable parameters initialized by meta-learning (Meta-Learning); through this formula, physical dynamic textures involving turbulence and temperature, such as water flow, smoke, and magma, can be simulated more quickly. For another example, a dynamic physical field selector and a contribution function design can be introduced to automatically activate the corresponding physical field and contribution function according to the input texture type, thereby obtaining the density field σ. By parameterizing the physical field, various physical simulation dynamic textures are uniformly supported; moreover, the design of the contribution function retains domain knowledge and allows data-driven optimization. This dynamic adjustment of the NeRF density field σ supports diverse textures such as water flow, smoke, cloth, and magic particles.

[0054] After calculating the density field σ, based on this density field σ, the light propagation is calculated, and then a high-fidelity dynamic texture sequence can be generated and output to the target scene for adaptation.

[0055] The method of this embodiment systematically solves the pain points of the "fidelity-efficiency paradox" and the "physical-art balance problem" existing in the existing dynamic texture generation field through three core steps: spatio-temporal quantization feature decoupling, physical-statistical dual-mode regulation, and generation-rendering joint optimization. It has the advantages of a short single-shot production cycle, a high physical rationality compliance rate, and the generated dynamic textures can be reused across scenes. On the premise of ensuring cinema-level image quality, the production efficiency of dynamic textures is greatly improved. Moreover, this method supports the generation of various dynamic textures (such as flames, water flows, smoke, lightning, magic particles, cloth fluttering, etc.), has strong coupling, and effectively reduces the time cost of repeated modeling.

[0056] As Figure 2 shown, Figure 2 is the flow schematic of the general dynamic texture generation method of the present invention Figure 2 . This method refines the S10 step and includes:

[0057] S101. Preprocess the input data to form a unified data format and perform noise filtering.

[0058] The preprocessing of the input data in this embodiment includes format unification and data alignment. Format unification can perform format processing according to different data types. For example, video data is uniformly processed into tensors, simulation data is uniformly processed into velocity field grids, and sensor signals are uniformly processed into voxelized representations; or video, simulation data, and sensor signals can be unified into tensors. Support direct input of industrial sensors (such as Lidar point clouds). Data alignment processing includes interpolating and synchronizing inputs with different sampling rates (such as 30FPS video and 100Hz sensor data), registering multi-view inputs to a unified coordinate system (such as the world coordinate system of an animation scene). Data alignment processing can avoid the accumulation of fusion errors. Through this preprocessing, different modality data such as video, simulation, and sensors can be mapped to a unified mathematical representation, such as tensors or grids, providing a consistent input interface for the subsequent model, eliminating heterogeneity, avoiding the overhead of format conversion during runtime, improving computational efficiency, and moreover, the standardized tensor format can be directly docked with inference engines such as TensorRT / OpenVINO, shortening the deployment cycle.

[0059] After data preprocessing, a unified data format is formed, and then the data is subjected to noise filtering. Non-local means filtering (NL-Means) can be used to remove image noise, or Kalman filtering to smooth sensor data, or other noise filtering methods. Through noise filtering, the spatial gradient energy E of STS-CNN spatial The calculation error is reduced, improving the robustness of subsequent feature extraction. Moreover, noise filtering can reduce the workload of later manual correction. Clean data makes the light and shadow transition of NeRF volume rendering more natural, enhancing the artistic performance.

[0060] In addition to the above format unification and data alignment processing, other conventional preprocessings can also be included, such as spatio-temporal augmentation processing, physical rationality constraint enhancement processing, normalization and quantization processing, etc.

[0061] S102. Parse the filtered unified data into a spatio-temporally separated feature representation through a spatio-temporal separation convolutional network.

[0062] This embodiment uses the STS-CNN spatio-temporal separation network for data parsing. This spatio-temporal separation network includes a spatial path (3D-ResNet34) and a temporal path (LSTM+TCN), where 3D-ResNet34 is used to extract geometric and color features F spatial, such as the flame color gradient, the FFT frequency domain characteristics of the water surface ripples, etc., and the output feature map size is 256×256×128; capture the dynamic evolution law F through LSTM+TCN temporal , such as the flame spread rate, the angular velocity ω of the water flow vortex, the time window is set to 15 frames, and the hidden state dimension is 512.

[0063] S103. Obtain the spatial gradient energy E spatial and the time difference sparsity E temporal , and use quantum annealing optimization to generate the dynamic weight α.

[0064] The α of this embodiment introduces a dynamic weight mechanism optimized by quantum annealing, and its formula is:[[]] where β is the annealing parameter, which is dynamically adjusted by the simulated annealing algorithm, the initial value β = 0.5, and it decays by 5% per iteration step, E spatial is the spatial feature gradient energy, E temporal is the time difference sparsity, E temporal =||ΔF temporal ||1. The spatial gradient energy E spatial and the time difference sparsity E temporal are general feature descriptors. For example: the gradient energy of the flame reflects the change of the temperature field, and the time difference captures the flicker frequency; the gradient energy of the water flow characterizes the ripple curvature, and the time difference describes the flow velocity change; the gradient energy of the smoke corresponds to the density field gradient, and the time difference tracks the diffusion speed. Moreover, the calculation of the annealing parameter β and the energy difference E spatial -E temporal does not depend on the physical properties of specific textures, but is adaptively adjusted in a data-driven manner, so the universality of the quantum annealing optimization is high.

[0065] In order to explicitly support non-physical simulation dynamic textures (such as magic special effects), a multi-modal feature abstraction layer is further introduced. Therefore, the formulas for the spatial gradient energy E spatial and the time difference sparsity E temporal are respectively:[[]]

[0066]

[0067] E temporal =‖ΔF temporal ||1 + μ×Sparsity(F temporal )

[0068] where Entropy(F) is the Shannon entropy of the spatial feature, which quantifies the texture complexity (applicable to unstructured magic particles); To measure the sparsity of time features (applicable to special effects with strong randomness); λ and μ are learnable parameters initialized through meta-learning. By entropy and sparsity, generality can be enhanced to achieve unified processing of physical and non-physical textures.

[0069] S104. Adopt the feature fusion formula F fusion = α·F spatial +(1 - α)·F temporal to fuse the spatial feature and the time feature.

[0070] α in the feature fusion formula of this embodiment is the above-mentioned dynamic weight optimized by quantum annealing. Through the competitive optimization of spatial gradient energy (E spatial ) and time difference sparsity (E temporal ), the dynamic balance of feature fusion weights is achieved. Therefore, compared with the traditional weighted average method (such as the dynamic fusion network proposed in CVPR 2020), in the task of generating mixed materials, the feature alignment accuracy has been greatly improved. For example, in the generation of animated smoke, this mechanism can make the artistic style weight change dynamically, such as physical dominance in the initial stage of explosion and enhanced artistic effects in the later stage.

[0071] The method of this embodiment details the processes of data preprocessing, data parsing, and fusion. The method of this embodiment applies the quantum optimization idea to dynamic texture feature fusion. Therefore, the dynamic balance between physics and art can be achieved through weight adjustment, so as to flexibly meet the requirements of different scenarios, such as being biased towards physical simulation or artistic style, and is also more suitable for the generation of mixed materials. Moreover, the energy function design combines gradient and sparsity constraints, avoiding the randomness of manually setting α, improving spatio-temporal consistency, and reducing physical errors.

[0072] As Figure 3 shown, Figure 3 is the flow schematic Figure 3 of the general dynamic texture generation method of the present invention. This method refines the S20 step and includes:

[0073] S201. Input the fused feature obtained by fusion into the fully connected layer to generate a gating value g, where g ∈ [0, 1].

[0074] This embodiment is based on the dynamic switching strategy of the gating value to achieve seamless connection between physical and non-physical models. The gating value formula of this embodiment is g = σ(W g F fusion + b g ). Where W g is the weight matrix used to map F fusion to a scalar value, a learnable parameter, with a dimension of 1×10241×1024; F fusionis the fused feature vector passed in the S10 step, with a dimension of 1024; b g is the bias term, which is used to adjust the reference activation threshold of the gating value and is also a learnable parameter; σ is the Sigmoid function, which is used to compress the output gating value g into the interval [0, 1], and the formula is

[0075] This embodiment W g and b g In the learning and training stage, they are automatically updated along with the overall model through gradient descent, that is, they are automatically adjusted according to the characteristics of the input data (such as the turbulence intensity of the smoke and the artistic style of the flame) W g and b g , without presetting a fixed threshold. Thus, the gating value g can be dynamically adapted to the input feature F fusion , automatically select the optimal modeling path (physical constraint or statistical modeling); moreover, the learned W g and b g can be generalized to different animation scene models, thereby reducing the cost of repeated parameter tuning.

[0076] S202. Select a modeling method according to the threshold of the gating value g. Among them, if g > 0.7, enable the physical simulation path; if g ≤ 0.7, enable the statistical modeling path.

[0077] The valve threshold of this embodiment is set to 0.7. The design of the gating threshold of 0.7 is a balanced choice verified by experiments. The experimental results show that: when the threshold < 0.6, the physical path mis-trigger rate is high (such as misjudging the artistic flame as requiring physical simulation), resulting in a decrease in the generation efficiency. When the threshold > 0.8, the physical path activation is insufficient (such as ignoring the real fluid interaction), resulting in a distortion of the dynamic effect. Only when the threshold = 0.7, the optimal balance is achieved between physical rationality (error < 3%) and artistic freedom (user score > 8.5 / 10). Of course, in some embodiments, the system can support adjusting the threshold, such as global adjustment, directly modifying the threshold (such as setting it to 0.65 or 0.75) to adapt to different scene requirements; or conditional threshold: dynamically calculating the threshold based on the input feature: g thres = MLP(F fusion ), this method can further improve the adaptive ability in complex scenes (such as mixed physical-art special effects).

[0078] The setting of the enabled type in this embodiment is as follows: if g > 0.7, enable the Physical Constraint Embedding Layer (PCEL); if g ≤ 0.7, enable the Probabilistic Graphical Model (PGM). In addition, a soft mixing strategy can also be enabled, that is, the output is g×PCEL + (1 - g)×PGM. When using soft mixing, the loss function L = L recon + λ1L physical + λ2LKL , where \(L\) recon is the pixel / feature matching loss between the generated texture and the ground truth, and \(L\) physical is the residual of the physical equation (calculated only when the PCEL path is activated), and \(L\) KL is the regularization term of the latent variable distribution of the PGM. In this soft mixing process, initially the PGM can be fixed to learn the data distribution; in the middle stage, the PCEL is gradually introduced to strengthen the physical constraints through the \(\lambda_1\) scheduling; in the later stage, the gating network is opened to achieve joint optimization.

[0079] In animation production, dynamic selection is performed according to the characteristics of the input data. For example, if the input fusion feature \(F\) fusion belongs to realistic ocean waves, then the hydrodynamics needs to be strictly followed. Therefore, according to \(F\) fusion it will be calculated that \(g > 0.7\), such as \(g = 0.92\). At this time, the PCEL module will be selected; for another example, if the input \(F\) fusion belongs to the fluttering of a cartoon character's cape, which is a performance where artistic exaggeration takes precedence over physical accuracy. Therefore, according to \(F\) fusion it will be calculated that \(g < 0.7\), such as \(g = 0.35\). At this time, the PGM module will be selected; for another example, if the input \(F\) fusion is a magic flame, that is, it needs to conform to basic physics and be enhanced superrealistically. According to \(F\) fusion it will be calculated that \(g\) is close to \(0.7\), such as \(g = 0.68\). At this time, the soft mixing mode will be activated.

[0080] Through this gating decision, the dynamic selection of the modeling path is realized, improving flexibility and accuracy, breaking through the fixed ratio mixing strategy of existing hybrid modeling methods (such as HybridFluid in SIGGRAPH 2022), and maintaining dynamic simulation details while improving computational efficiency.

[0081] S203. Generate the corresponding texture evolution rules according to the selected path.

[0082] In this embodiment, when \(g > 0.7\), the physical constraint embedding layer is enabled, and the Navier - Stokes equation, heat conduction equation, etc. are simplified into differentiable forms and embedded in the forward propagation constraint of the neural network. The formula of this physical constraint embedding layer is: where \(u\) is the velocity field (vector field); \(\nu\) is the fluid viscosity coefficient, which is a user - adjustable parameter; \(f\) is the external force field, driven by the parameters input by the user.

[0083] When \(g\leq0.7\), the probability graph model (PGM) is enabled. In this embodiment, the PPO algorithm is used to optimize the Markov chain transition probability \(P\) ij, maximizing the likelihood with real data. In this embodiment, physical priors such as fluid incompressibility are introduced into the probabilistic graphical model (PGM). Traditional PGMs rely on statistical laws in the data and may implicitly learn certain patterns, but they cannot guarantee strict physical correctness. After introducing physical priors, the model is forced to comply with explicit physical laws, and can generate physically reasonable results even when there is noise or insufficiency in the data.

[0084] Taking fluid incompressibility as an example, then the transition probability P ij of this embodiment is: where S valid is the set of all legal states that satisfy . Assuming that the divergence of the velocity field u j corresponding to the state S j is non-zero then: And the probability of the legal state S j is re-normalized and assigned. When training with PGM in this embodiment, in addition to the data reconstruction loss, a divergence constraint term is added to force the generated fluid motion to satisfy incompressibility. At the same time, a differentiable Navier-Stokes solver is integrated into the decoder of the PGM to correct the generated results in real time.

[0085] The generation of texture evolution rules is divided into the rule generation of the physical simulation path (PCEL) and the rule generation of the statistical modeling path (PGM). The rule generation process of the physical simulation path (PCEL) is as follows: map the fused feature F fusion to a set of physical parameters, then substitute these parameters into the differentiable control equation, and then generate the velocity field evolution rules through a numerical solver (such as the finite volume method). The rule generation process of the statistical modeling path (PGM) is as follows: decompose the dynamic texture into discrete states, and then optimize the state transition matrix

[0086] The method of this embodiment details the dynamic selection of the modeling path through the gating value and the implementation process of the two modeling path modules. In this generation method, the probabilistic graphical model (PGM) and the physical constraint embedding layer (PCEL) are two core modules, which are responsible for capturing data-driven statistical laws and injecting physical prior knowledge respectively. Their collaborative work realizes the dynamic balance between artistic expressiveness and physical rationality. Moreover, by explicitly introducing physical priors, the PGM is upgraded from a pure data-driven statistical model to a physical-data hybrid model, solving the core problem that the generated results violate natural laws in traditional methods, making the dynamic texture more compliant with physical laws and enhancing visual realism; and it can reduce post-production manual correction and lower production costs.

[0087] For exampleFigure 4 As shown Figure 4 is a schematic flow of the general dynamic texture generation method of the present invention Figure 4 .

[0088] This method refines step S30, which includes:

[0089] S301. Input noise and texture evolution rules into the generator of the generative adversarial network to obtain a basic texture sequence.

[0090] In this embodiment, the generator of the generative adversarial network (GAN) adopts a U-Net structure. Inputting noise z~N(0,1) and the evolution rule, it outputs a basic texture sequence

[0091] S302. Calculate the local adversarial loss through the discriminator of the generative adversarial network to ensure the authenticity of texture details.

[0092] In this embodiment, the discriminator adopts PatchGAN to calculate the local adversarial loss to ensure the authenticity of texture details. The adversarial loss formula is where D is the discriminator; G is the generator; x is the real texture sample; z is the noise input to the generator.

[0093] S303. Perform multi-physical field enhancement on the basic texture sequence through the neural radiance field to synthesize a dynamic texture sequence with multi-view lighting consistency.

[0094] In this embodiment, the neural radiance field (NeRF) realizes multi-view lighting consistency. Specifically, it calculates the pixel color C(r) through the volume rendering equation: where T(t) is the transmittance, σ is the density field, and c is the color. In this embodiment, based on this volume rendering equation, the conventional density field σ is adjusted. A dynamic physical field selector and a contribution function design are introduced, and the corresponding physical field and contribution function are automatically activated according to the input texture type to calculate the density field σ. Specifically, the formula for dynamically adjusting the NeRF density field σ is: where, σ0 is the basic density coefficient, which controls the static density distribution without an external physical field, and its default value is fitted through training data; d is the spatial distance, representing the Euclidean distance from the current sampling point to the core area of the dynamic texture (such as the center of the flame, the origin of the water flow vortex); k d is the distance attenuation coefficient, which determines the speed at which the density decays with distance. The larger its value, the gentler the decay; λ i is the weight coefficient of the i-th physical field, which is learned through backpropagation and is used to balance the contributions of different physical fields to the density; among the above parameters, the parameters σ0, d, k d control the static geometric attenuation, and λ iDynamically adjust the coupling strength of multiple physical fields. And is the normalized contribution function of the i-th physical field, representing the influence functions of different physical / artistic effects, and each function corresponds to an independent action mechanism; P i is a physical quantity related to the dynamic texture type, such as for water flow: P1 = ∈ (turbulent energy), P2 = T (temperature); for smoke: P1 = k (diffusion coefficient), P2 = C (concentration); for cloth: P1 = τ (tension), P2 = v (wind speed); and so on.

[0095] Therefore, in the specific animation production process, the system will select the corresponding contribution function according to the relevant information in the input basic texture sequence:

[0096] ① For fluid-like textures, define where ∈ is the turbulent energy dissipation rate, which can be calculated from the vorticity field solved by the Navier-Stokes equation, ∈0 is the reference energy dissipation rate, T is the temperature field, T0 is the reference temperature, ∈0 and T0 are learnable parameters, and is the normalization parameter.

[0097] ② For cloth-like textures, define (the product of tension τ and wind speed v).

[0098] ③ For magic particles, define where E represents the energy field of magic particles, which is measured by the velocity field calculated; f represents the modulation frequency, which controls the rate of periodic change of energy. This formula maps the energy field E to the periodic fluctuation of the density field through a sine function, thus simulating the oscillation characteristics of magic special effects (such as pulsed light spheres, energy ripples).

[0099] After calculating each physical field separately, then perform the superposition of multiple physical fields to achieve the enhancement of multiple physical fields. Finally, combine with the basic density distribution to obtain the density field σ. This density field σ is essentially a continuous function of spatial coordinates and time. Substituting this density field σ into the volume rendering equation can calculate the pixel color C(r), thereby generating a dynamic texture sequence with consistent multi-view illumination. As can be seen from the above, by introducing a dynamic physical field selector and contribution function design, the compatibility of static geometry and dynamic physical fields is achieved, thus supporting diverse textures such as water flow, smoke, cloth, magic particles, etc.

[0100] S304. Constrain the inter-frame smoothness through the optical flow loss, and synthesize a dynamically optimized texture sequence with temporal consistency.

[0101] This embodiment constrains the inter-frame smoothness of adjacent frames through the optical flow loss where W is the optical flow estimation network, such as the RAFT network; σ t ,σ t+1is the NeRF density field for adjacent frames t and t+1; F temporal is the dynamic feature extracted by the time path. By introducing the flow loss, the problem of inter-frame jitter after GAN processing is solved, and the motion smoothness index is effectively improved. After processing by the flow loss, the final output is a synthesized texture sequence where 4 represents the RGBA channels (red, green, blue, transparency). For example, a 4K transparent background flame sequence with 60 frames (T = 60, H = 2160, W = 3840) is generated.

[0102] S305. Output the dynamic texture adapted to the target scene.

[0103] The output dynamic texture resource file adapted to the target scene can be an EXR sequence, a GPU texture map, etc. Then bind the generated texture sequence (such as an EXR file) to the material sphere of the 3D model (such as Unity's Material or Blender's Shader Node), associate the transparency channel to achieve hybrid rendering; then upload the texture to the GPU video memory through the graphics API (such as OpenGL) as a dynamic texture map for real-time rendering. In the rendering loop, switch the texture frame according to the time stamp to achieve the dynamic effect (such as updating one frame every 1 / 60 second); if the texture is related to physical simulation (such as water flow affecting the movement of the character), then input the texture data into the physical engine (such as PhysX) to drive the particle behavior.

[0104] The method of this embodiment details the process of combining a generative adversarial network (GAN) with a neural radiance field (NeRF) for dynamic texture sequence processing, where a dynamic density field σ is introduced in NeRF to improve the generality of the dynamic texture processing of this method. Through the deep fusion of the strong generation ability of GAN and the physical-geometry constraints of NeRF, the complementary advantages of both can be fully utilized to achieve the collaborative enhancement of high-fidelity temporal consistency and physical rationality.

[0105] As Figure 5 shown, Figure 5 is the flow chart of the general dynamic texture generation method of the present invention Figure 5 . The general dynamic texture generation method may include:

[0106] S100. Separate the spatio-temporal features from the input data, construct the spatial features and temporal features of the texture, and fuse the spatial features and temporal features to obtain the fused features;

[0107] S200. Dynamically select the physical simulation or statistical modeling path according to the type of the fused features obtained by fusion, and generate a controllable texture evolution rule;

[0108] S300. Combine a generative adversarial network with a neural radiance field to process the texture evolution rules and generate a visually consistent and physically plausible dynamic texture sequence;

[0109] S400. According to user feedback and after online reinforcement learning, adjust the parameters in real time and output a dynamic texture adapted to the target scene.

[0110] Based on the first embodiment, this embodiment adds an interactive optimization method. This interaction method can be that after the system automatically processes and obtains a dynamically textured sequence with temporal consistency optimization, the user can perform interactive adjustment; it can also be that real-time interactive adjustment is performed in each processing step of the system; it can also be a combination of both methods, so that the producer can perform real-time interaction in each processing step, and the director can control every detail of the dynamic texture after its generation, making the dynamic texture more in line with the director's needs.

[0111] The user interaction interface of this embodiment can be a conventional physical parameter adjustment interface, such as directly adjusting the fluid viscosity coefficient ν, external force field f, etc. in the physically constrained embedding layer. It can also be a semantic interface, that is, an interaction interface that converts complex underlying technical parameters (such as fluid viscosity, optical flow velocity) into intuitive terms that the director can understand. This interaction interface can be a slider, a touch panel, gesture recognition, etc., so that the director can freely control through semantic parameters without having to understand the underlying parameter logic. For example, when making magic flames, the director drags the "rage level" slider to 8, and the system automatically enhances the external force field and reduces the viscosity to generate violently shaking flames without knowing the specific values of viscosity or external force; another example is quickly drawing a circle on the touchpad to map to parameters such as the angular momentum, turbulence intensity, and particle emission rate of a tornado. The faster the director's gesture, the more violently the tornado rotates; the larger the circle drawn by the gesture, the wider the influence range. It can also be a scoring system where the user rates the generated dynamic texture result (such as 1 - 5 stars), and the system converts the score into an optimization signal to optimize the dynamic texture.

[0112] In addition to user feedback, online reinforcement learning is also provided. Therefore, according to the user's semantic control, scoring and other feedback, implicit optimization can be achieved to autonomously adjust the underlying parameters; it can also solve the complex parameter coupling problem not covered by semantic control and adapt to the user's dynamic needs in real time.

[0113] This embodiment adds a user interaction step to the method of the first embodiment, making the dynamic texture generation method more flexible, supporting users to make real-time adjustments and ratings, and optimizing through online reinforcement learning, so as to shorten the adjustment response strategy and feedback time, and enable intuitive preview and instant visualization of creative ideas, enabling the director's ideas to be more clearly reflected.

[0114] As Figure 6 shownFigure 6 Schematic diagram of the general dynamic texture generation method of the present invention Figure 6 . This method refines step S400 and includes:

[0115] S4001. Implement user adjustment of semantic parameters through a semantic control interface and map them to real-time modification of underlying physical parameters.

[0116] User interaction can run through the entire dynamic texture generation process. For example, adjusting the PCA principal components of F spatial to control the spatial dimension; or adjusting the LSTM hidden state of F temporal to control the time dimension. For example, the gating threshold is adjustable, and the animator makes a trade-off according to the specific scene (such as realism vs. stylization). For example, in some scenes, by reducing the threshold from 0.7 to 0.65, the physical simulation accuracy of volcanic lava can be improved instead. Another example is adding an external force field f to the Navier-Stokes equation, which is the user control item. The animator can adjust this parameter to modify the evolution rule in real time. Another example is in the rule generation process of the statistical modeling path (PGM). When the transition probability P ij is obtained by optimizing the state through policy gradient, a user control item can be added to achieve artistic control. For example, increasing the randomness intensity a to adjust the transition probability: It is also possible to let NeRF update the volumetric lighting in real time by modifying the latent vector z of the GAN.

[0117] This embodiment uses a semantic control interface to implement the mapping with physical parameters or equations, so that users can directly adjust physical semantic parameters such as "viscosity" and "turbulence intensity", or statistical semantic parameters such as "style intensity" and "randomness", without having to pay attention to the underlying implementation logic and underlying parameters. Through this explicit rule representation and semantic control interface, the animation production side can accurately control every evolution detail of the dynamic texture, breaking through the bottleneck of the fuzzy "parameter adjustment - visual effect" association in traditional methods.

[0118] The semantic control interface of this embodiment is multimodal, such as sliders, gesture recognition, touchpads, drop-down menus, etc. Therefore, appropriate modalities can be selected for operation according to different requirements, supporting the director to directly "sculpt" the special effect form in the preview window. For example, when the user adjusts the "flicker frequency" through a slider, the system actually modifies the underlying parameters of the emission rate of the particle system, the periodic change of the texture color, and the temporal perturbation intensity of the optical flow field; for another example, when selecting the "ink painting style" preset in the drop-down menu, the system actually modifies the underlying parameters such as reducing the color saturation, increasing the brushstroke noise, and enabling the diffusion equation constraint. Therefore, the semantic control interface can convert technical parameters into the language of artists, lower the usage threshold, and endow animators with artistic control rights at the physical level of precision. Moreover, combined with the spatio-temporal separable convolutional network, a spatio-temporal feature representation system with high decoupling and strong semantics is realized, enabling animators to independently control the visual style (spatial features) and motion laws (temporal features) in dynamic textures just like layer-by-layer editing of Photoshop layers, completely changing the dilemma of "changing one thing and affecting the whole" in traditional animation production.

[0119] This embodiment also includes the function of online reinforcement learning. Introducing online reinforcement learning in dynamic texture generation not only shortens the response time but also enables real-time rendering and preview, supporting the director's instant feedback. At the same time, it can also make up for the limitations of semantic parameters. Since semantic parameters only adjust the explicit parameters of the preset mapping, while the actual generated effect may be affected by implicit factors (such as the coupling effect of material reflectivity and motion speed), and these un directly controlled hidden variables can be discovered and optimized through online reinforcement learning; at the same time, users may form complex preferences during multiple adjustments, and online learning can capture and record these preferences in real time.

[0120] S4002. Trigger online reinforcement learning according to the user's rating feedback to update and optimize the underlying physical parameters.

[0121] This embodiment also includes a user rating system. The rating interaction interface of this rating system can be star rating, such as 1 - 5 stars (1 star is the worst, 5 stars is the best); it can also be slider rating, such as a continuous value rating from 0 - 100 points; it can also be a simple binary feedback of "like" or "dislike". This rating interaction interface can be embedded in the sidebar of the preview window to support real-time rating. The user rating R is quantified as a part of the reward function and combined with other constraints (such as physical rationality) to form a comprehensive reward: r t = α·R user + β·R physics + γ·R style , where R user is the user rating, normalized to the interval [-1, 1]; R physics is the physical constraint reward, such as the negative value of the fluid divergence error; R styleFor style consistency rewards, such as the VGG feature distance from the preset style. Then, online reinforcement learning updates the generator parameters based on this reward information to drive model optimization. If the score is high, the current parameter direction is retained; if the score is low, the current parameter combination is penalized and the parameters are updated. In this dynamic texture generation system, it can convert the user's subjective feedback into a quantifiable reward signal through the user scoring system to drive the continuous improvement of the generation model.

[0122] The online reinforcement learning in this embodiment uses the DDPG algorithm (Deep Deterministic Policy Gradient). The DDPG is a reinforcement learning algorithm suitable for continuous action spaces and is suitable for optimizing real-time parameters. After the user rates or adjusts the semantic parameters each time, new training samples can be generated immediately, and the parameter update can be completed in a short time through the dual-network structure, the policy network (Actor) and the value network (Critic). Among them, the Actor network is responsible for generating parameter adjustment actions, taking the current state s t , and outputting the deterministic action a t = μ(s t |θ μ ), that is, the parameter adjustment strategy; the Critic network is used to evaluate the expected reward of the current state and action, taking the state s t and the action a t , and outputting the Q value Q(s t , a t |θ Q ) to evaluate the expected cumulative reward of the current action; at the same time, both the Actor and the Critic have a target network with delayed updates (θ μ′ , θ Q′ ) for stable training.

[0123] Furthermore, the DDPG optimization process driven by user scoring includes: ① User interaction and data collection, such as the user adjusting semantic parameters (such as dragging the "burning rate" slider to 80%) or rating the generated result (such as 3 stars); then the system responds to the user's operation, such as recording the current state s t (parameter configuration), action a t (user adjustment), reward r t (rating) and the next state s t+1 , and forming these data into a tuple (s t , a t , r t , s t+1 ) and storing it in the experience replay buffer D; then the Actor network generates an action according to the current state s tGenerate action suggestions, but if the user makes actual adjustments, overwrite this action. ② The Critic network randomly samples a small batch of data (s t , a t , r t , s t+1 ) from the buffer D, and then calculates the target Q-value: y i = r i + γ·Q′(s i+1 , μ′(s t+1 ))), where Q′ and μ′ are the outputs of the target network, and γ is the discount factor (usually taken as 0.99); then perform loss optimization and update the Critic parameter θ Q through backpropagation. The Critic network learns how to accurately evaluate the value of actions by minimizing the difference between the predicted Q-value and the target Q-value. ③ The Actor maximizes the Q-value through gradient ascent and updates the policy parameter θ μ : The gradient comes from the Q-value gradient of the action by the Critic, guiding the Actor to generate actions with higher rewards. ④ Soft-update the target network parameters θ μ′ and θ Q′ , θ μ′ ← τθ μ + (1 - τ)θ μ′ , θ Q′ ← τθ Q + (1 - τ)θ Q′ , where τ is the soft-update coefficient (such as 0.001), controlling the smoothness of parameter updates.

[0124] S4003. Generate a preview frame of the adjustment result for viewing. After confirmation, generate an optimized dynamic texture sequence.

[0125] Use the updated Actor network to generate a new parameter a t+1 = μ(s i+1 ). Finally, render the updated dynamic texture and display it to the user. The user can continue to adjust or rate, and continue the above steps to form a "interaction - learning - generation" closed loop. Finally, obtain a satisfactory dynamic texture sequence, bind the dynamic texture sequence to the target scene material, and output it to the rendering pipeline through a graphics API (such as Vulkan), supporting multi-platform deployment of OpenGL / WebGL.

[0126] The method of this embodiment details two forms of user interaction: the implementation methods of semantic control and rating feedback, and combines online reinforcement learning to make the real-time performance and adaptability of dynamic texture optimization stronger, and can continuously optimize the policy through online learning. Moreover, it can be visually previewed to achieve instant visualization of creative ideas.

[0127] To facilitate the better implementation of the general dynamic texture generation method provided by the present invention, the present invention also provides a system based on the above general dynamic texture generation method. The meanings of the nouns are the same as those in the above general dynamic texture generation method, and the specific implementation details can be referred to the description in the method embodiments.

[0128] As Figure 7 , Figure 7 is the architecture schematic of the general dynamic texture generation system of the present invention Figure 1 , and the general dynamic texture generation system includes a multi-modal feature extraction module 51, a hybrid dynamic texture analysis module 52, and a programmable texture synthesis module 53.

[0129] Among them, the multi-modal feature extraction module 51 is used to separate spatio-temporal features from the input data, construct the spatial and temporal features of the texture, and fuse the spatial and temporal features to obtain a fused feature; the hybrid dynamic texture analysis module 52 is used to dynamically select a physical simulation or statistical modeling path according to the type of the fused feature obtained by fusion, and generate a controllable texture evolution rule; the programmable texture synthesis module 53 is used to combine a generative adversarial network and a neural radiance field to process the texture evolution rule to generate a visually consistent and physically credible dynamic texture sequence, and output a dynamic texture adapted to the target scene.

[0130] The multi-modal feature extraction module 51 of the present invention adopts a spatio-temporal separation convolutional network to separately extract the spatial features (such as shape, color) and temporal features (such as motion speed, periodicity) of the dynamic texture, so it can support the parallel extraction of physical and non-physical features, for example, simultaneously capturing the color gradient (space) and diffusion rate (time) of the flame. The hybrid dynamic texture analysis module 52 combines a probabilistic graphical model (PGM) and a physical constraint embedding layer (PCEL). PGM models the texture state transition (such as the flame flickering frequency), and the physical constraint layer injects prior knowledge (such as the incompressibility of the fluid), and can adaptively select the modeling method (physical or statistical) through the gating value. The programmable texture synthesis module 53 is a hybrid architecture based on a generative adversarial network (GAN) and a neural radiance field (NeRF) to achieve 2D / 3D texture output. The system of the present invention supports the generation of multiple types of dynamic textures such as flames, water flows, smokes, magic special effects, etc., has strong versatility, and can reduce the repeated training cost through hybrid modeling.

[0131] Furthermore, as Figure 8As shown in the figure, this embodiment further includes an interactive optimization module 54, which is used to adjust parameters in real time after user feedback and trigger online reinforcement learning to optimize the generation of a dynamic texture sequence adapted to the target scene. A semantic control and a scoring mechanism are introduced into the interactive optimization module 54, and combined with the Deep Deterministic Policy Gradient (DDPG) algorithm, so that the generation policy can be dynamically optimized in real time according to the user's semantic control and scoring feedback, while generating a preview and recording preferences, thus improving the efficiency and effect of dynamic texture optimization, and finally outputting a dynamic texture adapted to the scene.

[0132] The system and method of the present invention will be further described below with three application cases.

[0133] Application Case 1: Generation of realistic water flow special effects (physical simulation path)

[0134] S110. Multi-modal feature extraction. 1.1) Input data preprocessing: The data sources include a real waterfall video (4K@60fps, H.264 encoding) and fluid simulation data (particle velocity field grid simulated by Houdini FLIP); the normalization process includes converting the video into a tensor Converting LIP data into a velocity field grid Noise filtering includes removing raindrop noise from the video using non-local means filtering (NL-Means), and smoothing particle jitter of FLIP data through Gaussian filtering. 1.2) Spatiotemporal Separation Convolutional Network (STS-CNN): The spatial path uses 3D-ResNet34 to extract the geometric structure features of the waterfall The key features involved include the gradient of the water curtain edge and the roughness of the rock surface; the temporal path uses TCN-LSTM to capture the dynamic law of water flow The key feature involved is the water flow impact frequency (peak frequency f = 2.3Hz). 1.3) Quantum annealing feature fusion: Calculate the spatial gradient energy E spatial and the temporal difference sparsity E temporal , where λ takes the value of 0.1 and μ takes the value of 0.05 through learning; the dynamic weight Substitute a into the feature fusion formula to finally obtain the fused feature Biased towards physical properties.

[0135] S120. Hybrid dynamic texture analysis. 2.1) Gated decision-making: Calculate the gate value g =

[0136] σ(W g F fusion +b g ) = 0.91 (threshold > 0.7); Path selection, enable the Physical Constraint Embedding Layer (PCEL). 2.2) Physical constraint path execution: Inject using a simplified Navier-Stokes equation Parameter prediction ν = 0.002; Boundary condition binding, set the rock geometry as a no-slip boundary.

[0137] S130, Programmable texture synthesis. 3.1) GAN-based generation: The generator inputs noise z ∼ N(0,1) + physical parameters ν, f; The U-Net outputs the basic water flow sequence 3.2) NeRF physical field enhancement: Density field correction ∈0 = 0.1; Volume rendering, ray marching to calculate refraction and caustic effects, the rendering time per frame is reduced from 12 seconds to 3.2 seconds (4K resolution). 3.3) Optical flow consistency constraint: Loss calculation Synthesize and output the water flow sequence (RGBA + depth).

[0138] S140, Interactive optimization. 4.1) Director feedback: Use the semantic parameter adjustment method to increase the "water flow impact force" from 70% to 85%; The score is 4 / 5 stars (reward R = 0.8). 4.2) DDPG online update: Calculate the target Q value y through the Critic network i = 0.8 + 0.99·Q′(s i+1 ,μ′(s t+1 )); The gradient update direction of the Actor network is to increase the vortex strength Δω = +0.12. 4.3) Real-time rendering: Generate water flow with the updated parameters, and the turbulence details are improved by 23% (based on PSNR 38.2 → 41.5), and the director confirms the output.

[0139] Application case 2: Cartoon flame special effect generation (statistical modeling path)

[0140] S210, Multimodal feature extraction. 1.1) Input data preprocessing: The data sources include hand-drawn flame animations (24fps, Adobe Animate vector sequence) and thermal infrared sensor data (temperature field time series); The standardization method is to convert the vector animation into a tensor (RGBA), and interpolate and align the temperature field data to the animation timeline. 1.2) Spatiotemporal separation convolutional network (STS-CNN): The spatial path extracts the flame edge jitter feature (stroke frequency f stroke = 5Hz); The temporal path extracts the "jump-shrink" cycle of the hand-drawn flame (cycle T = 0.8 seconds). 1.3) Quantum annealing feature fusion: The fusion weight α = 0.37, significantly biased towards artistic features; Substitute a into the feature fusion formula and finally obtain and output the fusion feature

[0141] S220, hybrid dynamic texture analysis. 2.1) Gating decision: g = 0.43 (threshold < 0.7), enable probabilistic graphical model (PGM). 2.2) Statistical modeling path execution: define s1 as the "flame shrinking" state (color saturation 0.8); define s2 as the "flame bursting" state (saturation 0.95, brightness +30%); learn the transfer probability P 12 =0.68.

[0142] S230, programmable texture synthesis. 3.1) GAN basic generation: the generator inputs noise z and state transfer matrix P, and outputs a cartoon flame sequence; the PatchGAN discriminator focuses on constraining the sharpness of the brush stroke edge (L1 loss weight 0.7). 3.2) NeRF physical field enhancement: density field correction σ(x) = sin(2πfE)·σ0(x), f = 2Hz, E = 0.7; volume rendering is a semi-transparent halo superimposed with a hand-painted style.

[0143] S240, interactive optimization. 4.1) Artist semantic parameter adjustment: gesture drawing fast spiral trajectory → mapping to “rotation energy” E spin =0.9; Rating: 5 / 5 stars (reward R = 1). 4.2) DDPG online update: Actor update, increase the weight of state transition randomness α Δa = +0.15; the output result is: the flame presents a more exaggerated spiral rise shape, and the inter-frame smoothness remains 0.89 (SSIM).

[0144] Application Case 3: Magic Particle Effects (Dynamic Switching of Mixed Paths)

[0145] S310, multimodal feature extraction. 1.1) Input data preprocessing: The data sources include real-time particle position data from the game engine (UE5 Niagara system) and wizard gesture capture data (Leap Motion sensor); 1.2) Spatiotemporal Separation Convolutional Network (STS-CNN): The spatial channel extracts the spatial distribution pattern of the particle group (fractal dimension D = 1.8); the temporal channel extracts the correlation between gesture acceleration and particle motion (correlation coefficient ρ = 0.79).

[0146] S320, hybrid dynamic texture analysis. 2.1) Dynamic gate switching: initial stage (particle aggregation), g = 0.82 → physical path (electromagnetic field confinement); burst stage (particle diffusion), g = 0.58 → statistical path (random walk model).

[0147] S330, programmable texture synthesis. 3.1) Physical stage generation: Solve Maxwell equations to generate particle trajectories, with energy conservation error <1%. 3.2) Statistical stage generation: Generate non-Gaussian motion patterns (Hurst index H = 0.3) based on Lévy Flight's random walk.

[0148] S340. Interactive optimization. 4.1) Wizard voice feedback: "Increase the particle velocity by 20%!" → The speech recognition module converts it to Δv = +20%. 4.2) DDPG online update: Enhance the Lorentz force coefficient ΔF in the physical path Lorentz = +0.3. 4.3) Real-time rendering: The generation delay of the particle spiral acceleration effect is <50 ms, and the user satisfaction is 9.2 / 10.

[0149] As can be seen from the above three application cases: The physical path (Application Case 1) strongly constrains scientific laws and is suitable for realistic special effects; the statistical path (Application Case 2) focuses on artistic expression and adapts to stylized requirements; the dynamic switch (Application Case 3) realizes the phased control of complex special effects. Therefore, the general dynamic texture generation method of the present invention covers a wide range of requirements from movie-level realism to abstraction through modular collaboration and intelligent decision-making.

[0150] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the embodiments of the present invention have been shown and described, those of ordinary skill in the art should understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and purposes of the present invention, and the scope of the present invention is defined by the claims and their equivalents.

Claims

1. A general dynamic texture generation method, characterized in that: include: Separate the spatiotemporal features from the input data, construct the spatial features and temporal features of the texture, and fuse the spatial features and temporal features to obtain the fused features; Dynamically select physical simulation or statistical modeling path according to the fusion feature type obtained by fusion to generate controllable texture evolution rules; Combining generative adversarial networks with neural radiance fields, the texture evolution rules are processed to generate visually consistent and physically credible dynamic texture sequences, and dynamic textures adapted to the target scene are output.

2. The general dynamic texture generation method according to claim 1, characterized in that: The step of separating the spatiotemporal features from the input data and constructing the spatial static features and temporal dynamic features of the texture includes the following sub-steps: Preprocess the input data to form a unified data format and filter out noise; The filtered unified data is parsed into spatiotemporally separated feature representations through a spatiotemporally separated convolutional network.

3. The general dynamic texture generation method according to claim 1, characterized in that: The spatial feature and the temporal feature are fused to obtain a fused feature. It includes the following sub-steps: Get the spatial gradient energy E spatial ; Get the temporal difference sparsity E temporal ; Quantum annealing optimization is used to generate the dynamic weight α, and its calculation formula is: Where β is the annealing parameter, which is dynamically adjusted by the simulated annealing algorithm; Using feature fusion formula F fusion =α·F spatial +(1-α)·F temporal The spatial feature and the temporal feature are fused, where α is a dynamic weight and F spatial Represents the spatial feature, F temporal Represents time characteristics.

4. The general dynamic texture generation method according to claim 1, characterized in that: The method of dynamically selecting a physical simulation path or a statistical modeling path according to the fusion feature type obtained by fusion to generate a controllable texture evolution rule includes the following sub-steps: The fused features are input into the fully connected layer to generate the gate value g, g∈[0,1]; The modeling method is selected according to the threshold of the gate value g, where if g>0.7, the physical simulation path is enabled; if g≤0.7, the statistical modeling path is enabled; Generate the corresponding texture evolution rules according to the selected path.

5. The general dynamic texture generation method according to claim 1, characterized in that: The method combines the generative adversarial network with the neural radiation field, processes the texture evolution rules to generate a visually consistent and physically credible dynamic texture sequence, and outputs a dynamic texture adapted to the target scene, including the following sub-steps: Input noise and texture evolution rules into the generator of the generative adversarial network to obtain the basic texture sequence; The discriminator of the generative adversarial network calculates the local adversarial loss to ensure the authenticity of texture details; Perform multi-physics field enhancement on the basic texture sequence through neural radiation field to synthesize dynamic texture sequence with multi-view illumination consistency; By constraining inter-frame smoothness through optical flow loss, dynamic texture sequences with optimized temporal consistency are synthesized; Output a dynamic texture that adapts to the target scene.

6. The general dynamic texture generation method according to claim 1, characterized in that: After the step of combining the generative adversarial network with the neural radiation field to process the texture evolution rules to generate a visually consistent and physically credible dynamic texture sequence, and before the step of outputting the dynamic texture adapted to the target scene, the method also includes the following steps: adjusting parameters in real time according to user feedback and after online reinforcement learning.

7. The general dynamic texture generation method according to claim 6, characterized in that: The method of adjusting parameters in real time according to user feedback and after online reinforcement learning includes the following sub-steps: The semantic control interface is used to enable users to adjust semantic parameters, which are mapped to real-time modification of underlying physical parameters; Trigger online reinforcement learning based on user rating feedback to update and optimize underlying physical parameters; Generate a preview frame of the adjustment results for viewing, and generate an optimized dynamic texture sequence after confirmation.

8. A general dynamic texture generation system, characterized in that: include: The multimodal feature extraction module is used to separate the spatiotemporal features from the input data, construct the spatial features and temporal features of the texture, and fuse the spatial features and temporal features to obtain the fused features; A hybrid dynamic texture analysis module is used to dynamically select a physical simulation or statistical modeling path according to the fusion feature type obtained by fusion, and generate controllable texture evolution rules; The programmable texture synthesis module is used to combine the generative adversarial network with the neural radiation field, process the texture evolution rules to generate visually consistent and physically credible dynamic texture sequences, and output dynamic textures that adapt to the target scene.

9. The general dynamic texture generation system according to claim 8, characterized in that: It also includes an interactive optimization module, which is used to adjust parameters in real time after user feedback and triggering online reinforcement learning to optimize the generation of dynamic texture sequences that are suitable for the target scene.

10. The universal dynamic texture generation system according to claim 8, characterized in that: The density field σ in the rendering equation of the neural radiation field is parameterized, and physical quantities related to the dynamic texture type are incorporated to support a variety of physical simulation dynamic textures.

Images