A digital content generation method and system based on adaptive frequency domain fusion

Abstract

The application is suitable for the technical field of digital content generation, and provides a digital content generation method and system based on adaptive frequency domain fusion, which comprises a data input module, a data preprocessing module, a frequency transformation module, an adaptive fusion weight generation module, a frequency domain fusion execution module and a frequency inverse transformation module. The data input module is responsible for receiving and preliminarily processing input source data. The data preprocessing module comprises a data standardization unit and a size unification unit. The frequency transformation module is responsible for mapping data to a frequency domain. The adaptive fusion weight generation module comprises a feature processing unit, a feature alignment fusion unit, a weight regression network unit and a weight map output unit for generating and outputting an adaptive fusion weight map. The frequency domain fusion execution module is responsible for performing a data fusion operation in the frequency domain. The frequency inverse transformation module is responsible for reconstructing data to obtain a digital result. Therefore, the application solves the technical problem of blurred details and structural distortion of generated content, and significantly improves the visual fidelity of generated content.

Description

Technical Field

[0001] This invention relates to the field of digital content generation technology, and more specifically, to a digital content generation method and system based on adaptive frequency domain fusion. Background Technology

[0002] Digital content generation refers to the technology of automatically creating or editing multimedia data such as images, videos, audio, and text using computer algorithms. With the development of artificial intelligence, this technology has been widely applied in many fields such as film and television special effects, game development, advertising design, artistic creation, and virtual reality, greatly improving the efficiency of content production and stimulating creative expression.

[0003] Currently, mainstream digital content generation methods, such as Generative Adversarial Networks (GANs) and Diffusion Models, typically follow an end-to-end learning paradigm within a latent space or pixel space. Specifically, these methods directly learn the intrinsic distribution and mapping relationships of the training data through deep neural networks. The generation process begins with a random noise vector or a purely noisy image; the model then progressively reconstructs or denoises it through a series of complex nonlinear transformations, ultimately outputting entirely new digital content that conforms to the user's instructions.

[0004] However, the aforementioned existing end-to-end learning-based technologies have inherent limitations. Because the models perform global optimization in a highly compressed and abstract latent space or directly in a high-dimensional pixel space, their learning objectives tend to capture the most prominent statistical features in the data. This leads to an imbalance in the processing of different frequency components. Specifically, high-frequency information (such as object edges and fine textures) and low-frequency information (such as overall contours and color schemes) are tightly coupled and processed together during generation, and the model lacks the ability to independently and finely control them. This coupling produces several adverse consequences: First, this process easily causes blurring or distortion of details in the generated content; for example, high-frequency details such as human hair, animal fur, and architectural textures are smoothed or lost, resulting in a lack of realism. Second, this process may cause global structural imbalances, such as distorted object shapes or incorrect spatial layouts, because low-frequency structural information is not stably maintained during optimization. Third, it limits the granularity of user control over the generated results, making it difficult to achieve precise creative intentions that require decoupling frequency components, such as "preserving the structure of image A but giving image B its texture." The root cause of these problems is that existing methods have not introduced a mechanism that can explicitly decouple, analyze, and adaptively fuse different frequency components, thus limiting further improvement in generation quality and expansion of application scope. Summary of the Invention

[0005] To address the aforementioned shortcomings, the present invention aims to provide a digital content generation method and system based on adaptive frequency domain fusion, in order to solve the problems raised in the background art, including a data input module responsible for receiving and initially processing various types of input source data;

[0006] The data preprocessing module includes a data standardization unit for normalizing input data with different numerical ranges to a unified interval, and a size normalization unit for detecting and adjusting the consistency of each input data in the spatial dimension.

[0007] Frequency conversion module; responsible for mapping preprocessed spatial or time domain data to the frequency domain;

[0008] An adaptive fusion weight generation module includes a feature processing unit, a feature alignment and fusion unit capable of forming a unified feature tensor from multiple input features along the channel dimension, a weight regression network unit composed of a lightweight convolutional neural network, and a weight map output unit for generating and outputting an adaptive fusion weight map. The feature processing unit includes one or more of a Fourier transform feature reconstruction unit and a wavelet transform frequency band specific processing unit.

[0009] Frequency domain fusion execution module; responsible for performing precise data fusion operations in the frequency domain;

[0010] The frequency inverse transformation module is responsible for receiving the fused frequency domain data from the frequency domain fusion execution module, reconstructing the fused frequency domain data back into the spatial domain, and obtaining a perceptible digital result.

[0011] The training optimization module can guide the entire system to learn the optimal fusion strategy by defining specific optimization objectives during the training phase.

[0012] Preferably, the weight map output unit consists of a 1x1 convolution layer and a Sigmoid activation function. The 1x1 convolution layer maps the number of channels to the target dimension, while the Sigmoid function compresses each pixel value into the (0, 1) range, ultimately outputting an adaptive fusion weight map with the same size as the target fusion region.

[0013] Preferably, the training optimization module includes a composite loss function calculation unit. The composite loss function calculation unit calculates the loss function through content loss. Combating losses Phase consistency loss The total loss L is calculated in three parts.

[0014] A digital content generation method based on adaptive frequency domain fusion, using an adaptive frequency domain fusion-based digital content generation system, includes the following steps:

[0015] S1: Input and preprocess data;

[0016] S2: Frequency transformation; The preprocessed spatial domain source data is transformed to the frequency domain through a mathematical transformation method to obtain its frequency domain representation; The preferred mathematical transformation method is Fourier transform or wavelet transform.

[0017] S3: Adaptive fusion weight graph generation; includes the following sub-steps:

[0018] S3.1: Perform feature processing using the following steps S3.1.1 and / or S3.1.2;

[0019] S3.1.1: Complex feature processing: The frequency domain representation of the complex number obtained by Fourier transform is decomposed into two independent data channels, namely the real part and the imaginary part, or the calculated amplitude spectrum and phase spectrum are used as two independent data channels to provide clear frequency structure information for subsequent networks.

[0020] S3.1.2: Frequency band specific processing: If wavelet transform is used for frequency transformation, then independent, lightweight convolutional subnetworks with the same structure are assigned in parallel to the multiple frequency subband coefficients obtained by decomposition. Each subnetwork independently processes its corresponding subband information.

[0021] S3.2: Input the processed features into a lightweight convolutional neural network for weight regression; finally, generate an adaptive fusion weight map with values ​​in the range of [0, 1] through 1x1 convolution and activation function regression at the end of the network.

[0022] S4: The system receives frequency domain data from step S2 and adaptive fusion weight map from step S3, and performs selective operations in the frequency domain; the selective operations include one or more of weighted combination, component selection and replacement.

[0023] S5: Inverse frequency transformation and content generation; The fused frequency domain data is reconstructed back into the spatial domain through the corresponding inverse transformation to generate the final digital content;

[0024] S6: Perform training optimization steps during the model training phase.

[0025] Preferably, the training optimization step in step S6 includes the following sub-steps:

[0026] S6.1: Forward Propagation: Input training samples and execute steps S1 to S5 sequentially to obtain the generated results;

[0027] S6.2: Composite Loss Calculation: Calculate the composite loss between the generated result and the true target; the composite loss L consists of content loss. Combating losses Phase consistency loss The weighted sum is obtained by applying the following formula:

[0028] ;

[0029] S6.3: Using the gradient descent algorithm, the gradient generated by the composite loss is backpropagated to all trainable parameters of the entire network, including the main generator network and the adaptive fusion weight generation module, and then updated.

[0030] Preferably, step S6.2 includes the following sub-steps:

[0031] S6.2.1: Calculate the two-dimensional Fourier transforms of the generated image and the real target image respectively, and obtain... and ;

[0032] S6.2.2: From respectively and Extracting the phase spectrum and ;

[0033] S6.2.3: Use the following formula:

[0034] ;

[0035] This allows us to calculate the norm loss between the two phase spectra.

[0036] Preferably, step S1 includes the following sub-steps:

[0037] S1.1: Input at least two types of source data; the source data includes conditional images, text encoding vectors, style reference images, or noise latent representations;

[0038] S1.2: Perform data standardization processing on the input source data, normalizing its pixel values ​​or feature values ​​to a predetermined range;

[0039] S1.3: Detect whether the spatial dimensions of all source data are consistent; if they are inconsistent, adjust all data to the same spatial dimensions through upsampling or downsampling algorithms.

[0040] Preferably, in step S1.3, bilinear interpolation or bicubic interpolation algorithms are used to upsample the feature map with a smaller spatial size; pooling operations are used to downsample the feature map with a larger size, so that all its inputs have a consistent height and width in the spatial dimension.

[0041] Preferably, in step S4, the system selectively operates on specific frequencies of different source data by sequentially performing frequency characteristic analysis, fusion strategy decision-making, and operation execution, based on adaptive fusion rules.

[0042] Frequency response analysis: The frequency domain representation in complex form obtained after Fourier transform is analyzed;

[0043] Fusion strategy decision-making: Based on the results of frequency characteristic analysis, an optimal operation command is generated for each frequency component;

[0044] Operation execution: Applying the operation instructions to subsequent steps.

[0045] Preferably, in the fusion strategy decision, the method for generating frequency domain results under dual-source data is as follows:

[0046] Let the two source data be A and B, and the transformed frequency domain representation be as follows: and Where (u, v) are frequency coordinates, the adaptive fusion weight generation module generates a map that is consistent with... Weighted graphs of the same size The value range is [0, 1], and the frequency domain result after fusion is... It can be derived from * + (1 - (u, .

[0047] Therefore, the technical solution of this application has at least the following advantages and beneficial effects:

[0048] 1. This invention effectively solves the technical challenges of blurred details and structural distortion in generated content, significantly improving the visual fidelity of the generated content. Existing technologies process data in the spatial domain or latent space, leading to the coupling optimization of high-frequency details (texture, edges) and low-frequency structures (contours, layout), resulting in detail loss and structural distortion. This invention innovatively introduces a frequency-domain adaptive fusion mechanism into digital content generation. It decouples data into independent high- and low-frequency components through Fourier / wavelet transform and utilizes a lightweight convolutional network specifically designed for the frequency domain to generate fusion weights; achieving the separation and targeted processing of image details and structures. The system can independently enhance high-frequency details to preserve texture while stabilizing low-frequency information to maintain structural correctness, thereby generating high-fidelity content with rich details and accurate structure, overcoming the fundamental defects of existing technologies.

[0049] 2. This invention avoids phase distortion issues in frequency domain operations, ensuring the structural integrity of the spatial domain reconstruction. Operating solely on the amplitude spectrum in the frequency domain while ignoring the coordinated changes in the phase spectrum leads to severe edge misalignment and structural distortion in the inverse-transformed image. This invention uniquely introduces phase consistency loss into the training loss function, directly constraining the phase spectrum differences between the generated and target images in the frequency domain. By directly optimizing phase alignment within the frequency domain, it ensures that the fused frequency components maintain the correct structural relationships when reconstructed back to the spatial domain, mathematically avoiding phase distortion—an effect that cannot be achieved by methods that only calculate loss in the spatial domain.

[0050] 3. Intelligent processing of frequency band heterogeneity is achieved, meeting the generation requirements in terms of both quality and controllability. This invention proposes a frequency band-specific adaptive processing architecture. When using wavelet transform, the system allocates independent lightweight convolutional sub-networks in parallel for different sub-bands. Each sub-network specializes in learning the optimal fusion strategy for its corresponding frequency band. This achieves true frequency band adaptive fusion. The system can intelligently prioritize preserving the structural information of the source data in low-frequency sub-bands and the texture details of the source data in high-frequency sub-bands, thereby enhancing details without compromising structural stability. This achieves a balance between extreme control over different attribute features and high-quality generation.

[0051] 4. It offers more refined control, improving the application effect of digital content generation. Existing methods have coarse control granularity, making it difficult to achieve advanced creative intentions such as "preserving structure A and blending texture B". This invention achieves pixel-level and channel-level frequency components through precise weighting, i.e., a data-driven adaptive fusion weight map. Users can provide reference sources with different characteristics, such as a structural sketch + a texture reference map, to precisely control the macroscopic semantics and microscopic style of the generated result. This provides a powerful technical tool for refined and customized content production in fields such as film and television special effects, game art, and creative design, expanding the application scenarios of generative models.

[0052] 5. An end-to-end learnable global optimization system was constructed, improving efficiency. This invention jointly trains a lightweight adaptive fusion weight generation module with the main generator network end-to-end, performing global optimization through a composite loss function including phase loss. The entire system can automatically learn the optimal fusion prior knowledge from the data, rather than relying on manual rules. The lightweight design ensures a significant improvement in generation quality with minimal computational overhead, exhibiting excellent performance-to-price ratio and easy integration into existing generative models. Attached Figure Description

[0053] Figure 1 This is the overall flowchart of the present invention;

[0054] Figure 2 Flowchart of data input and preprocessing;

[0055] Figure 3 Flowchart for generating adaptive fusion weight graph;

[0056] Figure 4 For training and optimizing the flowchart. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] See appendix Figure 1 -Appendix Figure 4 The method of this invention utilizes frequency domain fusion for digital content generation. Unlike traditional fusion methods that perform pixel-level operations directly in the spatial or temporal domains, frequency domain fusion in this invention refers to a technique that combines multiple signals or data in the frequency domain.

[0059] To achieve the aforementioned frequency domain fusion, the original data must first be transformed from the spatial or time domain to the frequency domain. This transformation is achieved through mathematical transformation methods, preferably Fourier transform or wavelet transform. After transforming the data to the frequency domain, the essence of the data, that is, its basic components, becomes clear, thus facilitating more precise fusion operations that are difficult to achieve in the original domain.

[0060] The use of Fourier transform to convert data to the frequency domain is a common technique in this field, and the following is only a brief explanation of the principle. The core idea of ​​converting data to the frequency domain using Fourier transform is that any functional signal can be decomposed into a superposition of sine and cosine waves of different frequencies, amplitudes, and phases. For this technical solution, the basic principle of applying two-dimensional Fourier transform to images is as follows: an image is transformed from the spatial domain (x, y coordinates) to the frequency domain, resulting in a complex array. Its amplitude spectrum displays the intensity of various frequency components in the image, while the phase spectrum determines the spatial position of these frequency components. This allows global frequency information to be provided by the amplitude and phase spectra.

[0061] To transform data to the frequency domain using wavelet transform, the signal is decomposed by scaling and translating a finite-length mother wavelet function. Its advantage lies in providing both frequency and spatial location information simultaneously, enabling time-frequency localization analysis of the signal, and is particularly suitable for characterizing local features such as edges and textures in images.

[0062] After data transformation using any of the methods described above, the image representation in the frequency domain can be decoupled into frequency components of different properties, including low-frequency and high-frequency components. Low-frequency components correspond to regions in the image where grayscale or color changes slowly, carrying the overall outline, general structure, and tonal information of smooth areas, thus determining the main content of the image. High-frequency components correspond to regions in the image where grayscale or color changes drastically, carrying the detailed features of the image, including edges, textures, and noise, thus determining the image's fine detail.

[0063] However, simply applying general deep learning models to frequency domain fusion presents inherent technical challenges. First, the complex representations in the frequency domain (including amplitude and phase, or real and imaginary parts) are tightly coupled. Conventional methods that only manipulate the amplitude spectrum while ignoring the coordinated changes in the phase spectrum will result in severe phase distortion in the spatial domain image after inverse transformation, manifesting as edge misalignment and structural distortion. Second, the semantic information carried by frequency components in different frequency bands (low and high frequencies) is drastically different, and their contribution to the final generated content also varies greatly. An adaptive network that is effective in the spatial domain or ordinary latent space struggles to simultaneously and coordinately handle this frequency band heterogeneity in the frequency domain, often enhancing details while compromising structural stability, or erasing key textures while preserving structure. This invention aims to provide an adaptive fusion mechanism that can deeply understand and follow the characteristics of frequency domain data.

[0064] In the frequency domain fusion process, after performing frequency transformation on the source data to obtain its frequency domain representation, the system selectively operates on specific frequency components of different source data according to adaptive fusion rules. These operations include weighted combination, component selection, or replacement. Finally, an inverse transformation is performed on the processed frequency components to reconstruct them into the spatial domain, thereby generating the fused data. The entire process achieves the separation and targeted processing of different attributes such as image structure and texture.

[0065] Furthermore, this invention optimizes the fusion strategy through an intelligent dynamic adjustment mechanism, namely adaptive frequency domain fusion. This adaptive mechanism automatically adjusts the weight parameters and fusion strategy of the frequency domain fusion based on the essential characteristics of the input data and the real-time state of the generation process, thereby overcoming the limitations of fixed fusion rules.

[0066] In terms of technical implementation, this invention achieves the above functions through an adaptive fusion weight generation module. The specific implementation of the adaptive fusion weight generation module is based on a lightweight convolutional network structure, which extracts the frequency distribution characteristics of input features through convolution operations. It receives feature representations from source data or intermediate layers of the generation network as input, performs real-time analysis and processing of the feature representations, and then regresses to generate a fusion weight map.

[0067] It is worth noting that the lightweight convolutional network structure in this technical solution is designed for processing frequency domain representations, which is highly compatible with the characteristics of frequency domain data. Specifically, the input of this network is not direct spatial domain RGB pixels, but rather a separated channel representation of complex data obtained after frequency transformation. For example, for a complex matrix obtained through Fourier transform, the network uses its real and imaginary parts as two independent input channels; or its amplitude and phase spectra as inputs. This forces the network to learn the complex, nonlinear coupling relationship between amplitude and phase from the first layer, thus laying the foundation for the subsequent generation of a fused weight map that can simultaneously maintain amplitude and phase consistency. This is the core difference between this invention and adaptive networks operating in the spatial domain or ordinary latent space.

[0068] Specifically, the adaptive fusion weight generation module receives input data from two sources: first, the original source data, including but not limited to latent representations of conditional images or videos; and second, the feature representations of intermediate layers in the generation network. These feature representations are typically activation maps in the form of multidimensional tensors, carrying visual feature information at different levels and scales during the generation process. In deep neural networks, when data is passed between network layers, its form is not the final generated image, but rather intermediate data called "feature maps" or "activation maps," i.e., multidimensional tensors. Their dimensions are typically represented by batch size, number of channels, height, and width. Height and width characterize the spatial dimensions of the feature map, while the number of channels represents the number of different types of features extracted by that layer. For example, some channels are specifically responsible for responding to edges, while others are responsible for responding to textures. Together, these feature maps constitute a deep, abstract, distributed representation of the input data, forming the basis for subsequent processing.

[0069] Upon receiving the aforementioned multiple data inputs, the system first preprocesses and aligns the input data to ensure consistency in spatial and channel dimensions. Since input data from different sources may differ in spatial size, number of channels, or numerical range, direct processing would lead to errors. Therefore, this module employs bilinear or bicubic interpolation algorithms to upsample feature maps with smaller spatial dimensions, or uses pooling operations to downsample feature maps with larger dimensions, ensuring that all inputs have consistent height and width in spatial dimensions.

[0070] For cases where the number of channels is inconsistent, this module uses a 1×1 convolution operation to transform the channel dimension, performing linear combination and dimensionality reduction or expansion of all channel information at each spatial location of the input feature map, thereby achieving channel number matching and feature fusion.

[0071] After alignment and channel number matching, the processed multipath feature maps are concatenated or summed to form a unified input tensor, which is then fed into the lightweight convolutional network of this module for depth analysis. This network consists of several convolutional layers. The core operation of each convolutional layer is convolution, which involves using a set of learnable parameter matrices (also called convolution kernels) to perform sliding window computation on the input feature map. Each convolution kernel systematically scans every spatial location of the input feature map, calculating the inner product of its parameters and the data within the local input window at each step, thereby generating a response value in the output feature map.

[0072] Multiple convolutional kernels produce output feature maps with multiple channels. Through this process, each convolutional kernel learns and extracts a specific local pattern from the input data, such as edges in a specific direction or textures with a specific frequency. Shallow convolutional kernels typically capture basic features, such as edges and corners, while deep convolutional kernels can combine these basic features to form more complex and abstract feature representations, such as object parts or complex texture patterns.

[0073] To further enhance the nonlinear expressive power of the network and enable it to fit the complex mapping relationship between the input and the fusion weights, a nonlinear activation function is introduced after each convolutional layer operation; the ReLU function is preferred in this invention. The ReLU function operates on the linear output of the convolution operation, setting all negative values ​​to zero while keeping positive values ​​unchanged. Applying the ReLU function to the convolution operation removes the linear limitation of the network, allowing multi-layer networks to learn and represent highly complex nonlinear functions, thereby enabling a better understanding and analysis of the frequency distribution characteristics of the input features.

[0074] After the aforementioned alternating processing of multiple convolutional layers and nonlinearities, the network finally uses 1×1 convolutions combined with an activation function to perform regression generation in the output layer. The 1×1 convolution maps the extracted features to the required number of channels. The activation function compresses each input value into the range [0, 1]. Finally, this module outputs one or more fusion weight maps with the exact same size as the target fusion feature map. This fusion weight map is a data matrix, where the value of each point represents the feature importance weight at the corresponding spatial location and channel in the subsequent fusion calculation. The closer the value is to 1, the more important the feature at that location is in the fusion; the closer the value is to 0, the less important it is.

[0075] The generated fusion weight map will be directly used to guide the frequency domain fusion process. In the frequency domain, this fusion weight map is subjected to element-wise weighted multiplication with frequency components from different sources, thereby achieving a dynamically adaptive fusion effect.

[0076] For example, when processing textured regions rich in detail, the module automatically generates higher weight values ​​to retain more high-frequency information; while when processing smooth structural regions, it generates relatively lower weight values ​​to maintain the stability of low-frequency information. The entire adaptive fusion weight generation module is jointly trained with the main generator network in an end-to-end manner, automatically learning and optimizing the mapping relationship from input features to optimal fusion weights through backpropagation and gradient descent optimization algorithms.

[0077] Specifically, the end-to-end training of the entire system relies on a composite loss function that includes frequency-domain specific constraints. This is in addition to the standard spatial domain content loss. and combat losses In addition, this invention also specifically introduces phase coherence loss under frequency domain-specific constraints. The specific calculation method for this loss function is as follows: calculate the absolute difference between the fusion result and the phase spectrum obtained after Fourier transform of the real target image;

[0078] The expression is:

[0079] ;

[0080] in, This is the complex frequency domain representation of the fused image after two-dimensional Fourier transform; φ(x) represents the frequency domain complex representation of the real target image after two-dimensional Fourier transform; φ(x) represents the operation of calculating the phase spectrum.

[0081] The total loss L is obtained by combining this loss item with other loss items, and the calculation formula is as follows:

[0082]

[0083] , These are the key weight parameters for balancing adversarial loss and phase consistency loss, respectively. Thus, the training process is explicitly guided towards an optimization direction: the weight graph generated by the adaptive fusion weight generation module must ensure that the phase information of the fused frequency data is as close as possible to the true target before being reconstructed back into the spatial domain. This loss directly affects the frequency domain, effectively solving the core problem of phase distortion in frequency domain operations, something that ordinary networks that only calculate losses in the spatial domain cannot achieve.

[0084] The core of the digital content generation method based on adaptive frequency domain fusion proposed in this invention lies in abandoning the traditional fixed fusion rules and proposing a dynamic and learnable fusion strategy. That is, the system selectively operates on specific frequency components of different source data according to the adaptive fusion rules. In other words, the aforementioned adaptive fusion rules are not a static formula, but a learnable intelligent process dynamically decided by the adaptive fusion weight generation module. These rules adaptively select and combine from multiple operation strategies based on the characteristics of the input source data, specifically including: frequency characteristic analysis, fusion strategy decision-making, and operation execution.

[0085] For frequency response analysis, the system first analyzes the frequency domain representation in complex form obtained after Fourier transform. It then calculates the energy distribution of each source data in different frequency bands, i.e., the statistical characteristics of its amplitude spectrum, such as variance and mean, to quantify the information stability of each source data at different frequencies.

[0086] For fusion strategy decision-making, the system generates an optimal operation instruction for each frequency component based on the results of frequency characteristic analysis. To further achieve refined adaptation processing for different frequency components, the lightweight convolutional network of this invention adopts a frequency band-specific design. Specifically, if the input data is decomposed into multiple sub-bands via wavelet transform, the adaptive fusion weight generation module will equip each sub-band with a dedicated sub-network branch in parallel. The structure and parameters of each sub-network branch are independent, and it specifically learns the feature patterns and fusion strategies of its corresponding sub-band. For example, it specifically learns the high-frequency HH sub-band or the low-frequency LL sub-band. Finally, the weight map output by each branch performs the fusion operation within its corresponding frequency sub-band. This allows the system network to independently determine: in the low-frequency sub-band, which source data's structural information should be retained first; in the high-frequency sub-band, which source data's texture details should be retained first. This fundamentally solves the problem that a single network cannot handle frequency band heterogeneity, achieving truly frequency-component-related adaptive fusion.

[0087] For example, maximizing the information entropy of the output image to retain more details or minimizing structural distortion with the conditional image to maintain structural consistency; this achieves an adaptive frequency domain processing approach. Finally, based on these decision-making operations, the system performs fine-grained operations such as weighted combination, component selection, and replacement on the complex data in the frequency domain.

[0088] Specifically, assuming the system inputs two source data, A and B, the transformed frequency domain representation is as follows: and Where (u, v) are frequency coordinates. The adaptive fusion weight generation module generates a map that is consistent with... Weighted graphs of the same size (Value range [0, 1]). Frequency domain result after fusion. It can be expressed by the following formula:

[0089]

[0090] Based on the optimization objective, the system learns the criteria for generating weight maps through a data-driven approach. The system's judgment criteria originate from multi-dimensional analysis of the input data, quantifying the analysis results into specific feature maps, which serve as input to the adaptive fusion weight generation module.

[0091] Specifically, the system uses a pre-trained semantic feature extraction network, such as the VGG system used for deep learning image processing (since the VGG system is very common in the industry and belongs to existing technology, it will not be elaborated here); and ensures that the fusion result is consistent with the conditional input in terms of the main structure according to the fidelity criterion.

[0092] In this invention, the "adaptiveness" of the adaptive fusion weight generation module is reflected in the generation of the fusion weight graph. That is, the fusion weight graph... The judgment criteria originate from prior knowledge learned through joint optimization on a large amount of training data, rather than from rigid rules preset by humans. Based on the "adaptability" of the fusion weight graph, the "adaptability" of the fusion rules is achieved, which is the realization of adaptive frequency domain fusion.

[0093] Regarding the aforementioned aspect of "the system using a pre-trained semantic feature extraction network to ensure the fusion result maintains consistency with the conditional input in terms of main structure based on a fidelity criterion," the fidelity criterion means that the module assigns higher weights to source data that better maintains content consistency in the corresponding semantic regions, thereby making the output of the fusion weight map closer to the input semantics. In other words, this process requires the fusion rules (up to the fusion result obtained based on the fusion result) to maintain consistency with the conditional input (such as semantic segmentation maps and text descriptions) in terms of main structure.

[0094] The fusion and decision-making of criteria ensures that the output of the fusion weight graph is the most "suitable," and then relies on global end-to-end training of the entire digital content generation system. The entire system is trained on a massive dataset, and through repeated training processes, the criteria are selected and solidified.

[0095] Specifically, during training, the system establishes a composite loss function to verify the generated results. Since composite loss functions are an existing technique in deep learning, they will only be briefly explained here. The loss function typically includes content loss, style loss, and content loss. Using the backpropagation algorithm, the gradient is propagated back from the final composite loss function, guiding the parameter updates of the main network and the adaptive fusion weight generation module. The parameter update direction of the adaptive fusion weight generation module is always towards generating a weight map that ensures the weighted fused features, after subsequent network processing, achieve a lower overall loss value.

[0096] Therefore, this technical solution introduces an adaptive mechanism that can automatically adjust the fusion strategy according to the specific content characteristics, thereby achieving effective processing of input data of different types, styles, and qualities. At the same time, through adaptive weighting of different frequency components, it can achieve independent and precise control of the structure and texture of the generated result, enabling users to control the generated result through macroscopic semantics by providing reference sources with different characteristics.

[0097] Based on the above principles, this invention proposes a digital content generation system based on adaptive frequency domain fusion, comprising: a data input module, a data preprocessing module, a frequency transformation module, an adaptive fusion weight generation module, a frequency domain fusion execution module, a frequency inverse transformation module, and a training optimization module; the modules work together to realize the entire process from data input to final content generation.

[0098] The data input module serves as the system's entry point, responsible for receiving and initially processing various types of input source data. It is responsible for receiving source data from external sources, which are at least two types, including but not limited to conditional images, text encoding vectors obtained after text descriptions have been processed by the encoder, reference images used to provide style, or latent noise representations that serve as a starting point in generative models.

[0099] The data preprocessing module includes a data standardization unit and a size normalization unit. The data standardization unit normalizes input data from different numerical ranges to a unified interval. The size normalization unit checks the consistency of the spatial dimensions (height and width) of the input data; if size inconsistencies are found, its internal algorithms are used, such as calling the image processing interpolation algorithm unit (using bilinear or bicubic interpolation) to upsample the smaller-sized data, or calling the pooling algorithm unit to downsample the larger-sized data, ensuring that all input data have the same spatial dimensions. Finally, the preprocessed unified data is output to the frequency transformation module.

[0100] After receiving unified data from the data input and preprocessing modules, the frequency transformation module maps the preprocessed spatial or temporal data to the frequency domain. Based on system configuration or input data characteristics, this module selects to perform either a Fourier transform or a discrete wavelet transform. Specifically, for image data, a two-dimensional Fourier transform or a discrete wavelet transform is used to output the frequency domain. The transformed frequency domain representation is then sent to the adaptive fusion weight generation module.

[0101] The adaptive fusion weight generation module receives frequency domain data from the frequency transform module, analyzes the frequency characteristics of the input data, and dynamically generates an optimal, adaptive fusion weight map. The adaptive fusion weight generation module includes a feature processing unit, a feature alignment and fusion unit, a weight regression network unit, and a weight map output unit; the feature processing unit includes one or more of the following: a Fourier transform feature reconstruction unit and a wavelet transform frequency band specific processing unit.

[0102] The Fourier transform feature reconstruction unit splits the complex data output by the Fourier transform into two independent data channels: the real part and the imaginary part. Alternatively, it uses the calculated amplitude spectrum and phase spectrum as two independent channels and feeds them into the network. This unit deeply utilizes the characteristics of the frequency domain, forcing the network to learn the complex coupling relationship between amplitude and phase from the bottom up, laying the foundation for generating a phase-consistent weight map.

[0103] Wavelet transform frequency band specific processing unit: When the input is multi-subband coefficients decomposed by wavelet transform, this unit configures a lightweight convolutional sub-network with the same structure but independent parameters for each frequency sub-band in parallel. Each sub-network specializes in processing its corresponding sub-band coefficients and independently learns the fusion strategy for that frequency band.

[0104] The feature alignment and fusion unit is responsible for concatenating or adding the features from multiple inputs along the channel dimension to form a unified feature tensor.

[0105] The weighted regression network unit consists of a lightweight convolutional neural network (CNN). It comprises multiple convolutional layers, each containing convolutional operations and a non-linear activation function. Its depth and width can be adjusted based on computational resources, but its parameter count is significantly smaller than that of the main generator network. This network extracts high-level abstract features for decision-making from the input features through the superposition of multiple convolutional layers and non-linearities.

[0106] The weight map output unit consists of a 1x1 convolutional layer and a sigmoid activation function. The 1x1 convolution maps the number of channels to the target dimension, while the sigmoid function compresses each pixel value into the (0, 1) interval, ultimately outputting an adaptive fusion weight map with the same size as the target fusion region. The value of each pixel in this map represents the trust weight of the corresponding source data's frequency component at that location. The weight map output unit outputs the generated adaptive fusion weight map to the frequency domain fusion execution module.

[0107] The frequency domain fusion execution module simultaneously receives frequency domain data from the frequency transformation module and a fusion weight map from the adaptive fusion weight generation module, performing precise data fusion operations in the frequency domain. Based on a preset strategy, it performs one or a combination of three operations: weighted combination, component selection, or replacement. Finally, it outputs the fused frequency domain representation to the frequency inverse transformation module.

[0108] The inverse frequency transform module receives the fused frequency domain data from the frequency domain fusion execution module and reconstructs the fused frequency domain data back into the spatial domain to obtain a perceptible final result. Specifically, this module's reconstruction involves calling the inverse transform algorithm corresponding to the frequency transform module. If Fourier transform was used previously, then the fast inverse Fourier transform is called; the generated spatial domain digital content (such as images or video frames) is output.

[0109] The training optimization module is used to ensure that the system achieves adaptive capabilities. By defining specific optimization objectives during the training phase, it guides the entire system to learn the optimal fusion strategy.

[0110] Specifically, this module is enabled in system training mode, and its core is a composite loss function calculation unit. The total loss L calculated by this unit consists of the content loss... Combating losses Phase consistency loss It consists of three parts, and the total loss is calculated according to the following formula:

[0111]

[0112] Content loss is used to ensure content consistency; adversarial loss calculates the adversarial loss of the generated image to improve the realism of the generated content; phase consistency loss directly constrains the phase information of the model in the frequency domain, effectively solving the structural distortion and edge misalignment problems that may occur after the inverse transform. At the same time, the loss gradient generated by the training optimization module is used to optimize the parameters of the main generator network and the adaptive fusion weight generation module through the backpropagation algorithm, realizing end-to-end joint training.

[0113] Based on the above system, this invention proposes a digital content generation method based on adaptive frequency domain fusion, the method comprising the following steps:

[0114] S1: Input and preprocess data;

[0115] S1.1: Input at least two types of source data; the source data includes conditional images, text encoding vectors, style reference images, or noise latent representations.

[0116] S1.2: Perform data standardization processing on the input source data, normalizing its pixel values ​​or feature values ​​to a predetermined range.

[0117] S1.3: Detect whether the spatial dimensions of all source data are consistent; if they are inconsistent, adjust all data to the same spatial dimensions through upsampling or downsampling algorithms.

[0118] S2: Frequency transformation; The preprocessed spatial domain source data is transformed to the frequency domain through a mathematical transformation method to obtain its frequency domain representation; The preferred mathematical transformation method is Fourier transform or wavelet transform.

[0119] S3: Adaptive fusion weight graph generation; includes the following sub-steps:

[0120] S3.1: Perform feature processing using the following steps S3.1.1 and / or S3.1.2;

[0121] S3.1.1: Complex feature processing: The frequency domain representation of the complex number obtained by Fourier transform is decomposed into two independent data channels, namely the real part and the imaginary part, or the calculated amplitude spectrum and phase spectrum are used as two independent data channels to provide clear frequency structure information for subsequent networks.

[0122] S3.1.2: Frequency band specific processing: If wavelet transform is used for frequency transformation, then independent, lightweight convolutional subnetworks with the same structure are assigned in parallel to the multiple frequency subband coefficients obtained by decomposition. Each subnetwork independently processes its corresponding subband information.

[0123] S3.2: Input the processed features into a lightweight convolutional neural network for weight regression; finally, generate an adaptive fusion weight map with values ​​in the range of [0, 1] through 1x1 convolution and activation function regression at the end of the network.

[0124] S4: The system receives frequency domain data from step S2 and adaptive fusion weight map from step S3, and performs selective operations in the frequency domain; the selective operations include one or more of weighted combination, component selection and replacement.

[0125] S5: Inverse Frequency Transform and Content Generation; The fused frequency domain data is reconstructed back into the spatial domain through the corresponding inverse transform (inverse Fourier transform or inverse wavelet transform) to generate the final digital content.

[0126] S6: Perform training optimization steps during the model training phase; including the following sub-steps:

[0127] S6.1: Forward Propagation: Input training samples and execute steps S1 to S5 sequentially to obtain the generated results.

[0128] S6.2: Composite Loss Calculation: Calculate the composite loss between the generated result and the true target; the composite loss L consists of content loss. Combating losses Phase consistency loss The weighted sum is obtained by applying the following formula:

[0129]

[0130] Specifically, step S6.2 includes the following sub-steps:

[0131] S6.2.1: Calculate the two-dimensional Fourier transforms of the generated image and the real target image respectively, and obtain... and .

[0132] S6.2.2: From respectively and Extracting the phase spectrum and .

[0133] S6.2.3: Calculate the norm loss between two phase spectra using the following formula:

[0134]

[0135] S6.3: Using the gradient descent algorithm, the gradient generated by the composite loss is backpropagated to all trainable parameters of the entire network, including the main generator network and the adaptive fusion weight generation module, and then updated to minimize the overall loss. By constraining the phase consistency loss, the phase fidelity of the model in the frequency domain is directly optimized, which is key to solving the structural distortion problem.

[0136] The various embodiments of the present invention have now been described in detail. To avoid obscuring the concept of the invention, some details known in the art have not been described. Those skilled in the art will fully understand how to implement the technical solutions of this invention based on the above description, and the scope of the invention is defined by the appended claims.

Claims

1. A digital content generation system based on adaptive frequency domain fusion, characterized in that, include: Data input module; Responsible for receiving and initially processing various types of input source data; Data preprocessing module; It includes a data standardization unit for normalizing input data with different numerical ranges to a unified interval, and a size unification unit for detecting and adjusting the consistency of each input data in the spatial dimension. Frequency conversion module; responsible for mapping preprocessed spatial or time domain data to the frequency domain; Adaptive fusion weight generation module; It includes a feature processing unit, a feature alignment and fusion unit that can form a unified feature tensor from multiple input features in the channel dimension, a weight regression network unit composed of a lightweight convolutional neural network, and a weight map output unit for generating and outputting an adaptive fused weight map; the feature processing unit includes one or more of a Fourier transform feature reconstruction unit and a wavelet transform frequency band specific processing unit. Frequency domain fusion execution module; responsible for performing precise data fusion operations in the frequency domain; The frequency inverse transformation module is responsible for receiving the fused frequency domain data from the frequency domain fusion execution module, reconstructing the fused frequency domain data back into the spatial domain, and obtaining a perceptible digital result. The training optimization module can guide the entire system to learn the optimal fusion strategy by defining specific optimization objectives during the training phase.

2. The digital content generation system based on adaptive frequency domain fusion according to claim 1, characterized in that, The weight map output unit consists of a 1x1 convolution layer and a Sigmoid activation function. The 1x1 convolution is responsible for mapping the number of channels to the target dimension, while the Sigmoid function compresses each pixel value into the (0, 1) range, and finally outputs an adaptive fusion weight map with the same size as the target fusion region.

3. The digital content generation system based on adaptive frequency domain fusion according to claim 1, characterized in that, The training optimization module includes a composite loss function calculation unit; the composite loss function calculation unit calculates content loss... Combating losses Phase consistency loss The total loss L is calculated in three parts.

4. A digital content generation method based on adaptive frequency domain fusion, characterized in that, The digital content generation system based on adaptive frequency domain fusion according to claim 3 includes the following steps: S1: Input and preprocess data; S2: Frequency transformation; The preprocessed spatial domain source data is transformed to the frequency domain using a mathematical transformation method to obtain its frequency domain representation; The mathematical transformation method is Fourier transform or wavelet transform; S3: Adaptive fusion weight graph generation; includes the following sub-steps: S3.1: Perform feature processing using the following steps S3.1.1 or S3.1.2; S3.1.1: Complex feature processing: The frequency domain representation of the complex number obtained by Fourier transform is decomposed into two independent data channels, namely the real part and the imaginary part, or the calculated amplitude spectrum and phase spectrum are used as two independent data channels to provide clear frequency structure information for subsequent networks. S3.1.2: Frequency band specific processing: If wavelet transform is used for frequency transformation, then independent, lightweight convolutional subnetworks with the same structure are assigned in parallel to the multiple frequency subband coefficients obtained by decomposition, and each subnetwork independently processes its corresponding subband information. S3.2: Input the processed features into a lightweight convolutional neural network for weight regression operation; finally, generate an adaptive fusion weight map with values ​​in the range of [0, 1] through 1x1 convolution and activation function regression at the end of the network. S4: The system receives frequency domain data from step S2 and adaptive fusion weight map from step S3, and performs selective operations in the frequency domain; the selective operations include one or more of weighted combination, component selection and replacement; S5: Inverse frequency transformation and content generation; The fused frequency domain data is reconstructed back into the spatial domain through the corresponding inverse transformation to generate the final digital content; S6: Perform training optimization steps during the model training phase.

5. The digital content generation method based on adaptive frequency domain fusion according to claim 4, characterized in that, The training optimization step in step S6 includes the following sub-steps: S6.1: Forward Propagation: Input training samples and execute steps S1 to S5 sequentially to obtain the generated results; S6.2: Composite Loss Calculation: Calculate the composite loss between the generated result and the real target; The composite loss L consists of content loss. Combating losses Phase consistency loss The weighted sum is obtained by applying the following formula: ； S6.3: Using the gradient descent algorithm, the gradient generated by the composite loss is backpropagated to all trainable parameters of the entire network, including the main generator network and the adaptive fusion weight generation module, and then updated.

6. The digital content generation method based on adaptive frequency domain fusion according to claim 5, characterized in that, Step S6.2 includes the following sub-steps: S6.2.1: Calculate the two-dimensional Fourier transforms of the generated image and the real target image respectively, and obtain... and ; S6.2.2: From respectively and Extracting the phase spectrum and ; S6.2.3: Use the following formula: ； This allows us to calculate the norm loss between the two phase spectra.

7. The digital content generation method based on adaptive frequency domain fusion according to claim 4, characterized in that, Step S1 includes the following sub-steps: S1.1: Input at least two types of source data; the source data includes conditional images, text encoding vectors, style reference images, or noise latent representations; S1.2: Perform data standardization processing on the input source data, normalizing its pixel values ​​or feature values ​​to a predetermined range; S1.3: Check whether the spatial dimensions of all source data are consistent; If there is a discrepancy, all data will be adjusted to the same spatial size using an upsampling or downsampling algorithm.

8. The digital content generation method based on adaptive frequency domain fusion according to claim 7, characterized in that, In step S1.3, bilinear interpolation or bicubic interpolation algorithms are used to upsample the feature map with a smaller spatial size; pooling operations are used to downsample the feature map with a larger size so that all its inputs have a consistent height and width in the spatial dimension.

9. The digital content generation method based on adaptive frequency domain fusion according to claim 4, characterized in that, In step S4, the system performs selective operations on specific frequencies of different source data in sequence through frequency characteristic analysis, fusion strategy decision-making, and operation execution, based on adaptive fusion rules. Frequency response analysis: The frequency domain representation in complex form obtained after Fourier transform is analyzed; Fusion strategy decision-making: Based on the results of frequency characteristic analysis, an optimal operation command is generated for each frequency component; Operation execution: Applying the operation instructions to subsequent steps.

10. The digital content generation method based on adaptive frequency domain fusion according to claim 9, characterized in that, In the fusion strategy decision, the method for generating frequency domain results under dual-source data is as follows: Let the two source data be A and B, and the transformed frequency domain representation be as follows: and Where (u, v) are frequency coordinates, the adaptive fusion weight generation module generates a map that is consistent with... Weighted graphs of the same size The value range is [0, 1], and the frequency domain result after fusion is... It can be expressed by the following formula: 。

Images