Image style transfer method and related device

Abstract

The application discloses an image style migration method and related equipment, solves how to explicitly decouple and fuse the content structure and local texture in multiple scales and multiple dimensions while maintaining spatial positioning accuracy, thereby realizing deep and accurate migration of style image color distribution and texture characteristics on the premise of effectively preserving the integrity of the content image structure. The application decomposes the encoder features into low-frequency approximation subbands and high-frequency detail subbands through two-dimensional discrete wavelet transform, and maps and interweaves the components into different frequency channels; uses a learnable frequency filter kernel to respectively retain the content skeleton and activate the style texture color in each subband, reorganizes after decoupling and restores through inverse transform; relies on wavelet time-frequency localization to maintain pixel-level spatial positioning, cooperates with a spectrum symmetry decoder to capture long-range feature dependence for step-by-step reconstruction, and realizes deep and accurate migration of style color texture on the premise of complete content structure.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image style transfer method and related equipment. Background Technology

[0002] In any image style transfer task, the content components (structure, topology, semantic layout) and style components (color patterns, local texture, brushstroke texture) of an image are highly intertwined and difficult to separate at the spatial pixel level. Most existing mainstream methods learn the decoupled representation of the two in the spatial domain. Due to this inherent intertwining, when processing complex images, the model often cannot clearly distinguish which features belong to structure and which belong to style, which can easily lead to problems such as content structure collapse, color distortion, and incomplete stylization, resulting in limited visual quality of the generated image.

[0003] Meanwhile, although Fourier transform can be used to decouple the global phase and amplitude of an image to alleviate large-scale structural distortion and global tone unevenness, Fourier transform lacks spatial localization capabilities in frequency domain processing. It is difficult to finely control complex textures of different scales and orientations in an image. Furthermore, the stylistic features of artworks (such as brushstrokes and local textures) usually have high temporal-frequency locality. Relying solely on global frequency domain alignment often results in insufficient local stylization and makes it impossible to achieve breakthroughs in the restoration of microscopic brushstrokes.

[0004] Therefore, how to explicitly decouple and fuse content structure and local texture in multiple scales and dimensions while maintaining spatial positioning accuracy, so as to achieve deep and accurate transfer of color distribution and texture features of style image while effectively preserving the integrity of content image structure, is a technical problem that urgently needs to be solved.

[0005] Therefore, an image style transfer method is proposed to solve or alleviate the above problems. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of existing technologies by proposing an image style transfer method and related equipment.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: An image style transfer method includes the following steps: The content image and style image are respectively input into a pre-trained encoder with frozen weights to extract content features and style features at multiple levels; For each level of content features and style features, two-dimensional discrete wavelet transform is used to decompose them into low-frequency approximate sub-bands that carry global topology and color tone, and high-frequency detail sub-bands that carry local texture and brushstrokes. A pre-trained learnable frequency filter kernel is used to decouple and filter the content components and style components on all sub-bands. The decoupled content sub-band components and style sub-band components are then recombined according to sub-band type, and the stylized features of that level are obtained by two-dimensional inverse discrete wavelet transform. The stylized features at each level are input into a spectral symmetric decoder, and then fused by upsampling and residual concatenation in a manner from high to low levels. Fast Fourier convolution is used to capture long-range feature dependencies in the global frequency range, gradually restoring spatial resolution, reconstructing and outputting the stylized image.

[0008] Preferably, the step of inputting the content image and style image into a pre-trained encoder with frozen weights to extract multiple levels of content features and style features includes: A pre-trained convolutional neural network is used as the encoder, and its network parameters are frozen during training and inference to extract stable feature representations. Multiple layers in the encoder are selected as feature output layers, including at least one high-level semantic layer and at least one low-level texture layer, in order to balance high-level semantic information and low-level texture details. The content image and style image are respectively input into the encoder. The content features and style features of the corresponding level are obtained through the feature mapping of the multiple levels. The content features and style features of each level are then input in parallel into the wavelet domain decoupling and fusion module that corresponds to the level.

[0009] Preferably, the content features and style features of each level are decomposed into low-frequency approximate subbands and high-frequency detail subbands using two-dimensional discrete wavelet transform, respectively. A pre-trained learnable frequency filter kernel is used to decouple the content and style components on all subbands. The decoupled content and style subband components are then recombined according to subband type, and the stylized features of that level are obtained through two-dimensional inverse discrete wavelet transform, including: Two-dimensional discrete wavelet transform is performed on the content features and style features of each level, and each is decomposed into an approximate sub-band representing low-frequency information and three detail sub-bands representing high-frequency information. A set of learnable frequency filtering kernels is introduced for this level. By applying point-by-point weighting to the approximate sub-bands and detail sub-bands of the content features and style features respectively, the structural skeleton is preserved from the content sub-bands and the texture and color modes are activated from the style sub-bands, resulting in filtered content sub-band components and style sub-band components. The filtered content subband components and style subband components are added element-wise according to the same subband type to obtain stylized approximate subbands and stylized detail subbands; Perform a two-dimensional inverse discrete wavelet transform on the stylized approximate subband and the stylized detail subband to map back to the original feature dimension and obtain the stylized features of this level.

[0010] Preferably, the step of inputting the stylized features of each level into a spectral symmetric decoder, fusing them through upsampling and residual concatenation in a high-to-low layer manner, and capturing long-range feature dependencies in the global frequency range using fast Fourier convolution, progressively restoring spatial resolution, reconstructing and outputting the stylized image includes: A spectrum-symmetric decoder with a mirror-symmetric structure to the encoder is adopted, and the ordinary convolutional layers in the decoder are replaced with fast Fourier convolutional blocks. The fast Fourier convolutional blocks capture long-range feature dependencies in the global frequency range to make up for the limitations of local convolution in reconstructing large-area color distributions. Following the principle of high-level semantics guiding low-level details, the high-level stylistic features are upsampled and then fused with the low-level stylistic features using residual concatenation. The spatial resolution is restored step by step through multiple processing stages of the decoder by the result of residual concatenation and fusion, and finally the stylized image is reconstructed and output.

[0011] Preferably, the pre-trained learnable frequency filter kernel and the spectral symmetric decoder are obtained through iterative optimization via the following training process: Input the content image and the style image into the method to generate a stylized image; Calculate spatial domain content loss and spatial domain style loss. The spatial domain content loss is used to constrain the consistency of the stylized image and the content image in spatial structure, and the spatial domain style loss is used to constrain the stylized image to absorb the style image in terms of statistical characteristics. To calculate the wavelet domain style contrast loss, within the wavelet sub-band space, other stylized images obtained from the same reference style transfer are used as positive samples, and stylized images obtained from different style transfers are used as negative samples. The cascaded features of all sub-bands are mapped to the metric space through the projection head and contrast constraints are applied to enhance the ability to discriminate and transfer style features. The wavelet domain content loss is calculated by performing two-dimensional discrete wavelet transforms on the features of the stylized image and the content image respectively, and then constraining the consistency of the two with different weights on the approximate subband and the detail subband respectively. The spatial domain content loss, spatial domain style loss, wavelet domain style contrast loss, and wavelet domain content loss are weighted and summed to obtain a composite loss function. The network other than the encoder is iteratively optimized with the goal of minimizing the composite loss function to obtain the trained learnable frequency filter kernel and the spectral symmetric decoder.

[0012] Preferably, the encoder is a pre-trained VGG-19 network, and all its weight parameters are frozen during training and inference; the multiple layers are the three layers ReLU3_1, ReLU4_1 and ReLU5_1 of the VGG-19 network; wherein the low-frequency approximate subbands corresponding to the high-level features output by the ReLU5_1 layer carry the global semantic information and energy distribution of the image, and anchor the overall tone of the stylized image during the subband reconstruction process.

[0013] Preferably, the two-dimensional discrete wavelet transform uses Haar wavelet as a fixed operator, and decomposition is achieved by alternately performing one-dimensional wavelet transform in the horizontal and vertical directions; the approximate subband is an LL subband, used to capture the macroscopic geometry and overall color of the image; the detail subband includes three subbands in the horizontal, vertical, and diagonal directions, used to encode the local edges and texture details of the image; the point-by-point weighting is implemented using Hadamard product; the learnable frequency filter kernel is independently learned for different feature levels and updated through backpropagation, wherein the frequency filter kernel acting on the content subband is used to preserve the topological structure of the content image, and the frequency filter kernel acting on the style subband is used to activate the texture and color modes of the style image.

[0014] Preferably, the fast Fourier convolutional block includes a local branch and a global branch. The global branch performs a fast Fourier transform on the features, convolves them in the frequency domain, and then performs an inverse fast Fourier transform to return them to the spatial domain, thereby capturing long-range feature dependencies within the global frequency range. The upsampling is bilinear interpolation upsampling. The decoder includes a first processing stage and a second processing stage. Each processing stage consists of at least one fast Fourier convolutional block. During reconstruction, the highest-level stylized features are upsampled and added to the next higher-level stylized features, then processed by the first processing stage. Finally, the highest-level stylized features are added to the lowest-level stylized features, and then processed by the second processing stage to obtain a stylized image.

[0015] Preferably, the spatial domain content loss is the sum of the Euclidean distances between the stylized image and the content image at selected level feature maps; the spatial domain style loss is the mean and variance of the stylized image and the style image at each level feature map; the wavelet domain style contrast loss is based on the InfoNCE criterion, which maps all sub-band cascaded features of the stylized image and its positive and negative samples after two-dimensional discrete wavelet transform to the metric space through a projection head, calculates the similarity of positive and negative samples using temperature coefficient scaling, and takes the negative logarithm of the ratio of positive sample similarity to all sample similarity; the wavelet domain content loss is the sum of the approximate sub-band differences and detail sub-band differences between the stylized image and the content image at selected level features, weighted by a first weight and a second weight, respectively, where the first weight is greater than the second weight; the projection head is a multilayer perceptron.

[0016] The present invention also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor executes the computer program to implement the image style transfer method as described above.

[0017] The present invention also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the image style transfer method described above.

[0018] The present invention has the following beneficial effects: This invention addresses the problems of content structure collapse, color distortion, and incomplete stylization caused by the high degree of interweaving and difficulty in separating content and style at the spatial pixel level. It explicitly decomposes the multi-level features extracted by the encoder into low-frequency approximate sub-bands carrying the global topological structure and color tone, and high-frequency detail sub-bands carrying local textures and brushstrokes through two-dimensional discrete wavelet transform. This maps the interwoven components to different frequency channels. Then, a pre-trained learnable frequency filter kernel is used to preserve the content structure skeleton and activate style textures and colors on all sub-bands, achieving decoupling of content and style. After recombining according to sub-band type, the content is restored using two-dimensional inverse discrete wavelet transform. Simultaneously, the time-frequency localization of wavelet transform maintains pixel-level spatial positioning accuracy while achieving frequency layering. Combined with a spectral symmetric decoder, long-range feature dependencies are captured and reconstructed level by level within the global frequency range. Ultimately, while effectively preserving the integrity of the content image structure, it achieves deep and accurate transfer of color distribution and texture features in style images. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of the present invention; Figure 2 This is a schematic diagram of the network structure of the present invention, wherein (a) is the overall network structure and (b) is the internal structure of the wavelet domain decoupling and fusion module (WaveFuse module); Figure 3 This is a quantitative comparison result between the method in this invention and mainstream methods; Figure 4 This is a comparison of the quantitative analysis results of the ablation experiment in this invention; Figure 5 This is a visualization of the ablation experiment results in this invention; Figure 6 The results of the sensitivity analysis of the weights of the wavelet domain content loss function in this invention are shown. Figure 7 This is a comparison of the visual effects of different sub-band fusion strategies in the WaveFuse module of this invention; Figure 8 This is the qualitative visual comparison result in this invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0024] In the description of this invention, it should be understood that the terms center, up, down, left, right, vertical, horizontal, inside, outside, etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0025] Furthermore, the terms first, second, third, etc., are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0026] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "setup," "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0027] This embodiment provides an image style transfer method, such as Figure 1 As shown, the method includes the following steps: Step S101: Input the content image and style image into the pre-trained encoder with frozen weights, respectively, to extract content features and style features at multiple levels.

[0028] In some embodiments, step S101 includes: using a pre-trained convolutional neural network as an encoder, and freezing its network parameters during training and inference to extract stable feature representations; selecting multiple layers in the encoder as feature output layers, the multiple layers including at least one high-level semantic layer and at least one low-level texture layer to balance high-level semantic information and low-level texture details; inputting the content image and style image into the encoder respectively, obtaining the corresponding content features and style features of the layers through the feature mapping of each layer respectively, and inputting the content features and style features of each layer in parallel into the wavelet domain decoupling and fusion module corresponding to the layer one by one.

[0029] Step S102: For each level of content features and style features, decompose them into low-frequency approximate sub-bands carrying the global topological structure and color tone, and high-frequency detail sub-bands carrying local texture and brushstrokes through two-dimensional discrete wavelet transform. Use a pre-trained learnable frequency filter kernel to decouple the content components and style components on all sub-bands. Then, reassemble the decoupled content sub-band components and style sub-band components according to the sub-band type, and obtain the stylized features of that level through two-dimensional inverse discrete wavelet transform.

[0030] In some embodiments, step S102 includes: performing a two-dimensional discrete wavelet transform on the content features and style features of each level, decomposing them into an approximate sub-band representing low-frequency information and three detail sub-bands representing high-frequency information; introducing a set of learnable frequency filtering kernels for this level, and applying point-by-point weighting to the approximate sub-bands and detail sub-bands of the content features and style features respectively, preserving the structural skeleton from the content sub-band and activating the texture and color modes from the style sub-band to obtain filtered content sub-band components and style sub-band components; adding the filtered content sub-band components and style sub-band components element-wise according to the same sub-band type to obtain stylized approximate sub-bands and stylized detail sub-bands; performing a two-dimensional inverse discrete wavelet transform on the stylized approximate sub-bands and stylized detail sub-bands to map them back to the original feature dimensions to obtain the stylized features of this level.

[0031] Step S103: The stylized features of each level are input into the spectral symmetric decoder. They are then fused by upsampling and residual concatenation in a way that proceeds from high to low levels. Fast Fourier convolution is used to capture long-range feature dependencies in the global frequency range, gradually restoring the spatial resolution, reconstructing and outputting the stylized image.

[0032] In some embodiments, step S103 includes: employing a spectral symmetric decoder with a mirror-symmetric structure to the encoder, and replacing the ordinary convolutional layers in the decoder with fast Fourier convolutional blocks. The fast Fourier convolutional blocks capture long-range feature dependencies in the global frequency range to compensate for the limitations of local convolution in reconstructing large-area color distributions; following the principle of high-level semantics guiding low-level details, the high-level stylized features are upsampled and then residually concatenated and fused with the low-level stylized features; the result of the residual concatenation is processed through multiple stages of the decoder to restore the spatial resolution step by step, and finally the stylized image is reconstructed and output.

[0033] The pre-trained learnable frequency filter kernel and the spectral symmetric decoder are obtained through iterative optimization via the following training process: Step 1: Input the content image and style image into the method to generate a stylized image; Step 2: Calculate the spatial domain content loss and spatial domain style loss. The spatial domain content loss is used to constrain the consistency of the stylized image and the content image in terms of spatial structure, and the spatial domain style loss is used to constrain the stylized image to absorb the style image in terms of statistical characteristics. Step 3: Calculate the wavelet domain style contrast loss. In the wavelet sub-band space, other stylized images obtained from the same reference style transfer are used as positive samples, and stylized images obtained from different styles transfer are used as negative samples. The cascaded features of all sub-bands are mapped to the metric space through the projection head and contrast constraints are applied to enhance the ability to discriminate and transfer style features. Step 4: Calculate the wavelet domain content loss. After performing two-dimensional discrete wavelet transforms on the features of the stylized image and the content image respectively, constrain the consistency of the two with different weights on the approximate subband and the detail subband respectively. Step 5: The spatial domain content loss, spatial domain style loss, wavelet domain style contrast loss and wavelet domain content loss are weighted and summed to obtain the composite loss function. The network outside the encoder is iteratively optimized with the goal of minimizing the composite loss function to obtain the trained learnable frequency filter kernel and spectral symmetric decoder.

[0034] The encoder is a pre-trained VGG-19 network, with all its weight parameters frozen during training and inference. The multiple layers are the ReLU3_1, ReLU4_1 and ReLU5_1 layers of the VGG-19 network. The low-frequency approximate subbands corresponding to the high-level features output by the ReLU5_1 layer carry the global semantic information and energy distribution of the image, and anchor the overall tone of the stylized image during the subband reconstruction process.

[0035] The two-dimensional discrete wavelet transform uses Haar wavelets as a fixed operator, and decomposition is achieved by alternately performing one-dimensional wavelet transforms in the horizontal and vertical directions. The approximate subband is the LL subband, which is used to capture the macroscopic geometry and overall color of the image. The detail subband includes three subbands in the horizontal, vertical and diagonal directions, which are used to encode the local edges and texture details of the image. Pointwise weighting is implemented using Hadamard product. The learnable frequency filtering kernel is independently learned for different feature levels and updated through backpropagation. The frequency filtering kernel acting on the content subband is used to preserve the topological structure of the content image, and the frequency filtering kernel acting on the style subband is used to activate the texture and color modes of the style image.

[0036] The Fast Fourier Transform (FFT) convolutional block includes local and global branches. The global branch performs FFT on the features, convolves them in the frequency domain, and then performs inverse FFT to return them to the spatial domain, thereby capturing long-range feature dependencies within the global frequency range. The upsampling is bilinear interpolation upsampling. The decoder includes a first processing stage and a second processing stage. Each processing stage consists of at least one FFT convolutional block. During reconstruction, the highest-level stylized features are upsampled and added to the next higher-level stylized features, then processed by the first processing stage. Finally, the highest-level stylized features are added to the lowest-level stylized features, and then processed by the second processing stage to obtain the stylized image.

[0037] Specifically, the spatial domain content loss is the sum of the Euclidean distances between the stylized image and the content image at the selected level feature maps; the spatial domain style loss is the mean and variance of the aligned stylized image and the style image at each level feature map; the wavelet domain style contrast loss is based on the InfoNCE criterion, which maps all sub-band cascaded features of the stylized image and its positive and negative samples after two-dimensional discrete wavelet transform to the metric space through a projection head, calculates the similarity of positive and negative samples using temperature coefficient scaling, and takes the negative logarithm of the ratio of positive sample similarity to all sample similarity; the wavelet domain content loss is the sum of the approximate sub-band differences and detail sub-band differences between the stylized image and the content image at the selected level feature maps, weighted by the first weight and the second weight, respectively, with the first weight being greater than the second weight; the projection head is a multilayer perceptron.

[0038] The present invention also provides an electronic device, including a processor and a memory, wherein the memory is used to store a computer program, and the processor executes the computer program to implement the image style transfer method described above.

[0039] The present invention also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the image style transfer method described above.

[0040] The working principle is as follows: Unlike existing mainstream approaches that focus on spatial pixel level to distinguish between content and style, this method first inputs the content image and style image into a pre-trained encoder with frozen weights to extract features at multiple levels. Then, it performs a two-dimensional discrete wavelet transform on each level of features to decompose the features into low-frequency approximate subbands that carry the global topological structure and color tone, and high-frequency detail subbands that carry local texture and brushstrokes.

[0041] Through this decomposition, the content and style that were originally intertwined and difficult to separate at the pixel level are placed into sub-bands of different frequencies. Structural information and texture information are distinguished in the frequency dimension, thereby avoiding the collapse of content structure, color distortion and incomplete stylization caused by the mixing of the two.

[0042] Based on this, the method utilizes a pre-trained learnable frequency filter kernel to decouple the content and style components in all sub-bands. The content sub-bands retain their topological structure, while the style sub-bands activate their texture and color. The filtered content and style sub-band components are then recombined according to the same sub-band type and restored to the stylized features of that level through two-dimensional inverse discrete wavelet transform. This allows the skeleton of the content and the color and texture of the style to be processed separately in their respective appropriate frequency bands before being merged, thus preserving the structural integrity of the content image while fully absorbing the color distribution and texture features of the style image.

[0043] Unlike Fourier transform, which can only align phase and amplitude in the global frequency domain, wavelet transform retains the spatial location information corresponding to each frequency band while completing frequency layering. Therefore, the method can layer textures of different scales and orientations in the image and adjust them with spatial positioning, making up for the shortcomings of global frequency domain alignment in local stroke restoration.

[0044] Finally, the stylized features obtained from each level are input to the encoder's mirror-symmetric spectral decoder and fused step by step through upsampling and residual concatenation from high to low levels. Then, long-range feature dependencies are captured in the global frequency range by fast Fourier convolution, and the spatial resolution is restored step by step to reconstruct the stylized image. Thus, while maintaining pixel-level spatial positioning accuracy, multi-scale and multi-dimensional explicit decoupling and fusion of content structure and local texture are achieved, and finally, a structurally complete result with sufficient style transfer is obtained.

[0045] This method is applied to a style transfer network with a symmetric encoder-decoder structure. The network includes a pre-trained encoder with frozen weights, several wavelet domain decoupled and fused modules corresponding one-to-one with feature levels, and a spectral symmetric decoder. Figure 2 (a) shows the overall network structure. Figure 2 In Figure (b), the internal structure of the wavelet domain decoupling and fusion module, hereinafter referred to as the WaveFuse module, is shown. The inference and training processes are described below.

[0046] In the feature extraction process, this embodiment uses a pre-trained VGG-19 network as the encoder E, and freezes its weight parameters throughout the training and inference process to extract stable multi-scale feature representations. To balance high-level semantic information and low-level texture details during stylization, this embodiment selects the outputs of the ReLU3_1, ReLU4_1, and ReLU5_1 layers of VGG-19 as feature representations. The content image Ic and style image Is are input into the encoder respectively to obtain the multi-scale content features Fcl and style features Fsl, as shown in the following formula:

[0047] Here, El represents the feature map of the l-th layer of the encoder. Subsequently, the content features and style features of the l-th layer are fed in parallel into the WaveFuse module corresponding to that layer, where decoupling and recombination are performed in the wavelet subband space to generate the stylized feature Fcsl, i.e.:

[0048] In the wavelet domain decoupling and fusion process, see Figure 2 As shown in section (b), the WaveFuse module utilizes the frequency separability of wavelet transform to selectively filter and reconstruct features within the sub-band space. The low-frequency approximate sub-band carries the global topology and color tone of the image, while the high-frequency detail sub-band carries the local texture and brushstrokes of the image. This includes sub-band decomposition, filtering decoupling, sub-band reconstruction, and inverse transform reconstruction.

[0049] Specifically, two-dimensional discrete wavelet transforms are performed on the content feature Fcl and style feature Fsl of the l-th layer. In this embodiment, the Haar wavelet is used as a fixed operator. By alternately performing one-dimensional wavelet transforms in the horizontal and vertical directions, each feature is decomposed into an approximate sub-band A (i.e., the LL sub-band) and three detail sub-bands D in three directions (i.e., the horizontal direction LH, the vertical direction HL, and the diagonal direction HH).

[0050] Among them, the approximate subband A has a dimension of C×(Hl / 2)×(Wl / 2) and is used to capture the macro geometry and overall color of the image; the detail subband D has a dimension of 3C×(Hl / 2)×(Wl / 2) and is used to encode the local edges and texture details of the image.

[0051] Next, a set of learnable frequency filter kernels specific to this scale is introduced for this level. By applying point-by-point weights to the approximate subbands and detail subbands of content features and style features respectively, the structural skeleton is preserved from the content subbands, and the texture and color modes are activated from the style subbands. In this embodiment, the point-by-point weighting is implemented using the Hadamard product, and the calculation formula is as follows:

[0052]

[0053] Where ⊙ represents the Hadamard product, and Kcl,A, Kcl,D, Ksl,A, and Ksl,D are learnable frequency filter kernels that are independently learned for different feature levels l and updated via end-to-end backpropagation.

[0054] Then, the filtered content subband components and style subband components are added element-wise according to the same subband type to obtain the stylized approximate subband Acsl and the stylized detail subband Dcsl, i.e.:

[0055] Finally, a two-dimensional inverse discrete wavelet transform is performed on the stylized approximate subband and detail subband to map back to the original feature dimension, thus obtaining the stylized features at this level:

[0056] Thanks to the orthogonality and completeness of wavelet transform, the WaveFuse module can achieve fine feature editing in the frequency dimension without introducing spatial artifacts. The above process is executed in parallel at the three levels of ReLU3_1, ReLU4_1, and ReLU5_1, respectively, to obtain stylized features at the three scales of Fcs3, Fcs4, and Fcs5.

[0057] During the decoding and reconstruction process, the spectral symmetric decoder is structurally mirror-symmetric to the encoder, replacing its ordinary convolutional layers with Fast Fourier Convolution (FFC) blocks. The FFC block includes local and global branches. The global branch performs a Fast Fourier Transform on the features, convolutions in the frequency domain, and then returns to the spatial domain via an inverse Fast Fourier Transform, thereby capturing long-range feature dependencies across the global frequency range and compensating for the limitations of local convolutions in reconstructing large-area color distributions. Following the principle of high-level semantics guiding low-level details, the decoder performs bilinear interpolation upsampling Up(·) on high-level stylized features and then concatenates them with low-level stylized features using residual concatenation. The decoder includes a first processing stage FD1 and a second processing stage FD2, each consisting of at least one FFC block. The final stylized image Ics is reconstructed using the following formula:

[0058] Fcs3, Fcs4, and Fcs5 are the stylization features of layers 3, 4, and 5, respectively. This symmetrical architecture, combined with the design of fast Fourier convolution, ensures that the model fully utilizes the frequency information decoupled from the wavelet domain while restoring spatial resolution, enabling the generated image to have richer spectral expression and artistic texture while maintaining content topological consistency.

[0059] During training, the learnable frequency filter kernel and the spectral symmetric decoder are obtained through iterative optimization using a composite loss function, while the encoder weights remain frozen. The total loss is weighted and composed as follows:

[0060] Specifically, the perceptual loss is calculated using the features of the pre-trained VGG-19, the spatial domain content loss is used to calculate the Euclidean distance between feature maps, and the spatial domain style loss is used to align the mean μ and variance σ of the feature maps. The calculation formulas are as follows:

[0061]

[0062] Wavelet domain style contrast loss is applied to all sub-bands obtained after performing discrete wavelet transform on the generated features. Other stylized images obtained from the same reference style transfer are used as positive samples Ics+, and stylized images obtained from different style transfers are used as negative samples Ics-. The cascaded features of all sub-bands are mapped to the metric space through a projection head P, and the metric is calculated based on the InfoNCE criterion, where the projection head P is a multilayer perceptron and τ is the temperature coefficient. The calculation formula is as follows:

[0063]

[0064]

[0065] The wavelet domain content loss first performs a two-dimensional discrete wavelet transform on the l-th layer features El(Ics) and El(Ic) to obtain the approximate subband A and the detail subband D. Then, the consistency between the approximate subband and the detail subband is constrained by weights α and β, respectively. The calculation formula is as follows:

[0066] Here, α and β are weighting factors that weigh macroscopic geometric structure and local edge details, respectively, with α > β. In this embodiment, α is set to 0.9 and β is set to 0.1. During training, the network other than the encoder is iteratively optimized with the goal of minimizing the total loss. In this embodiment, the weighting coefficients are λ1=1.0, λ2=1.5, λ3=0.3, and λ4=0.6. Only the parameters of the learnable frequency filter kernel, the projection head P, and the spectral symmetric decoder in the WaveFuse module are updated. After training convergence, the model used in the inference stage of this method is obtained.

[0067] Furthermore, this embodiment uses a large-scale dataset for training. The content dataset is the Microsoft COCO dataset, containing approximately 120,000 images covering various natural scenes; the style dataset is the WikiArt dataset, containing approximately 80,000 images of artworks covering different genres, periods, and artistic styles. During the training phase, all input images were randomly cropped and resized to 512×512 pixel resolution; the model is based on the PyTorch framework, the optimizer is Adam, and the initial learning rate is 1×10⁻. 4The batch size was 4, and a total of 160,000 iterations were performed. All experiments were conducted on a server configured with an NVIDIA GeForce RTX 4090 GPU (24GB VRAM), Ubuntu 20.04, and CUDA 11.3. During the inference phase, only the content image and style image need to be input into the trained network to output a stylized image end-to-end. In this embodiment, the average inference time per image at 256×256 and 512×512 resolutions is approximately 0.017 seconds and 0.020 seconds, respectively.

[0068] To verify the effectiveness of the proposed method, this embodiment first conducts a qualitative visual comparison. The experiment selected images covering various semantic types, including architecture, landscapes, portraits, and animals, and tested them in conjunction with style reference images from different art movements. The comparison results are as follows: Figure 8 As shown in the figure. The first two columns are the input content image and style image, respectively, the third column is the generation result of this method, and the following columns are the transfer effects of various mainstream comparison algorithms.

[0069] Depend on Figure 8 It can be observed that although EFDM improves the accuracy of color transmission by matching higher-order feature distributions, it still fails to completely solve the content distortion problem when dealing with objects with sharp edges, and is prone to outline overflow. IEContraAST enhances content consistency by using contrastive learning, but when dealing with style maps with strong contrast, the generated results often retain some content from the style map, failing to effectively extract style features from the style map. AdaIN is more flexible in handling arbitrary styles, but due to its simple global statistical alignment, it often leads to the loss of local details in the synthesized results, and its content preservation ability is weak when dealing with buildings with complex geometric structures. Although CAP-VSTNet performs well in terms of content preservation... While excellent, the stylization strength is clearly insufficient, making it difficult to reproduce the iconic brushstroke textures in style maps, resulting in weak overall artistic expression. AdaAttN introduces local attention, but when processing style maps with uneven texture distribution or large areas of single color blocks, it is prone to producing repetitive artifacts, leading to image distortion and content warping. S2WAT introduces hierarchical visual self-attention and strip window attention, but it still exhibits significant content loss or texture deformation in some complex samples. StyleID, based on a diffusion model, proposes a style injection scheme that does not require training, but in the absence of text guidance, it often alters the original structure of the content image and even produces unnatural semantic distortions in fine areas.

[0070] In contrast, this method, while preserving the structural integrity of the content image, can better transfer the global tone and local texture details of the style image. From the perspective of time-frequency analysis, wavelet transform possesses dual time-frequency localization capabilities: in the frequency domain, it achieves explicit decoupling and independent enhancement of content structure and style patterns through fine sub-band division; in the spatial domain, its excellent localization characteristics ensure pixel-level semantic alignment, effectively preventing the loss of high-frequency semantic information. Through this combination of time-frequency perspectives, this method significantly enhances the accuracy of style injection in local regions during multi-scale feature reconstruction, enabling the generated results to exhibit better visual consistency and artistic effect in various challenging scenarios.

[0071] To verify the effectiveness of the proposed method, this embodiment compares it with seven mainstream style transfer algorithms: AdaIN, EFDM, IEContraAST, CAP-VSTNet, AdaAttN, S2WAT, and StyleID. Fifty content images from MS-COCO and 100 style images from WikiArt (both with a resolution of 512×512) were randomly selected, and the average values ​​were calculated using four metrics: content loss, style loss, structural similarity (SSIM), and peak signal-to-noise ratio (PSNR). The quantitative comparison results are as follows: Figure 3 As shown, our method achieves a content loss of 7.88, a style loss of 1.93, a SSIM score of 0.75, and a PSNR score of 13.79. Among these, SSIM and PSNR significantly outperform all the other methods compared, demonstrating that our method can inject high-quality style features while preserving the geometric structure and spatial details of the content.

[0072] To further verify the effectiveness, an ablation experiment was conducted in this embodiment, specifically as follows: Figure 4 and Figure 5 As shown: when the WaveFuse module is removed and replaced with a standard spatial convolutional layer with the same number of parameters, spatial adhesion occurs between the content contour and the style texture, and both SSIM and PSNR drop significantly; when the wavelet domain content loss is removed, semantic blurring occurs at the edges of details in the generated image; when the wavelet domain style contrast loss is removed, the style loss degrades drastically from 1.93 to 4.45; for subband fusion methods, low-frequency fusion only and high-frequency fusion only are inferior to full subband fusion. The above results demonstrate the key role of wavelet domain decoupling, learnable frequency filtering kernels, and the two types of wavelet domain losses in improving stylization quality.

[0073] Effectiveness of the WaveFuse Module: As the core component of this method, the WaveFuse module achieves deep feature interaction at the frequency subband level. When this module was experimentally removed and replaced with a standard spatial convolutional layer with the same number of parameters, the structural quality of the generated image experienced a precipitous decline: the original contours of the content image and the stylistic textures became severely spatially intertwined, the originally sharp geometric boundaries were covered by messy stylistic brushstrokes, and the image lost its basic semantic readability. The underlying reason is that when the standard convolutional operator performs feature fusion in the pixel domain, it cannot effectively separate high-frequency edge signals and complex artistic textures at a single spatial scale. This unavoidable feature interweaving directly leads to structural collapse. Quantitatively, the SSIM and PSNR of this variant both dropped significantly, verifying the irreplaceable nature of the WaveFuse module in achieving deep decoupling of content topology and artistic textures through wavelet subband decomposition and accurately reconstructing them on this basis.

[0074] Effectiveness of Wavelet Domain Content Loss: Wavelet content loss plays a crucial in-situ anchoring role in maintaining the fine topology of the original image. Comparing the complete model with a variant that removes this constraint reveals that relying solely on spatial domain perceptual loss often fails to lock structural boundaries under complex artistic brushstroke interference, leading to significant semantic ambiguity at detail edges in the generated image. The model cannot effectively suppress the erosion of the content skeleton by style textures. Correspondingly, the content loss of this variant increases to 11.44, SSIM decreases to 0.31, and PSNR decreases to 8.61, all significantly worse than the complete model, strongly demonstrating the in-situ anchoring value of wavelet content loss for content structure. Furthermore, to explore the impact of the wavelet domain content loss weight λ4 on model performance, this embodiment compares the sensitivity of λ4 with values ​​of 0.0, 0.3, 0.6, 1.0, and 2.0. Figure 6 As shown, when λ4 is small, the model has weak constraints on the content structure, and the style texture will overly cover the content semantics, resulting in obvious structural distortion at the image edges. As λ4 gradually increases, the clarity of the content structure is significantly improved. When λ4 is around 0.6, the generated result achieves a relatively ideal balance between style expressiveness and content consistency.

[0075] Effectiveness of Wavelet Domain Style Contrast Loss: Wavelet style contrast loss significantly enhances the model's ability to represent style features. In the variant where this loss is removed, although the content outline of the image is maintained, the generated artistic texture is significantly degraded, with loose brushstrokes, lacking logical rhythm, and failing to form the aesthetic distribution of textures characteristic of a specific style; its style loss degrades dramatically from 1.93 in the full model to 4.45. This numerical difference reveals that this loss, by constructing positive and negative sample contrast constraints across the entire subband space, forces the model to learn and reconstruct more discriminative artistic patterns in the frequency domain, thereby achieving a qualitative leap from simple color alignment to deep, high-fidelity texture simulation.

[0076] Effectiveness of High-Level Features: To explore the contribution of deep semantic features to large-scale art style modeling, removing the highest-level features in the decoder stage significantly reduced the overall visual tension of the image, resulting in a locally cluttered appearance. Large-scale brushstrokes and macroscopic color blocks in the style map, which originally possessed global coherence, failed to achieve effective energy mapping in the content map due to the lack of guidance from high-level frequency features. The final result appeared flat and lacked the rhythm characteristic of the style from a macroscopic perspective; correspondingly, the style loss increased to 2.58. This indicates that although this level has lower spatial resolution, its low-frequency approximation components represent the most essential global semantic information and energy distribution of the image. They play a constraining role in global navigation during sub-band fusion and are an indispensable component in enhancing the overall macroscopic texture of the stylized result.

[0077] Ablation of subband fusion strategies: such as Figure 7 As shown, to further explore the necessity of full subband feature fusion, this embodiment compares three variants: low-frequency fusion only, high-frequency fusion only, and the full subband fusion used in this method. Quantitative results are as follows: Figure 4 As shown.

[0078] Qualitatively, while low-frequency fusion effectively transferred the overall tone and color distribution of the style image, it severely lacked high-frequency elements such as abstract brushstrokes and graffiti lines unique to the style image, and the visual expression was close to basic tone changes. While high-frequency fusion produced some artistic textures and edge details, the low-frequency color energy was not converted, and areas such as the sky and background still retained the original natural colors of the content image, resulting in a clear visual separation between the style texture and the underlying tone. Quantitatively, full subband fusion significantly outperforms in style loss, SSIM, and PSNR. Low-frequency fusion alone has the lowest SSIM and PSNR due to the lack of high-frequency texture guidance, reflecting its lack of detail reconstruction. High-frequency fusion seems to have slightly better content loss, but this does not mean that processing only high frequencies can better protect the structure. Rather, it is an objective reflection of style transfer failure. Since the macroscopic geometry and background semantics of the image are mainly carried by the low-frequency approximate subband, this variant directly transmits the underlying information of the content map without ignoring low-frequency style injection, making its feature response numerically closer to the original content map. This further proves that only by co-processing the entire subband can the content structure be preserved to the maximum extent while reducing style loss, which is a necessary condition for balancing content structure and artistic texture.

[0079] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image style transfer method, comprising the following steps: The content image and style image are respectively input into a pre-trained encoder with frozen weights to extract content features and style features at multiple levels; For each level of content features and style features, two-dimensional discrete wavelet transform is used to decompose them into low-frequency approximate sub-bands that carry global topology and color tone, and high-frequency detail sub-bands that carry local texture and brushstrokes. A pre-trained learnable frequency filter kernel is used to decouple and filter the content components and style components on all sub-bands. The decoupled content sub-band components and style sub-band components are then recombined according to sub-band type, and the stylized features of that level are obtained by two-dimensional inverse discrete wavelet transform. The stylized features from each level are input into a spectral symmetric decoder, and fused by upsampling and residual concatenation in a high-to-low layer manner. Fast Fourier convolution is then used to capture long-range feature dependencies across the global frequency range. The spatial resolution is restored step by step, and the stylized image is reconstructed and output.

2. The image style transfer method according to claim 1, characterized in that, The process involves inputting the content image and style image into a pre-trained encoder with frozen weights to extract multiple levels of content and style features, including: A pre-trained convolutional neural network is used as the encoder, and its network parameters are frozen during training and inference to extract stable feature representations. Multiple layers in the encoder are selected as feature output layers, including at least one high-level semantic layer and at least one low-level texture layer, in order to balance high-level semantic information and low-level texture details. The content image and style image are respectively input into the encoder. The content features and style features of the corresponding level are obtained through the feature mapping of the multiple levels. The content features and style features of each level are then input in parallel into the wavelet domain decoupling and fusion module that corresponds to the level.

3. The image style transfer method according to claim 1, characterized in that, The content features and style features at each level are decomposed into low-frequency approximate subbands and high-frequency detail subbands using two-dimensional discrete wavelet transform. A pre-trained learnable frequency filter kernel is used to decouple the content and style components across all subbands. The decoupled content and style subband components are then recombined according to subband type, and the stylized features for that level are obtained through two-dimensional inverse discrete wavelet transform, including: Two-dimensional discrete wavelet transform is performed on the content features and style features of each level, and each is decomposed into an approximate sub-band representing low-frequency information and three detail sub-bands representing high-frequency information. A set of learnable frequency filtering kernels is introduced for this level. By applying point-by-point weighting to the approximate sub-bands and detail sub-bands of the content features and style features respectively, the structural skeleton is preserved from the content sub-bands and the texture and color modes are activated from the style sub-bands, resulting in filtered content sub-band components and style sub-band components. The filtered content subband components and style subband components are added element-wise according to the same subband type to obtain stylized approximate subbands and stylized detail subbands; Perform a two-dimensional inverse discrete wavelet transform on the stylized approximate subband and the stylized detail subband to map back to the original feature dimension and obtain the stylized features of this level.

4. The image style transfer method according to claim 1, characterized in that, The process involves inputting the stylized features from each level into a spectral symmetric decoder, fusing them through upsampling and residual concatenation from high to low layers, and capturing long-range feature dependencies in the global frequency range using fast Fourier convolution to progressively restore spatial resolution, reconstructing and outputting the stylized image, including: A spectrum-symmetric decoder with a mirror-symmetric structure to the encoder is adopted, and the ordinary convolutional layers in the decoder are replaced with fast Fourier convolutional blocks. The fast Fourier convolutional blocks capture long-range feature dependencies in the global frequency range to make up for the limitations of local convolution in reconstructing large-area color distributions. Following the principle of high-level semantics guiding low-level details, the high-level stylistic features are upsampled and then fused with the low-level stylistic features using residual concatenation. The spatial resolution is restored step by step through multiple processing stages of the decoder by the result of residual concatenation and fusion, and finally the stylized image is reconstructed and output.

5. The image style transfer method according to claim 1, characterized in that, The pre-trained learnable frequency filter kernel and the spectral symmetric decoder are obtained through iterative optimization via the following training process: Input the content image and the style image into the method to generate a stylized image; Calculate spatial domain content loss and spatial domain style loss. The spatial domain content loss is used to constrain the consistency of the stylized image and the content image in spatial structure, and the spatial domain style loss is used to constrain the stylized image to absorb the style image in terms of statistical characteristics. To calculate the wavelet domain style contrast loss, within the wavelet sub-band space, other stylized images obtained from the same reference style transfer are used as positive samples, and stylized images obtained from different style transfers are used as negative samples. The cascaded features of all sub-bands are mapped to the metric space through the projection head and contrast constraints are applied to enhance the ability to discriminate and transfer style features. The wavelet domain content loss is calculated by performing two-dimensional discrete wavelet transforms on the features of the stylized image and the content image respectively, and then constraining the consistency of the two with different weights on the approximate subband and the detail subband respectively. The spatial domain content loss, spatial domain style loss, wavelet domain style contrast loss, and wavelet domain content loss are weighted and summed to obtain a composite loss function. The network other than the encoder is iteratively optimized with the goal of minimizing the composite loss function to obtain the trained learnable frequency filter kernel and the spectral symmetric decoder.

6. The image style transfer method according to claim 2, characterized in that, The encoder is a pre-trained VGG-19 network, and all its weight parameters are frozen during training and inference. The multiple layers are the three layers ReLU3_1, ReLU4_1 and ReLU5_1 of the VGG-19 network. The low-frequency approximate subbands corresponding to the high-level features output by the ReLU5_1 layer carry the global semantic information and energy distribution of the image, and anchor the overall tone of the stylized image during the subband reconstruction process.

7. The image style transfer method according to claim 3, characterized in that, The two-dimensional discrete wavelet transform uses Haar wavelets as a fixed operator, and decomposition is achieved by alternately performing one-dimensional wavelet transforms in the horizontal and vertical directions. The approximate subband is the LL subband, which is used to capture the macroscopic geometry and overall color of the image. The detail subband includes three subbands in the horizontal, vertical, and diagonal directions, which are used to encode the local edges and texture details of the image. The point-by-point weighting is implemented using Hadamard product. The learnable frequency filter kernel is independently learned for different feature levels and updated through backpropagation. The frequency filter kernel acting on the content subband is used to preserve the topological structure of the content image, and the frequency filter kernel acting on the style subband is used to activate the texture and color modes of the style image.

8. The image style transfer method according to claim 4, characterized in that, The Fast Fourier Transform (FFT) convolutional block includes local and global branches. The global branch performs FFT on the features, convolves them in the frequency domain, and then performs inverse FFT to return them to the spatial domain, thereby capturing long-range feature dependencies within the global frequency range. The upsampling is bilinear interpolation upsampling. The decoder includes a first processing stage and a second processing stage. Each processing stage consists of at least one FFT convolutional block. During reconstruction, the highest-level stylized features are upsampled, added to the next higher-level stylized features, and processed by the first processing stage. Then, the highest-level stylized features are added to the lowest-level stylized features and processed by the second processing stage to obtain the stylized image.

9. The image style transfer method according to claim 5, characterized in that, The spatial domain content loss is the sum of the Euclidean distances between the stylized image and the content image at selected level feature maps; the spatial domain style loss is the mean and variance of the stylized image and the style image at each level feature map; the wavelet domain style contrast loss is based on the InfoNCE criterion, which maps all sub-band cascaded features of the stylized image and its positive and negative samples after two-dimensional discrete wavelet transform to the metric space through a projection head, calculates the similarity of positive and negative samples with temperature coefficient scaling, and takes the negative logarithm of the ratio of positive sample similarity to all sample similarity; the wavelet domain content loss is the sum of the approximate sub-band differences and detail sub-band differences of the stylized image and the content image at selected level features, weighted by a first weight and a second weight, respectively, where the first weight is greater than the second weight; the projection head is a multilayer perceptron.

10. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program, and the processor executing the computer program to implement the image style transfer method as described in any one of claims 1-9.

Images