A clothing image super-resolution method and device

Abstract

The present invention proposes an innovative clothing image super-resolution method and device. The method is based on the diffusion method of the frequency domain guided attention mechanism. The clothing images and their super-resolution real labels are screened and obtained from the database to construct a data set, which is divided into a training set, a validation set and a test set in a ratio of 8:1:1; the pre-trained wavelet convolutional neural network module is used to extract key features, and the high-frequency and low-frequency components are separated by the frequency domain information splitter module; the frequency domain guided cross attention module and the diffusion model based on residual transmission are combined, and the denoising and restoration process is optimized with the assistance of a flexible noise scheduling method, the sampling steps are reduced, and the reasoning speed is accelerated; high-resolution images are output, the richness and quality of details are improved, the mode collapse is effectively avoided, and the image diversity and realism are enhanced. The present invention can significantly enhance the display effect of digital media content, which is conducive to merchants to display product details in a clearer and more realistic way, thereby improving user experience.

Description

Technical Field

[0001] The invention belongs to the field of computer and network technology, and in particular to a clothing image super-resolution method and device. Background Art

[0002] Image super-resolution (SR) is a core problem in the field of low-resolution vision, which aims to recover a high-resolution (HR) image from a low-resolution (LR) image. This problem is subject to serious uncertainties due to the complexity and unknown nature of the degradation model in real scenes.

[0003] Traditional methods based on generative adversarial networks (GANs) are prone to mode collapse when dealing with clothing images with diverse styles, making it difficult to capture the diversity of images, resulting in a lack of variation in generated samples. In recent years, as diffusion models have made significant progress in the field of image generation, more and more researchers have begun to explore the potential of diffusion models in super-resolution (SR) tasks. At present, there are two main ways to apply diffusion models to SR tasks: one is to take LR images as input and retrain the diffusion model to generate SR images; the other is to use the unconditionally pre-trained diffusion model as a priori and adjust its reverse process to generate the desired HR images. Unfortunately, both strategies inherit the Markov chain underlying the DDPM, which leads to inefficient inference and usually requires hundreds or even thousands of sampling steps. Although some acceleration techniques have been proposed to reduce the number of sampling steps, these techniques often significantly affect the performance, resulting in overly smooth generated results and loss of detailed features, which cannot meet the actual industrial production needs. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention proposes a clothing image super-resolution method and device.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] In one aspect, the present invention provides a clothing image super-resolution method, comprising the following steps:

[0007] S1: Obtain clothing image data from the database and its corresponding super-resolution image real labels to form a super-resolution clothing dataset, and divide it into training set, validation set and test set according to 8:1:1;

[0008] S2: The user manually uploads the clothing image that needs to be super-resolution generated in the system, and enters it into the system after preprocessing to save it for subsequent operations;

[0009] S3: Design a frequency-domain guided attention mechanism based diffusion method (FGA-D) to achieve super-resolution clothing image generation. The FGA-D method consists of a pre-trained wavelet convolutional neural network module (D-CNN), a frequency-domain guided cross attention module (FDG-CA), a residual transmission based diffusion model (RT-DDM) and a flexible noise scheduling method (NS);

[0010] S4: During the training process, the clothing images in the training set are input into FGA-D. First, the input clothing image (LR) is extracted using the pre-trained wavelet convolutional neural network module (D-CNN). Then, the extracted image features are input into the residual transmission diffusion model (RT-DDM). The noise image and the frequency domain information splitter module (FDIS) are combined to further obtain the feature map of low-frequency information and the feature map of high-frequency components. The learned features are spliced ​​in the channel dimension and input into the UNet module. Next, the UNet method equipped with the frequency domain guided cross attention module (FDG-CA) is used to denoise and restore the input features to obtain a high-resolution image (HR). Finally, the residual transmission and flexible noise scheduling methods are used to further optimize the entire diffusion process to effectively reduce the diffusion steps.

[0011] S5: Load the model weights and network structure of the FGA-D trained in S4, obtain the image that needs super-resolution input by the user from the system, and input it into the loaded FGA-D method;

[0012] S6: During the inference process, the user input image is first fed into the pre-trained D-CNN to extract image features, and then the high- and low-frequency information is extracted through the frequency domain information splitter module. The extracted features are input into the UNet module of the RT-DDM containing the frequency domain guided cross attention module. The denoised version of the super-resolution generation result is gradually estimated from the input high- and low-frequency features through a flexible noise scheduling method, and the super-resolution result of the corresponding clothing image is output;

[0013] Optionally, the specific operation steps of acquiring clothing image data in step S1 are:

[0014] S11: selecting clothing images with the term “super-resolution” from the existing clothing database, and dividing the selected images into a low-resolution clothing image set and a high-resolution clothing image set according to the resolution;

[0015] S12: combining the low-resolution clothing images and the high-resolution clothing images in pairs according to the IDs of the clothing images;

[0016] S13: labeling the combined clothing pairs, wherein the high-resolution clothing image data is used as the true label and the low-resolution clothing image data is used as the training sample to form a super-resolution clothing dataset;

[0017] S14: Divide the super-resolution clothing dataset into a training set, a validation set, and a test set according to the ratio of 8:1:1.

[0018] Optionally, the wavelet convolutional neural network module (D-CNN) in step S4 is:

[0019] The D-CNN module aims to reduce the additional training cost while extracting the features of the input image as conditional features. The module uses a convolutional neural network with a small number of parameters to extract the main low-frequency components and some high-frequency components, which helps the diffusion model to recover more complex high-frequency details. In order to ensure the feature extraction capability, the present invention introduces a loss function based on fast Fourier transform (FFT) and discrete wavelet transform (DWT), as well as a spatial domain loss function to form a joint loss function for collaborative optimization, wherein the FFT loss function is optimized by calculating the mean square error (MSE) between the FFT coefficients of the real image and the predicted image in the frequency domain; the DWT loss function further restores high-frequency details by calculating the MSE between the real image and the predicted image in each high-frequency sub-band; the spatial domain loss function directly calculates the MSE between the real high-resolution image and the predicted image. In practical applications, it is necessary to pre-train the D-CNN module using image data and these loss functions to ensure that it can effectively extract features and generate high-quality super-resolution images. In this way, the D-CNN module can provide powerful initial features for the subsequent diffusion model, thereby improving the performance and efficiency of the entire super-resolution process.

[0020] Optionally, the frequency domain guided cross attention (FDG-CA) method in step S4 is:

[0021] The frequency-domain guided cross-attention (FDG-CA) module aims to address the shortcomings of traditional diffusion models in high-frequency detail recovery by introducing frequency domain information to enhance the model's ability to capture high-frequency features. FDG-CA first extracts high-frequency information from the input image through wavelet transform and combines this information with linear projection to aggregate high-frequency features; then, different linear mappings are constructed using the feature maps extracted by fast Fourier transform to enhance the model's attention to high-frequency details through the cross-attention mechanism; finally, the final output feature map is calculated by combining the aggregated high-frequency information and the output of the cross-attention, which helps the model better predict high-frequency details. FDG-CA uses frequency domain information to guide the attention mechanism, so that the model focuses more on the high-frequency details of the image during the restoration process to generate feature maps containing rich high-frequency details, thereby improving the quality of super-resolution images.

[0022] Optionally, the residual transmission-based diffusion model (RT-DDM) in step S4 is:

[0023] The residual transfer based diffusion model (RT-DDM) is designed to solve the image super-resolution problem by using the iterative generation paradigm of the diffusion model. RT-DDM realizes the conversion from LR image to HR image by constructing a new Markov chain as a bridge between high-resolution (HR) and low-resolution (LR) images. Different from the Markov chain in the traditional diffusion model, on the one hand, in the forward process, the present invention adds a translation sequence to make it monotonically grow with time steps to gradually translate the residuals between LR and HR images to achieve the conversion between different distributions, wherein the hyperparameters for controlling the noise variance are designed to increase the flexibility of the design. On the other hand, in the reverse process, the mean variable and variance variable are recalculated and parameterized by a specific formula to further optimize the training objective function to improve the stability and performance of the model. At the same time, in the denoising UNet network, by adding a frequency domain information splitter module (FDIS) module in the input stage and inserting a frequency domain guided cross attention method between the jump links of each layer, the high-frequency and low-frequency information of the image is better captured to facilitate the learning of the high-frequency features of the image to generate fine-grained and richer image details. Through the above method, RT-DDM can effectively combine the advantages of traditional computer vision and diffusion models, improve the detail richness and quality of super-resolution images, while significantly reducing the necessary sampling steps, speeding up inference and improving overall efficiency.

[0024] Optionally, the flexible noise scheduling method in step S4 is:

[0025] The flexible noise scheduling method can flexibly control the movement speed and noise intensity in the diffusion process through a carefully designed noise plan, reasonable adjustment of hyperparameters and translation sequences, thereby maintaining the quality of super-resolution images while improving inference efficiency. First, the noise scheduling method uses hyperparameters to adjust the overall noise intensity in the diffusion process, which directly affects the performance of the model; then, by constructing a specific translation sequence, it can control the growth rate of noise to adapt to the diffusion needs of different stages; next, for the intermediate time step, the present invention proposes a non-uniform geometric scheduling scheme to adapt to the noise level of different time steps; finally, for the final stage of the diffusion process, the distribution conversion speed is precisely controlled by setting hyperparameters, achieving a trade-off between the fidelity of the super-resolution result and the inference speed, ensuring that the final state can converge to the detailed perturbations around the LR image, thereby shortening the length of the Markov chain and improving inference efficiency. The flexible noise scheduling method reduces the necessary number of sampling and speeds up the inference speed by optimizing the diffusion steps, while avoiding the deterioration of performance and enhancing the diversity and realism of the image.

[0026] Optionally, in step S4, the frequency domain information splitter module (FDIS) is:

[0027] The frequency domain information splitter module (FDIS) is used to efficiently separate the high-frequency and low-frequency information in the input image features, providing clear high-frequency details and low-frequency background information for the diffusion model, so as to more accurately restore the pixel-level details of the image. First, 2D FFT is performed on the image features extracted by D-CNN to convert the spatial domain features to the frequency domain for subsequent frequency domain processing; then, an adaptive Gaussian high-pass filter is multiplied element-by-element with the frequency domain feature map in the frequency domain to enhance the high-frequency components of the image, where the standard deviation of the filter is dynamically calculated based on the frequency domain feature map to achieve adaptive high-pass filtering; then, the processed frequency domain feature map is converted back to the spatial domain through inverse 2D FFT to obtain a feature map containing high-frequency details; next, the attention weights learned in the frequency domain are multiplied element-by-element with the low-frequency feature map to enhance the expression of low-frequency information; finally, the processed high-frequency and low-frequency feature maps are concatenated in the channel dimension to provide features that combine high- and low-frequency information for subsequent network layers. At the same time, FDIS can be combined with a flexible noise scheduling method to adaptively denoise noisy images with large time steps to mitigate the impact of noise on network reasoning. The FDIS module effectively separates high-frequency and low-frequency information from mixed frequency components, providing important details and background information for the generation of super-resolution images, thereby significantly improving the quality of image restoration.

[0028] Optionally, the method of performing reasoning using FGA-D according to the clothing image input by the user in step S6 is:

[0029] S61: The clothing image uploaded by the user is first pre-processed in the system to ensure that the image data is suitable for subsequent super-resolution operations;

[0030] S62: The preprocessed image is input into a pre-trained wavelet convolutional neural network module (D-CNN) to extract key features of the image, including the main low-frequency components and some high-frequency components;

[0031] S63: The frequency domain information splitter module (FDIS) is used to effectively separate the low-frequency and high-frequency components in the features extracted by D-CNN, providing a basis for the recovery of high-frequency details;

[0032] S64: The features after separation of low-frequency and high-frequency components are input into the UNet structure of RT-DDM, and the frequency-domain guided cross-attention module (FDG-CA) is used to focus on restoring the high-frequency details of the image, which are crucial for maintaining the clarity of clothing textures and patterns;

[0033] S65: Combining the residual transmission diffusion model (RT-DDM) with the flexible noise scheduling method (NS) to optimize the denoising and restoration process, reduce the necessary sampling steps, and speed up the inference;

[0034] S66: gradually estimate and generate a denoised version of the super-resolution result through a diffusion process;

[0035] S67: The system outputs a super-resolution result corresponding to the clothing image input by the user, and provides the user with a clearer and more realistic clothing image.

[0036] On the other hand, the present invention also provides a clothing image super-resolution device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned FGA-D-based clothing image super-resolution method when executing the computer program.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] This paper proposes a frequency-domain guided attention diffusion method (FGA-D), which achieves significant performance improvement in the generation of clothing image super-resolution.

[0039] Firstly, we carefully select and obtain clothing image data and their corresponding super-resolution image true labels from the database to construct a super-resolution clothing dataset.

[0040] Then, the low-resolution clothing images uploaded by users are obtained and pre-processed in the system to facilitate subsequent super-resolution processing;

[0041] Subsequently, a pre-trained wavelet convolutional neural network module (D-CNN) is used to extract key features of the image;

[0042] Next, the frequency domain information splitter module (FDIS) is used to effectively separate the low-frequency components from the high-frequency content in the key features extracted by D-CNN, laying a solid foundation for the subsequent high-frequency detail recovery;

[0043] Secondly, a frequency-domain guided cross-attention module (FDG-CA) is inserted into the denoising UNet network to focus on recovering the high-frequency details of the image, which are crucial to maintaining the clarity of clothing textures and patterns;

[0044] Finally, the residual transmission based diffusion model (RT-DDM) is combined with a flexible noise scheduling method (NS) to optimize the denoising and restoration process and significantly reduce the necessary sampling steps, thereby speeding up the inference speed and improving the overall efficiency.

[0045] The present invention uses an innovative super-resolution generation method to significantly improve the detail richness and quality of super-resolution images while reducing the number of inference steps, effectively avoiding the mode collapse problem and enhancing the diversity and realism of images. In addition, by applying the clothing image super-resolution solution, it can provide e-commerce platforms with clearer and more realistic clothing display images, enhance consumers' online shopping experience, and thus promote sales and market competitiveness. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a logic flow chart of the present invention;

[0047] Figure 2 It is the FGA-D network structure diagram of the present invention;

[0048] Figure 3 is a diagram of the RT-DDM residual connection structure in the FGA-D network of the present invention;

[0049] Figure 4 is a structural diagram of FDG-CA in the FGA-D network of the present invention;

[0050] Figure 5 It is a structural diagram of FDIS in the FGA-D network of the present invention;

[0051] Figure 6 It is a D-CNN structure diagram in the FGA-D network of the present invention;

[0052] Figure 7 This is a diagram showing the super-resolution results of the present invention. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.

[0054] like Figure 1-7 As shown, the embodiment provides a clothing image super-resolution method,

[0055] The following steps are involved:

[0056] (1) First, clothing image data and their corresponding super-resolution image true labels are obtained from the database to form a super-resolution clothing dataset, which is then divided into training set, validation set and test set in the ratio of 8:1:1.

[0057] (2) The user manually uploads the clothing images that need super-resolution into the system and saves them into the system after preprocessing to facilitate subsequent operations.

[0058] (3) A diffusion method based on frequency domain guided attention mechanism (FGA-D) is established to realize the generation of super-resolution clothing images. The model mainly consists of a pre-trained wavelet convolutional neural network module (D-CNN), a frequency domain guided cross attention module (FDG-CA), a diffusion model based on residual transmission (RT-DDM) and a flexible noise scheduling method (NS), which respectively realize the prediction of initial conditional features, the extraction of high-frequency details of the image, the enhancement of the transmission efficiency between high-resolution images and low-resolution images, and the reduction of the diffusion steps of the diffusion model.

[0059] Image super-resolution (SR) is a difficult problem in computer vision, which aims to restore high-resolution (HR) images from low-resolution (LR) images. During image degradation, high-frequency components are lost, and multiple HR images can produce the same LR image, which makes this task subject to serious uncertainty. After the introduction of generative adversarial networks (GANs), the main SR methods based on generative models are driven by GAN methods. However, GAN-based methods are difficult to train and prone to mode collapse, resulting in a lack of diversity in generated samples. With the widespread application of diffusion probability models (DDPMs) in the fields of image synthesis and image restoration, diffusion methods using conditional probability generation have become increasingly popular among researchers. Current diffusion-based image super-resolution methods, such as SR3, directly generate HR images from random noise and use LR images only as conditional inputs to the diffusion process, which not only increases the convergence time, but also inhibits the model's ability to focus on fine-grained information, resulting in the loss of texture details. At the same time, due to the need for hundreds or even thousands of sampling steps, the inference speed is relatively slow. The existing accelerated sampling technology inevitably sacrifices performance to a certain extent, resulting in SR results that are too blurry. To address this problem, this paper proposes a novel and effective diffusion method based on frequency-domain guided attention mechanism (FGA-D) to maintain important pixel-level details when generating super-resolution images and significantly reduce the number of diffusion steps, thereby eliminating the post-acceleration during inference and its associated performance degradation. In addition, a carefully designed noise scheme is used to flexibly control the movement speed and noise intensity during the diffusion process.

[0060] (4) The pre-trained wavelet convolutional neural network module (D-CNN) implemented in the diffusion method based on frequency domain guided attention mechanism (FGA-D) is:

[0061] In order to reduce the additional training cost, the present invention uses a convolutional neural network (CNN) with a small number of parameters to generate initial predictions as conditional features. The CNN is designed to extract the main low-frequency components and some high-frequency components, which is conducive to the diffusion model to recover more complex high-frequency details. In order to ensure its feature extraction ability, the present invention innovatively introduces two other loss functions, namely, L based on fast Fourier transform (FFT) and FFT and L based on discrete wavelet transform (DWT) DWT , and the spatial domain loss function L GT .

[0062] Specifically, L FFT It can be defined as the mean square error (MSE) between the magnitudes of the FFT coefficients of two images:

[0063]

[0064] Among them, M and Represents the frequency domain image results obtained by performing FFT on the real image and the generated image.

[0065] In order to enable the convolutional neural network (CNN) to further restore part of the high-frequency content on the basis of restoring the main low-frequency content, the present invention designs L DWT Loss. Performing DWT on an image decomposes it into four sub-bands: low-low (LL), low-high (LH), high-low (HL), and high-high (HH). The LL sub-band contains the low-frequency content of the image, while the remaining three sub-bands contain the high-frequency components of the image in the horizontal, vertical, and diagonal directions, respectively. The LL sub-band can also be further decomposed similarly to obtain multiple layers of high-frequency components. DWT The method is used to extract the wavelet coefficients of the high-frequency bands H, V, and D, which refer to the high-frequency components in the horizontal, vertical, and diagonal directions, respectively. For both real and generated images, L DWT Calculate the MSE between each high frequency subband:

[0066]

[0067] Among them, H i ,V i ,G i is the subband of the real image in the i-th downsampling, is the subband of the predicted image in the i-th downsampling, and L represents the total number of downsampling times.

[0068] Finally, the present invention also adds a new CNN module named L GT The spatial domain loss is defined as the loss between the real image Y and the generated HR image The MSE value between:

[0069]

[0070] To this end, combining the above loss functions can obtain the total loss function of the D-CNN module as follows:

[0071] L D-CNN =L GT +αL FFT +βL DWT

[0072] Among them, α and β are custom hyperparameters. It is worth noting that in actual use, it is necessary to use image data based on L D-CNN The loss function is used to pre-train the D-CNN module.

[0073] (5) The frequency domain guided cross attention module (FDG-CA) implemented in the diffusion method based on the frequency domain guided attention mechanism (FGA-D) is:

[0074] In the original diffusion model UNet architecture, the encoder features are directly connected to the features obtained by the decoder. This fusion helps the network to effectively integrate high-resolution and low-resolution features, but the lack of the ability to extract high-frequency features easily causes the loss of high-frequency details. To address this problem, this paper proposes a frequency-domain guided cross-attention module (FDG-CA) to restore fine-grained high-frequency details and further improve the quality of super-resolution image generation.

[0075] First, the input image is processed by using the H extracted by the DWT method in the i-th layer D-CNN i ,V i ,G i The coefficients are added to the linear projection to obtain a feature map Q with aggregated high-frequency information:

[0076]

[0077] Then, the feature map R output by the residual block at the i-th scale in the UNet network is i As input and construct its different linear mappings to obtain K and V in the calculation process of the cross attention mechanism:

[0078] K=Conv 1×1 (R i )

[0079] V=Conv 1×1 (R i )

[0080] Finally, the output feature map R can be obtained by the following formula i ′ :

[0081]

[0082] Among them, d k is the number of columns of matrix Q. By adopting the frequency domain to guide the cross attention module, the diffusion model can be helped to better predict high-frequency details.

[0083] (6) The method for implementing the residual transmission-based diffusion model (RT-DDM) in the frequency domain guided attention mechanism diffusion method (FGA-D) is:

[0084] Inspired by the iterative generation paradigm of the diffusion model in capturing complex distributions, it is combined with traditional computer vision methods to iteratively solve the SR problem. To this end, the method of the diffusion model based on residual transmission proposed in this invention reconstructs a new Markov chain as a bridge between HR and LR images. Given any LR image, it can complete the SR task by reverse sampling from the Markov chain.

[0085] Specifically, for the forward process of RT-DDM, the residual between the LR and HR images is denoted as e0, i.e. The core idea of ​​the present invention is to gradually translate the residual e0 through a Markov chain of length T, from x0 to First, introduce the translation sequence It increases monotonically with the time step t, satisfying η1→0 and η T → 1. Then, the distribution transformation based on this translation sequence is as follows:

[0086]

[0087] Among them, when t>1 and α1=η1, α t =η t -η t-1 , k is a hyperparameter that controls the noise variance, which helps increase the flexibility of the design. I is the identity matrix. It is worth noting that it is integrable on the marginal distribution at any time step t, that is:

[0088]

[0089] The design of the above distribution transformation is mainly based on two important principles. The first principle involves the standard deviation. Its purpose is to promote and Assuming the image data falls within the range [0,1], then and The expected distance between The restriction can be expressed mathematically as:

[0090]

[0091] Among them, max[·] represents the pixel-by-pixel maximization operation. The second principle is related to the average parameter, namely It yields the marginal distribution in Equation 1. In addition, and The marginal distribution of and They are two approximate distributions of HR image and LR image respectively. By reconstructing the Markov chain in the above way, the given LR image x0 can be input and inversely sampled to achieve the SR task.

[0092] For the reverse process of RT-DDM, we aim to estimate the posterior distribution by the following formula

[0093]

[0094] in, For arrive The reverse transformation kernel has a learnable parameter θ. Similar to the traditional diffusion model, the present invention adopts The optimization of θ is achieved by minimizing the negative evidence lower bound, which is defined as:

[0095]

[0096] Among them, D KL [·||·] represents the Kullback-Leibler (KL) divergence. Combined with the above formula, its target distribution is can be made more tractable as follows:

[0097]

[0098] Considering the variance parameter and It has nothing to do with x0, so the present invention sets For the average parameter Recalculate and parameterize using the following formula:

[0099]

[0100] Among them, f θ (·) is a deep neural network with parameters θ, which aims to predict By exploring μ θ Different parameterization forms on the above average parameters are found It shows excellent stability and performance. Therefore, the objective function can be simplified by using the average parameter into the calculation formula of the negative evidence lower bound, and its mathematical form is as follows:

[0101]

[0102] in, The weight w can be inferred through multiple trials t This will significantly improve the model performance.

[0103] After obtaining the image feature x0 corresponding to the input image by the pre-trained D-CNN, the above-mentioned residual transmission diffusion model (RT-DDM) is used to calculate the image feature x0 and the predicted image The residual between them is the high-frequency component of the predicted image. First, the present invention designs a frequency domain information splitter module (FDIS), whose input is the image feature x0 and the shallow spatial feature X after adding noise. t, whose output is then fed into the UNet part of the RT-DDM model, where a frequency-domain guided cross-attention method is inserted between the skip links of each layer, and the high-frequency features obtained by the DWT calculation of each layer are used to generate fine-grained and richer detail features.

[0104] (7) The flexible noise scheduling method (NS) implemented in the frequency domain guided attention mechanism diffusion method (FGA-D) is:

[0105] The flexible noise scheduling method (NS) proposed in this invention adopts hyperparameter k and translation sequence To determine the noise schedule during the diffusion process. Specifically, the hyperparameter k adjusts the overall noise intensity during the forward process, and its impact on performance has been empirically discussed in previous studies. Therefore, in the following, we will focus on the design of the noise schedule and the translation sequence. The structure is explained as follows:

[0106] By calculation The latent space state can be known The noise level in Proportional to the proportional factor k, which prompts the present invention to focus on designing Instead of η t Previous studies have shown that Enough hours to ensure By incorporating the additional constraint of η1→0, the present invention sets η1 to and 0.001. For the last step T, we set η T Set to 0.999 to ensure that η T →1. For the intermediate time step, i.e., t∈[2,T-1], A non-uniform geometry schedule is proposed as follows:

[0107]

[0108] in, It is worth noting that β t The selection of b0 is based on β1=0, β T =T-1 and The hyperparameter p controls growth rate.

[0109] The noise scheduling proposed in this invention is highly flexible in three key aspects. First, for smaller values ​​of k, the final state It is able to converge to the detail perturbations around the LR image. Compared with the corruption under Gaussian noise, this design greatly shortens the length of the Markov chain, thereby improving the inference efficiency. Secondly, the hyperparameter p provides precise control over the speed of the change, thereby achieving a trade-off between the fidelity of the SR results and the inference speed. Finally, by setting k = 40 and p = 0.8, FGA-D achieves a diffusion process very similar to LDM (a model that achieves fast diffusion).

[0110] (8) The method for implementing the frequency domain information splitter module (FDIS) in the residual transmission diffusion model (RT-DDM) is:

[0111] For the conditional features extracted by D-CNN, the low-frequency components are mixed with the high-frequency content. Since the diffusion model only needs to restore the high-frequency details, the input low-frequency and high-frequency features have different states: the former mainly assists the generation of global high-frequency components, while the latter needs to provide guidance for the fine-grained generation of images. To this end, the present invention introduces a frequency domain information splitter module (FDIS) to efficiently separate high-frequency and low-frequency information in order to better restore the pixel-level details of the image. In addition, it can also be combined with the above-mentioned flexible noise scheduling scheme to effectively reduce the noise of noisy images with large time steps, thereby obtaining better noise prediction.

[0112] For image features extracted by D-CNN First, a two-dimensional fast Fourier transform (FFT) is performed along the spatial dimension to obtain the frequency domain feature map M:

[0113]

[0114] Among them, FFT(·) represents the two-dimensional fast Fourier transform method (2D FFT), which is incorporated into a residual squeezing and activation module (ResSE), which consists of a pre-trained residual network. In order to achieve adaptive high-pass filtering, a Gaussian high-pass filter is used, and its standard deviation can be calculated by M, and the calculation formula is as follows:

[0115]

[0116] Where l = min(H, W). The operation on the obtained ResSE(M) is to maintain the stability of the value. After calculating σ, the adaptive Gaussian high-pass filter can be directly defined as:

[0117]

[0118] Where D(μ,v) is the distance from the point (μ,v) in the frequency domain to the center point. Then, the Gaussian high-pass filter is multiplied element by element with M to obtain the adaptive high-pass filter feature map M′ :

[0119]

[0120] Finally, the inverse FFT is used to transform M ′ Restore to the spatial domain and obtain the feature map x containing rich high-frequency components HF :

[0121]

[0122] Among them, IFFT(·) represents the inverse two-dimensional fast Fourier transform method. At the same time, M ′ Input into the ResSE module, obtain the attention weight learned in the frequency domain, and then multiply it element by element with x0 to obtain the feature map x containing rich low-frequency information LF :.

[0123]

[0124] The two feature maps dominated by high-frequency and low-frequency components are concatenated in the channel dimension. By explicitly separating the mixed high-frequency and low-frequency components of the input, the network can more effectively utilize both in different ways. For a noisy image with a large time step t The noise component may be so large that it hinders network reasoning. Use adaptive denoising to obtain a partially denoised noisy image

[0125]

[0126] Three feature maps x HF 、x LF , and x0 and All are concatenated in the channel dimension and fed into the UNet network.

[0127] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A clothing image super-resolution method, characterized in that: The following steps are involved: S1: Obtain clothing image data from the database and its corresponding super-resolution image real labels to form a super-resolution clothing dataset, and divide it into training set, validation set and test set according to 8:1:1; S2: The user manually uploads the clothing image that needs to be super-resolution generated in the system, and enters it into the system after preprocessing to save it for subsequent operations; S3: Design a diffusion method based on frequency-domain guided attention mechanism to achieve super-resolution clothing image generation. The diffusion method based on frequency-domain guided attention mechanism consists of a pre-trained wavelet convolutional neural network module, a frequency-domain guided cross-attention module, a diffusion model based on residual transmission and a flexible noise scheduling method. S4: During the training process, the clothing images in the training set are processed by a diffusion method based on the frequency domain guided attention mechanism; S5: Load the model weights and network structure of the diffusion method based on the frequency domain guided attention mechanism trained in S4, and obtain the image that needs to be super-resolution input by the user from the system, and input it into the loaded diffusion method based on the frequency domain guided attention mechanism; S6: During the inference process, the user input image is first fed into the pre-trained D-CNN to extract image features, and then the high- and low-frequency information is extracted through the frequency domain information splitter module. The extracted features are input into the UNet module containing the frequency domain guided cross attention module and the diffusion model of residual transmission. The denoised version of the super-resolution generation result is gradually estimated from the input high- and low-frequency features through a flexible noise scheduling method, and the super-resolution result of the corresponding clothing image is output.

2. A clothing image super-resolution method according to claim 1, characterized in that: The specific operation steps for obtaining clothing image data in step S1 are: S11: Screening clothing images with the term "super-resolution" from the existing clothing database, and dividing the screened images into a low-resolution clothing image set and a high-resolution clothing image set according to the resolution; S12: combining the low-resolution clothing images and the high-resolution clothing images in pairs according to the IDs of the clothing images; S13: labeling the combined clothing pairs, wherein the high-resolution clothing image data is used as the true label and the low-resolution clothing image data is used as the training sample to form a super-resolution clothing dataset; S14: Divide the super-resolution clothing dataset into a training set, a validation set, and a test set according to the ratio of 8:1:

1.

3. The clothing image super-resolution method according to claim 1, characterized in that: The steps of the processing method in step S4 are: Firstly, the input clothing image is used to extract image features using a pre-trained wavelet convolutional neural network module; Then, the extracted image features are input into the diffusion model of residual transmission, and the feature map of low-frequency information and the feature map of high-frequency components are further obtained by combining the noise image and the frequency domain information splitter module. The learned features are spliced ​​in the channel dimension and input into the UNet module. Next, the UNet method equipped with a frequency-domain guided cross-attention module is used to denoise and restore the input features to obtain a high-resolution image; Finally, residual transmission and flexible noise scheduling methods are used to further optimize the entire diffusion process to effectively reduce the number of diffusion steps.

4. A clothing image super-resolution method according to claim 3, characterized in that: The wavelet convolutional neural network module in step S4 is used to reduce additional training costs and extract features of the input image as conditional features. The module uses a convolutional neural network with a small number of parameters to extract the main low-frequency components and some high-frequency components, which helps the diffusion model to recover more complex high-frequency details. In order to ensure the feature extraction capability, a loss function based on fast Fourier transform and a discrete wavelet transform, as well as a joint loss function composed of a spatial domain loss function are introduced to perform collaborative optimization, wherein the fast Fourier transform loss function is optimized by calculating the mean square error between the real image and the predicted image based on the fast Fourier transform coefficients in the frequency domain; the discrete wavelet transform loss function is further restored by calculating the mean square error between the real image and the predicted image in each high-frequency sub-band; the spatial domain loss function directly calculates the mean square error between the real high-resolution image and the predicted image.

5. The clothing image super-resolution method according to claim 3, characterized in that: In step S4, the frequency domain guided cross attention is used to solve the deficiency of the traditional diffusion model in high-frequency detail recovery. The frequency domain information is introduced to enhance the model's ability to capture high-frequency features. The frequency domain information is used to guide the attention mechanism, so that the model is more focused on the high-frequency details of the image during the recovery process to generate a feature map containing rich high-frequency details, thereby improving the quality of the super-resolution image. The specific method is as follows: First, the high-frequency information of the input image is extracted by wavelet transform, and this information is combined with linear projection to aggregate high-frequency features; Then, we use the feature maps extracted by fast Fourier transform to construct different linear mappings, and use the cross-attention mechanism to enhance the model's attention to high-frequency details. Finally, the final output feature map is calculated by combining the aggregated high-frequency information and the output of the cross-attention, which helps the model better predict the high-frequency details.

6. The clothing image super-resolution method according to claim 3, characterized in that: The diffusion model based on residual transmission in step S4 uses the iterative generation paradigm of the diffusion model to solve the image super-resolution problem, and realizes the conversion from low-resolution image to high-resolution image by constructing a new Markov chain as a bridge between high-resolution and low-resolution images. The specific method is: In the forward process, a translation sequence is added to make it monotonically grow with time steps to gradually translate the residual between LR and HR images to achieve the conversion between different distributions. The hyperparameters of controlling the noise variance are designed to increase the flexibility of the design. In the reverse process, the mean and variance variables are recalculated and parameterized through specific formulas to further optimize the training objective function to improve the stability and performance of the model; At the same time, in the denoising UNet network, by adding a frequency domain information splitter module at the input stage and inserting a frequency domain guided cross attention method between the jump links of each layer, the high-frequency and low-frequency information of the image can be better captured to facilitate learning the high-frequency features of the image to generate fine-grained and richer image details.

7. The clothing image super-resolution method according to claim 1, characterized in that: The flexible noise scheduling method in step S4 flexibly controls the movement speed and noise intensity in the diffusion process by designing a noise plan, reasonably adjusting hyperparameters and translation sequences, thereby improving the inference efficiency while maintaining the quality of the super-resolution image. By optimizing the diffusion steps, the necessary number of samples is reduced, the inference speed is accelerated, and the performance degradation is avoided, thereby enhancing the diversity and realism of the image. The specific method is as follows: First, the noise scheduling method uses hyperparameters to adjust the overall noise intensity in the diffusion process, which directly affects the performance of the model; Then, by constructing a specific translation sequence, it is possible to control the growth rate of noise, thereby adapting to the diffusion requirements of different stages; Next, for the intermediate time steps, the present invention proposes a non-uniform geometric scheduling scheme to adapt to the noise levels of different time steps; Finally, for the final stage of the diffusion process, the distribution conversion speed is precisely controlled by setting hyperparameters, achieving a trade-off between the fidelity of the super-resolution results and the inference speed, ensuring that the final state can converge to the detailed perturbations around the low-resolution image, thereby shortening the length of the Markov chain and improving inference efficiency.

8. The clothing image super-resolution method according to claim 3, characterized in that: In step S4, the frequency domain information splitter module is used to efficiently separate the high-frequency and low-frequency information in the input image features, provide clear high-frequency details and low-frequency background information for the diffusion model, so as to more accurately restore the pixel-level details of the image, effectively separate the high-frequency and low-frequency information from the mixed frequency components, and provide important details and background information for the generation of super-resolution images, thereby significantly improving the quality of image restoration. The specific method is: First, a 2D fast Fourier transform is performed on the image features extracted by the wavelet convolutional neural network module to convert the spatial domain features into the frequency domain for subsequent frequency domain processing; Then, in the frequency domain, an adaptive Gaussian high-pass filter is element-wise multiplied with the frequency domain feature map to enhance the high-frequency components of the image, wherein the standard deviation of the filter is dynamically calculated based on the frequency domain feature map to achieve adaptive high-pass filtering; Subsequently, the processed frequency domain feature map is converted back to the spatial domain through inverse 2D fast Fourier transform to obtain a feature map containing high-frequency details; Next, the attention weights learned in the frequency domain are multiplied element-wise with the low-frequency feature map to enhance the expression of low-frequency information. Finally, the processed high-frequency and low-frequency feature maps are concatenated in the channel dimension to provide the subsequent network layers with features that integrate high- and low-frequency information. At the same time, FDIS can be combined with a flexible noise scheduling method to adaptively denoise noisy images with large time steps to reduce the impact of noise on network reasoning.

9. The clothing image super-resolution method according to claim 1, characterized in that: The method for performing reasoning in step S6 using a diffusion method based on a frequency domain guided attention mechanism according to the clothing image input by the user is: S61: The clothing image uploaded by the user is first pre-processed in the system to ensure that the image data is suitable for subsequent super-resolution operations; S62: The preprocessed image is input into the pre-trained wavelet convolutional neural network module to extract the key features of the image, including the main low-frequency components and some high-frequency components; S63: Use the frequency domain information splitter module to effectively separate the low-frequency and high-frequency components in the features extracted by the wavelet convolutional neural network module, providing a basis for the recovery of high-frequency details; S64: The features after separation of low-frequency and high-frequency components are input into the UNet structure of the residual propagation diffusion model, and the cross-attention module is guided by the frequency domain to focus on restoring the high-frequency details of the image, which are crucial to maintaining the clarity of clothing textures and patterns; S65: Combining a diffusion model based on residual transmission with a flexible noise scheduling method to optimize the denoising and restoration process, reduce the necessary sampling steps, and speed up inference; S66: gradually estimate and generate a denoised version of the super-resolution result through a diffusion process; S67: The system outputs a super-resolution result corresponding to the clothing image input by the user, and provides the user with a clearer and more realistic clothing image.

10. A clothing image super-resolution device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor runs the computer program, the steps of the clothing image super-resolution method based on the diffusion method of the frequency domain guided attention mechanism are implemented.

Images