A bottom layer vision pre-training method without images

Abstract

This invention relates to the field of low-level visual pre-training technology and discloses a low-level visual pre-training method without images. The method includes: randomly sampling from a Gaussian distribution to obtain a pure Gaussian noise image; performing a frequency-aware masking strategy on the pure Gaussian noise image to obtain an input image; spectral information reconstruction pre-training, modifying the structure of the low-level visual model to be pre-trained for the first time; and fine-tuning for a downstream degradation restoration task, retaining the parameters of the first modified low-level visual model and modifying the structure of the low-level visual model a second time for training the downstream task. Based on the class differences of interference patterns exhibited by degradation information in the spectral domain, this invention proposes a frequency-aware masking strategy, breaking the traditional pre-training paradigm of learning general task information from data. It achieves universally transferable representation learning with degradation type agnosticity using only a randomly generated Gaussian noise image at extremely low computational cost.

Description

Technical Field

[0001] This invention relates to the field of low-level visual pre-training technology, and specifically to a low-level visual pre-training method that does not require images. Background Technology

[0002] Image pre-training has garnered significant attention in computer vision to mitigate overfitting and achieve further improvements with limited data. However, unlike its remarkable success in high-level vision, the application of pre-training in low-level vision remains immature. The prevailing view attributes this to the lack of large-scale low-level vision datasets comparable to ImageNet in the community. Influenced by the view that data size plays a decisive role in pre-training effectiveness, existing techniques attempt to synthesize pairs of degraded data on ImageNet using various data corruption pipelines and perform corresponding image restoration tasks. These data-driven pre-training methods require the manual introduction of target degradation information during the pre-training phase, a requirement that limits their applicability to tasks where synthesizing degradation is costly.

[0003] The emergence of Degenerative Autoencoders (DegAEs) has enabled high-cost tasks to benefit from pre-training. Instead of focusing on data acquisition, it employs a novel pre-training paradigm that learns general low-level visual representations by transferring degradations between images. This new paradigm, after fine-tuning, significantly improves the performance of downstream image restoration tasks, even if the target degradation was not present during the pre-training phase. However, DegAEs do not deviate from the traditional pre-training paradigm of learning from data, as they still rely on manually generating over 10GB of degradation data pairs. Furthermore, it requires an additional degradation information injection module, which demands substantial computational resources and can potentially introduce task bias in downstream applications. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a low-level visual pre-training method that does not require images.

[0005] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0006] A low-level vision pre-training method that does not require images includes the following steps:

[0007] Step 1: Randomly sample from a Gaussian distribution to obtain a pure Gaussian noise image I. o ;

[0008] Step 2, for the pure Gaussian noise image I o Execute a frequency-aware masking strategy to obtain the input image I for the pre-training of the underlying visual model. mThe frequency-aware masking strategy includes a random mask generation process and a spectrum masking process; the random mask generation process is used to simulate the uncertainty of degraded information interference; the spectrum masking process is used to simulate the interference of degraded information.

[0009] Step 3, Pre-training for spectrum information reconstruction: The structure of the underlying visual model that needs to be pre-trained is modified for the first time, including: modifying the underlying visual model so that the modified underlying visual model is used as the encoder module, and adding a decoder module.

[0010] Step 4, fine-tuning of downstream degradation recovery task: retain the parameters of the pre-trained, first modified underlying visual model, and modify the structure of the underlying visual model a second time. The process of modifying the structure of the underlying visual model a second time includes retaining the encoder module and replacing the decoder module with the adaptation module for training of downstream tasks.

[0011] Furthermore, step one specifically includes:

[0012] Define a Gaussian distribution I(X) ~ N(μ,δ) 2 ), where μ = 0, δ = 1, a two-dimensional matrix I is obtained by random sampling from a Gaussian distribution, and then replicated in three channels:

[0013] I = Sample(N(μ,δ) 2 ));

[0014] I0 = concat[I,I,I];

[0015] Where Sample(·) represents a random sampling operation; concat[·,·] represents a concatenation operation, ultimately resulting in a pure Gaussian noise image I0.

[0016] Furthermore, the random mask generation process specifically includes:

[0017] The amplitude spectrum of the input image is considered as a combination of concentric squares with a unit width. Concentric squares are randomly sampled at a set ratio and following a uniform distribution to obtain a random mask M of the same size as the input image, in order to eliminate information redundancy caused by the continuity of the frequency distribution.

[0018] R = min(W / 2, H / 2);

[0019] N = R * r;

[0020] M = FAM(R,N);

[0021] Where W and H are the length and width of the random mask, respectively, and are the same size as the input image; R is the maximum sampling radius of the mask box; r is the sampling ratio; N is the number of mask boxes required in the mask generation process; FAM(·) represents the process of uniformly and randomly sampling concentric square boxes.

[0022] Furthermore, the spectral masking process specifically includes: mapping the pure Gaussian noise image I0 from the spatial domain to the frequency domain through Fourier transform, thereby obtaining the corresponding amplitude spectrum A. o and phase spectrum P o Then, the random mask M obtained through the random mask generation process is used to analyze the amplitude spectrum A. o Perform a masking operation to obtain the amplitude spectrum with information corrupted. Finally, the phase spectrum P with complete information is obtained through inverse Fourier transform. o Amplitude spectrum with information destruction Mapping back to the spatial domain yields the input image I for the pre-training of the underlying visual model. m :

[0023] A o ,P o =FFT(I0);

[0024]

[0025]

[0026] FFT(·) is the Fourier transform, which maps the image from the spatial domain to the frequency domain; IFFT(·,·) is the inverse Fourier transform, which maps the image from the frequency domain back to the spatial domain.

[0027] Furthermore, the pre-training for spectrum information reconstruction specifically includes:

[0028] The structure of the original low-level visual model is modified as follows: the final output dimension of the original low-level visual model is enlarged and used as an encoder module to output intermediate features; if the original low-level visual model has global residual settings, these settings are removed to align with the modified output dimension; a decoder module is added for image reconstruction during the pre-training stage.

[0029] I r =ResBlock(E(I m ));

[0030] Where ResBlock(·) is the decoder module, E is the modified underlying visual model of the output channel, and I... r To reconstruct the image;

[0031] The input is reconstructed by predicting missing values ​​in the amplitude spectrum. The loss function L includes pixel-level reconstruction loss and amplitude spectrum reconstruction loss.

[0032]

[0033] In the formula, λ is the balancing weight; Let A denote the inverse of the random mask M, ⊙ be the Hadamard product between matrices, and A o and A r They are I o and I r The corresponding amplitude spectrum; where the first term is the pixel-level reconstruction loss and the second term is the amplitude spectrum reconstruction loss.

[0034] Furthermore, the fine-tuning of the downstream degradation recovery task specifically includes:

[0035] In the downstream task fine-tuning stage, the encoder is initialized with the parameters of the pre-trained, first-modified underlying visual model, and the decoder module is replaced with an adaptation module consisting of convolutional layers.

[0036] I out =Conv(E(I) in ));

[0037] Among them, I in E represents the image to be restored containing degraded information, E represents the pre-trained, first-modified low-level visual model, and Conv represents the convolutional layer.

[0038] Then, fine-tuning is performed based on the specific training strategy of the underlying visual model.

[0039] Compared with the prior art, the beneficial technical effects of the present invention are:

[0040] 1. Based on the class differences of interference patterns exhibited by degradation information in the spectral domain, this invention proposes a frequency-aware masking strategy, breaking the traditional pre-training paradigm of learning general information of the task from data. Using only a randomly generated Gaussian noise image, it achieves general transferability representation learning with unknown degradation type at extremely low computational cost, releasing the huge potential of pre-training under data-constrained conditions.

[0041] 2. The frequency-aware masking strategy in this invention only requires a randomly generated Gaussian noise image during runtime, making the general prior representation learned by the model invariable to degradation, thus making it applicable to different low-level visual tasks. That is, the algorithm constructed in this invention has good generalization performance. Attached Figure Description

[0042] Figure 1 A flowchart of an image-free low-level visual pre-training method provided in an embodiment of the present invention;

[0043] Figure 2This is a schematic diagram of an image-free low-level visual pre-trained network architecture provided in an embodiment of the present invention. Detailed Implementation

[0044] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0045] This invention addresses the data dependency problem of current mainstream low-level visual pre-training methods by designing an image-free low-level visual pre-training method independent of downstream tasks. Compared with existing low-level visual pre-training methods, the method in this invention can utilize task-agnostic data for low-level visual pre-training, simplifying the complex data collection and synthesis process and unleashing enormous potential under data-limited conditions. Furthermore, the fact that the entire pre-training process uses only a single randomly generated Gaussian noise image means very low computational cost, achieving a combined improvement in performance and efficiency.

[0046] like Figure 1 As shown, a noisy image is first randomly sampled from a Gaussian distribution to meet the data requirements for pre-training. Then, the noisy image is preprocessed, and a frequency-aware masking strategy is executed, consisting of two processes: random mask generation and spectral masking. The original model is modified to serve as the encoder, and a decoder module is added to perform pre-training for spectral information reconstruction. The model structure is modified a second time, retaining the parameters of the pre-trained model, and adding an adaptation module to fine-tune the downstream degradation recovery task, thereby improving the performance of the original model on the downstream task.

[0047] 1. Generation of pre-training noisy data

[0048] Theoretically, it is proven that pre-training for masked image modeling ensures that the general representation learned by the model is unrelated to the image content. Furthermore, the gray-world color invariance theory states that the average values ​​of the RGB channels of a natural image converge to the same gray value k. Therefore, Gaussian random noise is used as a simplified representation of the natural image. Specifically, for the input, a Gaussian distribution I(X) ~ N(μ,δ) is defined. 2 ), where μ = 0, δ = 1, a two-dimensional matrix I is randomly sampled from it, and then copied in three channels:

[0049] I = Sample(N(μ,δ) 2 ));

[0050] I0 = concat[I,I,I];

[0051] Where Sample(·) represents a random sampling operation; concat[·,·] represents a concatenation operation, and finally a pure Gaussian noise image I0 is obtained as the data for the pre-training process.

[0052] 2. Frequency-aware masking strategy

[0053] Frequency-aware masking (FAM) of the amplitude spectrum is performed for the following reasons: (1) Degradation affects the original image by randomly destroying low- or high-frequency information in the Fourier space. (2) Phase represents content information and spatial details, while amplitude represents global statistical characteristics. When designing interface tasks, it is necessary to avoid the influence of specific image content.

[0054] 2.1 Random Mask Generation

[0055] The amplitude spectrum of the image is viewed as a combination of concentric squares with a unit width. These concentric squares are then randomly sampled at a certain ratio following a uniform distribution to obtain a mask M the same size as the input image. This operation largely eliminates information redundancy caused by the continuity of the frequency distribution.

[0056] R = min(W / 2, H / 2);

[0057] N = R * r;

[0058] M = FAM(R,N);

[0059] Where W and H are the length and width of the mask, respectively, and are the same size as the input image; R is the maximum sampling radius of the mask box; r is the sampling ratio; N is the number of mask boxes required in the mask generation process; FAM(·) is the process of uniformly and randomly sampling these concentric square boxes.

[0060] 2.2 Spectrum Masking

[0061] The pure Gaussian noise image I0 is mapped from the spatial domain to the frequency domain using Fourier transform, yielding its amplitude spectrum A. o and phase spectrum P o Then, a random mask M is used to visualize the amplitude spectrum A. o Perform a masking operation to obtain the amplitude spectrum with corrupted information. Finally, the phase spectrum P with complete information is obtained through inverse Fourier transform. o Amplitude spectrum with information destruction Mapping back to the spatial domain yields the preprocessed network input I. m The complete process can be expressed by the following formula:

[0062] A o ,P o =FTT(I0);

[0063]

[0064]

[0065] FFT(·) is the Fourier transform, which maps the image from the spatial domain to the frequency domain; IFFT(·,·) is the inverse Fourier transform, which maps the image from the frequency domain back to the spatial domain.

[0066] 3. Pre-training for spectrum information reconstruction

[0067] 3.1 Model Structure Modification

[0068] For the encoder, the final output dimension of the original model is increased to serve as the encoder, outputting intermediate features. If the original model has a global residual setting, this setting is removed to align with the modified output dimension.

[0069] For the decoder, since it is only used for image reconstruction during the pre-training phase, its structure can be flexibly designed in a way that is independent of the encoder design. It can be of any form, as long as its input is compatible with the encoder's output and is simple enough. Specifically, a convolutional neural network with four residual blocks can be used as the decoder.

[0070] I r =ResBlock(E(I m ));

[0071] Here, ResBlock(·) is a convolutional neural network consisting of four residual blocks, and E is the target model with modified output channels.

[0072] 3.2 Prediction of missing values ​​on the amplitude spectrum

[0073] Specifically, the loss function of the reconstruction process includes amplitude spectrum reconstruction loss and pixel-level reconstruction loss, which can be expressed as:

[0074]

[0075] In the formula, I o For the original Gaussian noise image, I r For reconstructing the image, λ is the balancing weight. Let A denote the inverse of the random mask M, ⊙ be the Hadamard product between matrices, and A o and A r They are I o and I r The corresponding amplitude spectrum. The first part of the formula is the pixel-level reconstruction loss, and the second part is the amplitude spectrum reconstruction loss. We follow the settings in MAE and only calculate the masked region in practical applications.

[0076] 4. Minor adjustments to downstream degradation and restoration tasks.

[0077] In the downstream task fine-tuning stage, the encoder is initialized with the parameters pre-trained in the first stage, and the decoder module consisting of four residual blocks is replaced with a convolutional layer.

[0078] I out =Conv(E(I) in ));

[0079] Among them, I in The input is the image to be restored, which contains degraded information. E is the target model pre-trained in the first stage, and Conv represents a simple convolutional layer. Fine-tuning is then performed according to the model-specific training strategy.

[0080] In summary, this invention proposes a low-level visual pre-training method that does not require images. Based on the categorical differences in interference patterns exhibited by degradation information in the spectral domain, a frequency-aware masking strategy is designed for the first time from a frequency domain perspective to simulate the interference of degradation information on the spectrum. The data requirement is only a single randomly generated Gaussian noise image, which eliminates the dependence on data in traditional pre-training methods and greatly reduces the pre-training cost.

[0081] In theory, this invention is applicable to any end-to-end underlying visual model architecture. In a preferred embodiment, the underlying visual model can be Restormer, Uformer, or SwinIR.

[0082] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0083] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A low-level visual pre-training method without images, comprising the following steps: Step 1: Randomly sample from a Gaussian distribution to obtain a pure Gaussian noise image. Specifically, this includes: setting a Gaussian distribution. ,in, , A two-dimensional matrix is ​​obtained by random sampling from a Gaussian distribution. Perform three-channel replication: ; ;in, Indicates a random sampling operation; This indicates a stitching operation, ultimately resulting in a pure Gaussian noise image. ; Step 2, process the image with pure Gaussian noise. A frequency-aware masking strategy is executed to obtain the input image for the pre-training of the underlying visual model. The frequency-aware masking strategy includes a random mask generation process and a spectral masking process. The random mask generation process is used to simulate the uncertainty of degradation information interference; the spectral masking process is used to simulate the interference of degradation information. Specifically, the random mask generation process includes: treating the amplitude spectrum of the input image as a combination of concentric squares with unit width, randomly sampling concentric squares at a set ratio and following a uniform distribution to obtain a random mask M of the same size as the input image, so as to eliminate information redundancy caused by the continuity of frequency distribution. ; ; Where W and H are the length and width of the random mask, respectively, and are the same size as the input image; R is the maximum sampling radius of the mask box; r is the sampling ratio; and N is the number of mask boxes required for the mask generation process. This describes the process of uniformly and randomly sampling concentric square frames. The spectral masking process specifically includes: transforming the pure Gaussian noise image using Fourier transform. Mapping from the spatial domain to the frequency domain yields the corresponding amplitude spectra. and phase spectrum Then, the random mask M obtained through the random mask generation process is used to analyze the amplitude spectrum. Perform a masking operation to obtain the amplitude spectrum with information corrupted. Finally, the complete phase spectrum of the information is obtained through inverse Fourier transform. Amplitude spectrum with information destruction Mapping back to the spatial domain yields the input image for the pre-training of the underlying visual model. : ; ; ;in, The Fourier transform maps the image from the spatial domain to the frequency domain. This is the inverse Fourier transform, which maps the image from the frequency domain back to the spatial domain; Step 3, Pre-training for spectrum information reconstruction: The structure of the underlying visual model that needs to be pre-trained is modified for the first time, including: modifying the underlying visual model so that the modified underlying visual model is used as the encoder module, and adding a decoder module. Step 4, fine-tuning of downstream degradation recovery task: retain the parameters of the pre-trained, first modified underlying visual model, and modify the structure of the underlying visual model a second time. The process of modifying the structure of the underlying visual model a second time includes retaining the encoder module and replacing the decoder module with the adaptation module for training of downstream tasks.

2. The low-level visual pre-training method without images according to claim 1, characterized in that, The pre-training for spectrum information reconstruction specifically includes: The structure of the original low-level visual model is modified as follows: the final output dimension of the original low-level visual model is enlarged and used as an encoder module to output intermediate features; if the original low-level visual model has global residual settings, these settings are removed to align with the modified output dimension; a decoder module is added for image reconstruction during the pre-training stage. ； in, For decoder modules, The underlying visual model of the output channel was modified. To reconstruct the image; The input is reconstructed by predicting missing values ​​in the amplitude spectrum; the loss function is... Including pixel-level reconstruction loss and amplitude spectrum reconstruction loss: ； In the formula, To balance the weights; Denotes the inverse of the random mask M. It is the Hadamard product between matrices. and They are and The corresponding amplitude spectrum; where the first term is the pixel-level reconstruction loss and the second term is the amplitude spectrum reconstruction loss.

3. The low-level visual pre-training method without images according to claim 1, characterized in that, The fine-tuning of the downstream degradation restoration task specifically includes: In the downstream task fine-tuning stage, the encoder is initialized with the parameters of the pre-trained, first-modified underlying visual model, and the decoder module is replaced with an adaptation module consisting of convolutional layers. ； in, For the image to be restored, which contains information about deterioration. This is the pre-trained, first-modification low-level visual model. Indicates a convolutional layer; Then, fine-tuning is performed based on the specific training strategy of the underlying visual model.

Images