Image-free underlying vision pre-training method

Abstract

The invention relates to the technical field of underlying vision pre-training, and discloses an image-free underlying vision pre-training method, which comprises the following steps of: randomly sampling from Gaussian distribution to obtain a pure Gaussian noise image; executing a frequency perception masking strategy on the pure Gaussian noise image to obtain an input image; performing spectrum information reconstruction pre-training, and performing first modification on the structure of the bottom-layer visual model needing pre-training; and performing fine tuning on the downstream degradation and restoration task, retaining the parameters of the pre-trained underlying visual model after the first modification, and modifying the structure of the underlying visual model for the second time so as to train the downstream task. According to the method, a frequency perception masking strategy is provided based on the category difference of interference modes shown by degradation information in a frequency spectrum domain, a traditional pre-training normal form of learning task general information from data is broken, and only one randomly generated Gaussian noise image is used, so that the probability of interference is reduced; degradation type-agnostic general transferability representation learning is realized at extremely low calculation cost.

Description

Technical Field

[0001] The present invention relates to the technical field of underlying visual pre-training, and in particular to an underlying visual pre-training method without images. Background Art

[0002] Image pre-training has received extensive attention in computer vision to alleviate the overfitting problem and bring further improvements in the case of limited data. However, unlike the remarkable success achieved in high-level vision, the application of pre-training in low-level vision is still immature. The mainstream view is that this is due to the lack of large-scale low-level vision datasets comparable to ImageNet in the community. Influenced by the view that data scale plays a decisive role in the effectiveness of pre-training, existing techniques attempt to synthesize paired degraded data on ImageNet by using various data corruption pipelines and perform corresponding image restoration tasks. These data-driven pre-training methods require artificial introduction of target degradation information in the pre-training stage, a requirement that limits their applicability to tasks where synthetic degradation is expensive.

[0003] The emergence of Degraded Autoencoder (DegAE) enables high-cost tasks to benefit from pre-training. Instead of focusing on data acquisition, it designs a new pre-training paradigm to learn universal low-level visual representations by transferring degradations between images. This new paradigm significantly improves the performance of downstream image restoration tasks after fine-tuning, even if the target degradation does not appear in the pre-training stage. However, DegAE does not deviate from the traditional pre-training paradigm of learning from data. Because it still relies on manually generating more than 10G degraded data pairs. In addition, it requires an additional degradation information injection module, which requires a lot of computing resources and may cause task deviations in downstream applications. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a low-level visual pre-training method without images.

[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0006] A low-level visual pre-training method without images, comprising the following steps:

[0007] Step 1: Randomly sample from the Gaussian distribution to obtain a pure Gaussian noise image I o ;

[0008] Step 2: For the pure Gaussian noise image I o Execute the frequency-aware masking strategy to obtain the input image I of the underlying visual model pre-training m; The frequency-aware masking strategy includes a random mask generation process and a spectrum masking process; the random mask generation process is used to simulate the uncertainty of the interference of the degraded information; the spectrum masking process is used to simulate the interference of the degraded information;

[0009] Step 3: pre-training of spectrum information reconstruction: modifying the structure of the underlying visual model that needs to be pre-trained for the first time, including: modifying the underlying visual model so that the modified underlying visual model is used as an encoder module, and adding a decoder module;

[0010] Step 4: Fine-tuning of downstream degraded restoration tasks: retain the parameters of the pre-trained underlying visual model after the first modification, and modify the structure of the underlying visual model for the second time. The process of modifying the structure of the underlying visual model for the second time includes retaining the encoder module and replacing the decoder module with the adaptation module for training downstream tasks.

[0011] Furthermore, step one specifically includes:

[0012] Set Gaussian distribution I(X)~N(μ,δ 2 ), where μ = 0, δ = 1, and a two-dimensional matrix I is obtained by random sampling from a Gaussian distribution, and three-channel replication is performed:

[0013] I=Sample(N(μ,δ 2 ));

[0014] I 0 =concat[I,I,I];

[0015] Among them, Sample(·) represents a random sampling operation; concat[·,·] represents a concatenation operation, and finally a pure Gaussian noise image I is obtained. 0 .

[0016] Furthermore, the random mask generation process specifically includes:

[0017] The amplitude spectrum of the input image is regarded as a combination of concentric squares with unit width. Concentric squares are randomly sampled in a set ratio and following a uniform distribution to obtain a random mask M of the same size as the input image to eliminate the information redundancy caused by the continuity of the frequency distribution:

[0018] R = min(W / 2,H / 2);

[0019] N = R * r;

[0020] M = FAM (R, N);

[0021] Where W and H are the length and width of the random mask, which are the same as the size of the input image; R is the maximum sampling radius of the mask box; r is the sampling ratio; N is the number of mask boxes required in the mask generation process; FAM(·) represents the process of uniformly randomly sampling concentric square boxes.

[0022] Furthermore, the spectrum masking process specifically includes: transforming the pure Gaussian noise image I 0 Mapping from the spatial domain to the frequency domain, we get the corresponding amplitude spectrum A o and phase spectrum P o , and then use the random mask M obtained by the random mask generation process to the amplitude spectrum A o Perform masking operation to obtain the amplitude spectrum with destroyed information Finally, the complete phase spectrum P is obtained by inverse Fourier transform. o Amplitude spectrum with information destroyed Mapping back to the spatial domain, we get the input image I of the underlying visual model pre-training m :

[0023] A o ,P o =FFT(I 0 );

[0024]

[0025]

[0026] Among them, FFT(·) is the Fourier transform, which maps the image from the spatial domain to the frequency domain; IFFT(·,·) is the inverse Fourier transform, which maps the image from the frequency domain back to the spatial domain.

[0027] Furthermore, the spectrum information reconstruction pre-training specifically includes:

[0028] Modify the structure of the original underlying visual model: After the final output dimension of the original underlying visual model is enlarged, it is used as an encoder module to output intermediate features; if the original underlying visual model has a global residual setting, remove the global residual setting to align the modified output dimension; add a decoder module for image reconstruction in the pre-training stage:

[0029] I r =ResBlock(E(I m ));

[0030] Among them, ResBlock(·) is the decoder module, E is the underlying visual model with modified output channels, and I r To reconstruct the image;

[0031] The input is reconstructed by predicting the missing values ​​on the amplitude spectrum. The loss function L includes pixel-level reconstruction loss and amplitude spectrum reconstruction loss:

[0032]

[0033] In the formula, λ is the balance weight; represents the inverse of the random mask M, ⊙ is the Hadamard product between matrices, A o and A r I o and I r The corresponding amplitude spectrum; the first term is the pixel-level reconstruction loss, and the second term is the amplitude spectrum reconstruction loss.

[0034] Furthermore, the downstream degradation recovery task fine-tuning specifically includes:

[0035] In the downstream task fine-tuning phase, the encoder is initialized with the parameters of the pre-trained first modified underlying visual model, and the decoder module is replaced by an adaptation module consisting of convolutional layers:

[0036] I out =Conv(E(I in ));

[0037] Among them, I in is the image to be restored containing degraded information, E is the underlying visual model after the first modification after pre-training, and Conv represents the convolutional layer;

[0038] It is then fine-tuned according to the specific training strategy of the underlying vision model.

[0039] Compared with the prior art, the beneficial technical effects of the present invention are:

[0040] 1. Based on the category differences of interference patterns manifested by degradation information in the spectral domain, this paper proposes a frequency-aware masking strategy, which breaks the traditional pre-training paradigm of learning general task information from data. It only uses a randomly generated Gaussian noise image to achieve universal transferable representation learning that is agnostic to the degradation type at an extremely low computational cost, thus releasing the huge potential of pre-training under data-constrained conditions.

[0041] 2. The frequency-aware masking strategy in the present invention only requires a randomly generated Gaussian noise image during operation, so that the general prior representation learned by the model is degradation-independent, and thus suitable for different underlying visual tasks. That is, the algorithm constructed by the present invention has good generalization performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flowchart of a low-level visual pre-training method without images provided in an embodiment of the present invention;

[0043] Figure 2 A schematic diagram of an underlying visual pre-training network architecture without the need for images provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0045] In view of the data dependency problem of the current mainstream underlying visual pre-training methods, the present invention designs a low-level visual pre-training method that is independent of downstream tasks and does not require images. Compared with the existing low-level visual pre-training methods, the method in the present invention can use task-agnostic data for low-level visual pre-training, simplify the complex data collection and synthesis process, and release huge potential under the condition of limited data scale. In addition, the entire pre-training process only uses a randomly generated Gaussian noise image, which means that the computational cost is very low, achieving a joint improvement in performance and efficiency.

[0046] like Figure 1 As shown in the figure, first, a noise image is randomly sampled from the Gaussian distribution to meet the data requirements of pre-training; then, the noise image is preprocessed and a frequency-aware masking strategy is implemented, with two processes: random mask generation and spectrum masking; the original model is modified to use it as an encoder, and a decoder module is added to perform pre-training of spectrum information reconstruction; the model structure is modified a second time, the parameters of the pre-trained model are retained, and an adaptation module is added to fine-tune the downstream degradation restoration task, thereby improving the performance of the original model on downstream tasks.

[0047] 1. Pre-training Noise Data Generation

[0048] Theoretical proof: Masked image modeling pre-training makes the general representation learned by the model irrelevant to the image content. In addition, the gray world color invariance theory points out that the average values ​​of the three RGB channels of natural images converge to the same gray value k. Therefore, Gaussian random noise is used as a simplified representation of natural images. Specifically, for the input, the Gaussian distribution I(X)~N(μ,δ 2 ), where μ = 0, δ = 1, and a two-dimensional matrix I is obtained by random sampling, and three-channel replication is performed:

[0049] I=Sample(N(μ,δ 2 ));

[0050] I 0 =concat[I,I,I];

[0051] Among them, Sample(·) represents a random sampling operation; concat[·,·] represents a concatenation operation, and finally a pure Gaussian noise image I is obtained. 0As data for the pre-training process.

[0052] 2. Frequency-aware masking strategy

[0053] Frequency-aware masking (FAM) is performed on the amplitude spectrum for the following reasons: (1) The degradation effect on the original image manifests itself as random destruction of low-frequency or high-frequency information in Fourier space. (2) The phase represents content information and spatial details, while the amplitude represents global statistical characteristics. When designing pretext tasks, it is necessary to avoid the influence of specific image content.

[0054] 2.1 Random Mask Generation

[0055] The amplitude spectrum of the image is regarded as a combination of concentric squares with unit width. Then, these concentric squares are randomly sampled in a certain proportion following a uniform distribution to obtain a mask M of the same size as the input image. This operation largely eliminates the information redundancy caused by the continuity of the frequency distribution.

[0056] R = min(W / 2,H / 2);

[0057] N = R * r;

[0058] M = FAM (R, N);

[0059] Where W and H are the length and width of the mask, respectively, which are the same size as the input image; R is the maximum sampling radius of the mask box; r is the sampling ratio; N is the number of mask boxes required in the mask generation process; and FAM(·) is the process of uniformly randomly sampling these concentric square boxes.

[0060] 2.2 Spectrum Masking

[0061] Through Fourier transform, the pure Gaussian noise image I 0 Mapping from the spatial domain to the frequency domain, we get the amplitude spectrum A o and phase spectrum P o , and then use the random mask M to the amplitude spectrum A o Perform masking operation to obtain the amplitude spectrum with destroyed information Finally, the complete phase spectrum P is obtained by inverse Fourier transform. o Amplitude spectrum with information destroyed Mapping back to the spatial domain, we get the preprocessed network input I m , the complete process is expressed by the formula:

[0062] A o ,P o =FTT(I 0 );

[0063]

[0064]

[0065] Among them, FFT(·) is the Fourier transform, which maps the image from the spatial domain to the frequency domain; IFFT(·,·) is the inverse Fourier transform, which maps the image from the frequency domain back to the spatial domain.

[0066] 3. Spectral Information Reconstruction Pre-training

[0067] 3.1 Model structure modification

[0068] For the encoder, expand the final output dimension of the original model so that it can be used as an encoder to output intermediate features. If the original model has a global residual setting, remove the setting to align the modified output dimension.

[0069] For the decoder, since it is only used for image reconstruction in the pre-training stage, its structure can be flexibly designed in a way that is independent of the encoder design. As long as its input is compatible with the encoder output and is simple enough, it can be of any form. Specifically, a convolutional neural network with four residual blocks is used as the decoder.

[0070] I r =ResBlock(E(I m ));

[0071] Among them, ResBlock(·) is a convolutional neural network composed of four residual blocks, and E is the target model with modified output channels.

[0072] 3.2 Missing value prediction on the amplitude spectrum

[0073] Specifically, the loss function of the reconstruction process includes the amplitude spectrum reconstruction loss and the pixel-level reconstruction loss, which can be expressed as:

[0074]

[0075] In the formula, I o is the original Gaussian noise image, I r To reconstruct the image, λ is the balancing weight. represents the inverse of the random mask M, ⊙ is the Hadamard product between matrices, A o and A r I o and I r The corresponding amplitude spectrum. The first part of the formula is the pixel-level reconstruction loss, and the second part is the amplitude spectrum reconstruction loss. We follow the setting in MAE and only calculate the masked area in practical applications.

[0076] 4. Fine-tuning of downstream degradation recovery tasks

[0077] In the downstream task fine-tuning stage, the encoder is initialized with the parameters pre-trained in the first stage, and the decoder module consisting of four residual blocks is replaced by a convolutional layer.

[0078] I out =Conv(E(I in ));

[0079] Among them, I in is the image input to be restored containing degraded information, E is the target model pre-trained in the first stage, and Conv represents a simple convolutional layer. It is then fine-tuned according to the model-specific training strategy.

[0080] In summary, the present invention proposes a low-level visual pre-training method that does not require images. Based on the category differences of the interference patterns of degraded information in the spectrum domain, a frequency-aware masking strategy is designed for the first time from the frequency domain perspective to simulate the interference of degraded information on the spectrum. The data requirement is only a randomly generated Gaussian noise picture, which gets rid of the dependence of traditional pre-training methods on data and greatly reduces the pre-training cost.

[0081] Theoretically, the present invention is applicable to any end-to-end underlying visual model architecture. In a preferred embodiment, the underlying visual model may adopt Restormer, Uformer or SwinIR.

[0082] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0083] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A low-level visual pre-training method without images, comprising the following steps: Step 1: Randomly sample from the Gaussian distribution to obtain a pure Gaussian noise image I o ; Step 2: For the pure Gaussian noise image I o Execute the frequency-aware masking strategy to obtain the input image I of the underlying visual model pre-training m ; The frequency-aware masking strategy includes a random mask generation process and a spectrum masking process; the random mask generation process is used to simulate the uncertainty of degraded information interference; The spectral masking process is used to simulate interference that degrades information; Step 3: pre-training of spectrum information reconstruction: modifying the structure of the underlying visual model that needs to be pre-trained for the first time, including: modifying the underlying visual model so that the modified underlying visual model is used as an encoder module, and adding a decoder module; Step 4: Fine-tuning of downstream degraded restoration tasks: retain the parameters of the pre-trained underlying visual model after the first modification, and modify the structure of the underlying visual model for the second time. The process of modifying the structure of the underlying visual model for the second time includes retaining the encoder module and replacing the decoder module with the adaptation module for training downstream tasks.

2. The low-level visual pre-training method without images according to claim 1, characterized in that: Step 1 specifically includes: Set Gaussian distribution I(X)~N(μ,δ 2 ), where μ = 0, δ = 1, and a two-dimensional matrix I is obtained by random sampling from a Gaussian distribution, and three-channel replication is performed: I=Sample(N(μ,δ 2 )); I0 = concat[I,I,I]; Among them, Sample(·) represents a random sampling operation; concat[·,·] represents a concatenation operation, and finally a pure Gaussian noise image I0 is obtained.

3. The low-level visual pre-training method without images according to claim 1, characterized in that: In step 2, the random mask generation process specifically includes: The amplitude spectrum of the input image is regarded as a combination of concentric squares with unit width. Concentric squares are randomly sampled in a set ratio and following a uniform distribution to obtain a random mask M of the same size as the input image to eliminate the information redundancy caused by the continuity of the frequency distribution: R = min(W / 2,H / 2); N = R * r; M = FAM (R, N); Where W and H are the length and width of the random mask, which are the same as the size of the input image; R is the maximum sampling radius of the mask box; r is the sampling ratio; N is the number of mask boxes required in the mask generation process; FAM(·) represents the process of uniformly randomly sampling concentric square boxes.

4. The low-level visual pre-training method without images according to claim 1, characterized in that: The spectrum masking process specifically includes: mapping the pure Gaussian noise image I0 from the spatial domain to the frequency domain through Fourier transform, and obtaining the corresponding amplitude spectrum A o and phase spectrum P o , and then use the random mask M obtained by the random mask generation process to the amplitude spectrum A o Perform masking operation to obtain the amplitude spectrum with destroyed information Finally, the complete phase spectrum P is obtained by inverse Fourier transform. o Amplitude spectrum with information destroyed Mapping back to the spatial domain, we get the input image I of the underlying visual model pre-training m : A o ,P o =FFT(I0); Among them, FFT(·) is the Fourier transform, which maps the image from the spatial domain to the frequency domain; IFFT(·,·) is the inverse Fourier transform, which maps the image from the frequency domain back to the spatial domain.

5. The low-level visual pre-training method without images according to claim 1, characterized in that: The spectrum information reconstruction pre-training specifically includes: Modify the structure of the original underlying visual model: After the final output dimension of the original underlying visual model is enlarged, it is used as an encoder module to output intermediate features; if the original underlying visual model has a global residual setting, remove the global residual setting to align the modified output dimension; add a decoder module for image reconstruction in the pre-training stage: I r =ResBlock(E(I m )); Among them, ResBlock(·) is the decoder module, E is the underlying visual model with modified output channels, and I r To reconstruct the image; The input is reconstructed by predicting the missing values ​​on the amplitude spectrum. The loss function Including pixel-level reconstruction loss and amplitude spectrum reconstruction loss: In the formula, λ is the balance weight; represents the inverse of the random mask M, ⊙ is the Hadamard product between matrices, A o and A r I o and I r The corresponding amplitude spectrum; the first term is the pixel-level reconstruction loss, and the second term is the amplitude spectrum reconstruction loss.

6. The low-level visual pre-training method without images according to claim 1, characterized in that: The downstream degradation recovery task fine-tuning specifically includes: In the downstream task fine-tuning phase, the encoder is initialized with the parameters of the pre-trained first modified underlying visual model, and the decoder module is replaced by an adaptation module consisting of convolutional layers: I out =Conv(E(I in )); Among them, I in is the image to be restored containing degraded information, E is the underlying visual model after the first modification after pre-training, and Conv represents the convolutional layer; It is then fine-tuned according to the specific training strategy of the underlying vision model.

Images