Camera original image blind super-resolution method based on metadata guidance

By introducing metadata guidance into the image blind super-segment method, the MetaSRNet network is designed, and the cross-channel attention module and self-attention module are used to solve the problem of complex and variable image recovery effects in the existing technology, and the image repair is improved Performance and robustness simplifies the model and training process.

CN119006285APending Publication Date: 2024-11-22UNIV OF ELECTRONICS SCI & TECH OF CHINA +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411097322.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The existing blind super-scoring method is not effective in the face of complex, changeable and unknown real-world degraded images. The model parameters are large, the training data is large, the training time is long, and the camera metadata is not used, resulting in the testing. The data generalization effect is poor and the recovery process is uncontrollable and interpretable.

Method used

Using the camera's original image blind super-segment method based on metadata guidance, by collecting low-resolution and high-resolution image pairs in various scenarios and recording camera metadata, designing the metadata guide blind super-segment network MetaSRNet, and using metadata Information guides image repair, and improves image recovery quality through cross-channel attention modules and self-attention modules.

Benefits of technology

Effectively improve the performance of image repair tasks, improve the robustness of complex and unknown degradation, improve image recovery quality, reduce model parameters and training time, and enhance generalization ability and recovery process on unseen data controllability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006285A_ABST
    Figure CN119006285A_ABST
Patent Text Reader

Abstract

The invention discloses a metadata guidance-based camera original image blind super-division method, which comprises the following steps of: acquiring a low-resolution image and a high-resolution image by using a camera in various scenes to form an image pair, and recording corresponding metadata when the low-resolution image is acquired at the same time; secondly, designing a blind super-resolution network capable of utilizing rich meta-information carried by an original image of the camera, and training the blind super-resolution network through collected data; and finally, the blind super-division network is guided to judge the form of image degradation and the severity of image degradation in a manner of introducing metadata information, so that the quality of an image output by the blind super-division network is improved, and the problem that blurring is introduced due to excessive restoration of the image by the network or degradation still exists due to insufficient restoration is effectively solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and more specifically, relates to a blind super-resolution method of camera original images based on metadata guidance. Background Art

[0002] Blind image super-resolution refers to the technology of restoring low-resolution images containing unknown degradation (such as noise, blur, and artifacts) to high-resolution, high-quality images. Blind image super-resolution does not rely on a clear degradation model, but directly infers and restores high-resolution details from low-resolution images through deep learning and artificial intelligence algorithms. This method shows higher robustness and flexibility in the face of complex and unknown degradation processes in the real world. Camera raw images refer to the image form captured by the camera without further processing by the ISP system. It contains rich metadata such as sensitivity (ISO), exposure time, aperture size, focal length, etc.

[0003] Since the degradation contained in the blind image super-resolution dataset is complex, variable, and unknown, it is difficult for neural networks to determine the type and severity of degradation contained in the image and correctly perform the corresponding image restoration. Real-ESRGAN uses a more complex degraded image synthesis method to construct training data. AirNet introduces a contrastive learning architecture to constrain the image restoration process by modifying the network architecture. OneRestore introduces language guidance and guides the model to restore different types of degradation through word embedding. However, faced with the more variable degradation in the real world, these methods still perform poorly in actual scenarios.

[0004] Existing image blind super-resolution methods have the following disadvantages: 1. The restoration effect of degraded images that are more complex, changeable, and unknown in the real world is poor. 2. The model parameters are large, requiring more training data and a long training time. 3. The camera-related metadata containing rich information is not used, the generalization effect of test data outside the training data distribution is poor, and the restoration process is not controllable and explainable. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a metadata-guided camera original image blind super-resolution method, which uses the rich metadata in the camera original image to guide the blind super-resolution network for image restoration. The method can effectively improve the performance in image restoration tasks and is robust to various complex and unknown degradations.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a method for blind super-resolution of camera raw images based on metadata guidance, characterized in that it comprises the following steps:

[0007] (1) Image acquisition and preprocessing;

[0008] (1.1) Set the camera's image acquisition format to RAW format, and arrange the acquired images in an RGGB array;

[0009] (1.2) Use the camera to collect image pairs in various scenes, where a low-resolution image I is collected in each scene. low and a high-resolution image I high Form an image pair and record the camera's acquisition I low Metadata M low ,Each set of metadata consists of sensitivity, exposure time, aperture size and focal length;

[0010] (1.3) The set of low-resolution images collected in various scenes is Set(I low ), the corresponding metadata constitutes a set Set(M low ); Set (I high );

[0011] (1.4), respectively for the set Set(I low ) and Set(I high ) is first normalized, and then channel converted, so that each image is converted from the original one channel to four channels;

[0012] (2) Build and train the metadata-guided blind super-resolution network MetaSRNet;

[0013] (2.1), from the set Set(I low ) and Set(I high ) to select the image pair I low with I high , from the set Set(M low ) low The corresponding metadata M low ;

[0014] (2.2) The image is aligned to I low with I high And the corresponding metadata M low Input to the MetaSRNet network, and then the image I is upsampled by the upsampling module low Perform two times upsampling to get the image Then the image Input to the convolution layer with a convolution kernel size of 3*3, a step size of 1, and a number of channels of 32 and 128 respectively, and obtain the feature map I after the convolution operation feature ;

[0015] (2.3) The feature map is passed through the cross-channel attention module CCAB with embedding dimensions of 128, 256, 512, 512, 256, and 128 in sequence;

[0016] Among them, the input of the first CCAB module with an embedding dimension of 128 is the feature map I feature With metadata M low ;

[0017] In the first CCAB module, the metadata M is first embedded in the low Convert it into vector V, and then according to the feature map I feature The size of the vector V is adjusted, and the output is consistent with the feature map I feature Eigenvectors of the same size Next, the feature map I feature With the eigenvector Input to the multi-head cross attention module, and obtain a new feature map through cross attention operation Then the new feature map Input to the multi-head self-attention module for self-attention operation to obtain a feature map with channel attention Finally, the feature map Input the feedforward layer for feedforward operation to obtain the feature map I1 of the final output of the first CCAB module feature ;

[0018] In the subsequent CCAB modules, the second, third, and fourth CCAB modules all take the output of the previous CCAB module as input, the fifth CCAB module takes the output of the second CCAB module plus the output of the fourth CCAB module as input, and the sixth CCAB module takes the output of the first CCAB module plus the output of the fifth CCAB module as input. Finally, the feature map I6 is output at the output end of the sixth CCAB module. feature ;

[0019] (2.4) The feature map I6 output by the sixth CCAB module feature The convolution kernel size is 3*3, the step size is 1, and the output channel dimensions are 128, 64, and 4 respectively. After the three-layer convolution operation, the upsampled image is Add together to get the restored image of the metadata-guided blind super-resolution network

[0020] (2.5) Calculate the total loss Loss after this round of training;

[0021] Loss=Loss ssim +Loss mse

[0022]

[0023] Among them, Loss mse is the mean square error loss, Loss ssim is the structural consistency loss; MSE() means calculating the mean square error between each pixel of the two images; SSIM() means calculating the similarity index between the two images in terms of brightness, contrast and structure, which is the weighted product of the average brightness difference, contrast difference and structural similarity of the images;

[0024] (2.6) Repeat steps (2.1) to (2.5) to iteratively train the metadata-guided blind super-resolution network until the metadata-guided blind super-resolution network converges;

[0025] (3) Real-time acquisition of a low-resolution image And record the corresponding metadata Then the image Metadata Input the trained metadata to guide the blind super-resolution network, thereby outputting the restored high-resolution image

[0026] The object of the invention of the present invention is achieved in this way:

[0027] The present invention is based on a metadata-guided camera raw image blind super-resolution method. First, a camera is used to collect low-resolution images and high-resolution images in various scenarios to form image pairs, and the metadata corresponding to the low-resolution images are recorded at the same time. Then, a blind super-resolution network that can utilize the rich metadata carried by the camera raw images is designed, and the blind super-resolution network is trained by collecting data. Finally, the blind super-resolution network is guided to judge the form of image degradation and the severity of image degradation by introducing metadata information, thereby improving the quality of the image output by the blind super-resolution network, which effectively solves the problem of blurring caused by over-repairing of the network or degradation caused by under-repairing. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of the blind super-resolution method of camera raw images based on metadata guidance of the present invention;

[0029] Figure 2 This is the metadata-guided blind super-resolution network structure diagram. DETAILED DESCRIPTION

[0030] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0031] Example

[0032] Figure 1 It is a flow chart of the blind super-resolution method of camera raw image based on metadata guidance of the present invention.

[0033] In this embodiment, if Figure 1 As shown, the present invention provides a method for blind super-resolution of camera raw images based on metadata guidance, comprising the following steps:

[0034] S1, image acquisition and preprocessing;

[0035] S1.1. Set the image acquisition format of the camera to RAW format, and the acquired pictures are arranged in RGGB array form (i.e., the image is one channel, the upper left corner of a 2*2 small block is the red pixel value obtained by the camera, the upper right corner and the lower left corner are green pixel values, and the lower right corner is the blue pixel value);

[0036] S1.2. Use the camera to collect image pairs in various scenes (including indoor, outdoor, cloudy, sunny, near and far scenes, etc.), where a low-resolution image I is collected in each scene. low and a high-resolution image I high Form an image pair and record the camera's acquisition I low Metadata M low ,Each set of metadata consists of sensitivity, exposure time, aperture size and focal length;

[0037] S1.3, the set of low-resolution images collected in various scenes is Set(I low ), the corresponding metadata constitutes a set Set(M low ); Set (I high );

[0038] S1.4, respectively, for the set Set(I low ) and Set(I high ) is first normalized, and then channel converted, so that each image is converted from the original one channel to four channels;

[0039] S2. Build and train the metadata-guided blind super-resolution network MetaSRNet;

[0040] S2.1. From the set Set(I low ) and Set(I high ) to select the image pair I low ∈R H×W×4 with I high ∈R H×W×4 , from the set Set(Mlow ) to select the corresponding metadata M low ;

[0041] S2.2, image alignment low with I high And the corresponding metadata M low Input to the MetaSRNet network, such as Figure 2 As shown, the image I is then upsampled by the upsampling module low Perform two times upsampling to get the image Then the image Input to the convolution layer with a convolution kernel size of 3*3, a step size of 1, and a number of channels of 32 and 128 respectively, and obtain the feature map I after the convolution operation feature ∈R 2H×2W×128 ;

[0042] S2.3, pass the feature map through the CCAB module with embedding dimensions of 128, 256, 512, 512, 256, and 128 in sequence;

[0043] Among them, the input of the first CCAB module with an embedding dimension of 128 is the feature map I feature ∈R 2H×2W×128 With metadata M low ;

[0044] In the first CCAB module, the metadata M is first embedded in the low Convert to vector V∈R 1×128 , and then according to the feature map I feature ∈R 2H×2W×128 The size of the vector V is adjusted, and the output is consistent with the feature map I feature Eigenvectors of the same size

[0045] Next, the feature map I feature With the eigenvector Input to the multi-head cross attention module for cross attention operation: In the multi-head cross attention module, first the feature map I feature With the eigenvector Map and reshape through different fully connected layers to form a new feature vector Q∈R 4HW×128 and Then calculate the attention weight Get the channel attention weight W∈R 128×128 , and finally perform channel attention operation Get new feature map

[0046] Then, the new feature map Input to the multi-head self-attention module for self-attention operation: In the multi-head self-attention module, first convert the new feature map Mapping is performed through three different fully connected layers to form three new feature vectors Then calculate the attention weights for the three new feature vectors Get new channel attention weights Finally, perform channel attention operation Get feature map with channel attention

[0047] Finally, the feature map Input feedforward layer for feedforward operation: In the feedforward layer, the feature map First, it is mapped to 4 times the embedding dimension by the fully connected layer 1 Then the fully connected layer 2 maps back to twice the embedding dimension And arrange the shape into Then downsample to obtain the feature map I1 of the final output of the first CCAB module feature ∈R H×W×256 ;

[0048] In the subsequent CCAB modules, the second, third, and fourth CCAB modules all take the output of the previous CCAB module as input, the fifth CCAB module takes the output of the second CCAB module plus the output of the fourth CCAB module as input, and the sixth CCAB module takes the output of the first CCAB module plus the output of the fifth CCAB module as input. Finally, the feature map I6 is output at the output end of the sixth CCAB module. feature ∈R H×W×128 ;

[0049] S2.4, the feature map I6 output by the sixth CCAB module feature The convolution kernel size is 3*3, the step size is 1, and the output channel dimensions are 128, 64, and 4 respectively. After the three-layer convolution operation, the upsampled image is Add together to get the restored image of the metadata-guided blind super-resolution network

[0050]

[0051] S2.5. Calculate the total loss Loss after this round of training.

[0052] Loss=Loss ssim +Loss mse

[0053]

[0054] Among them, Lossmse is the mean square error loss, Loss ssim is the structural consistency loss; MSE() means calculating the mean square error between each pixel of the two images; it means calculating the similarity index between the two images in brightness, contrast and structure, which is the weighted product of the average brightness difference, contrast difference and structural similarity of the images.

[0055] S2.6, repeating steps S2.1 to S2.5, training the metadata-guided blind super-resolution network in an iterative manner until the metadata-guided blind super-resolution network converges;

[0056] S3, real-time acquisition of a low-resolution image And record the corresponding metadata Then the image Metadata Input the trained metadata to guide the blind super-resolution network, thereby outputting the restored high-resolution image

[0057] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A metadata-guided camera raw image blind super-resolution method, characterized in that: The following steps are involved: (1) Training image acquisition and preprocessing; (1.1) Set the camera's image acquisition format to RAW format, and arrange the acquired images in an RGGB array; (1.2) Use the camera to collect image pairs in various scenes, where a low-resolution image I is collected in each scene. low and a high-resolution image I high Form an image pair and record the camera's acquisition I low Metadata M low ,Each set of metadata consists of sensitivity, exposure time, aperture size and focal length; (1.3) The set of low-resolution images collected in various scenes is Set(I low ), the corresponding metadata constitutes a set Set(M low ); Set (I high ); (1.4), respectively for the set Set(I low ) and Set(I high ) is first normalized, and then channel converted, so that each image is converted from the original one channel to four channels; (2) Build and train the metadata-guided blind super-resolution network MetaSRNet; (2.1), from the set Set(I low ) and Set(I high ) to select the image pair I low with I high , from the set Set(M low ) low The corresponding metadata M low ; (2.2) The image is aligned to I low with I high And the corresponding metadata M low Input to the MetaSRNet network, and then the image I is upsampled by the upsampling module low Perform two times upsampling to get the image Then the image Input to the convolution layer with a convolution kernel size of 3*3, a step size of 1, and a number of channels of 32 and 128 respectively, and obtain the feature map I after the convolution operation feature ; (2.3) The feature map is passed through the CCAB module with embedding dimensions of 128, 256, 512, 512, 256, and 128 in sequence; Among them, the input of the first CCAB module with an embedding dimension of 128 is the feature map I feature With metadata M low ; In the first CCAB module, the metadata M is first embedded in the low Convert it into vector V, and then according to the feature map I feature The size of the vector V is adjusted, and the output is consistent with the feature map I feature Eigenvectors of the same size Next, the feature map I feature With the eigenvector Input to the multi-head cross attention module, and obtain a new feature map through cross attention operation Then the new feature map Input to the multi-head self-attention module for self-attention operation to obtain a feature map with channel attention Finally, the feature map Input the feedforward layer for feedforward operation to obtain the feature map I1 of the final output of the first CCAB module feature ; In the subsequent CCAB modules, the second, third, and fourth CCAB modules all take the output of the previous CCAB module as input, the fifth CCAB module takes the output of the second CCAB module plus the output of the fourth CCAB module as input, and the sixth CCAB module takes the output of the first CCAB module plus the output of the fifth CCAB module as input. Finally, the feature map I6 is output at the output end of the sixth CCAB module. feature ; (2.4) The feature map I6 output by the sixth CCAB module feature The convolution kernel size is 3*3, the step size is 1, and the output channel dimensions are 128, 64, and 4 respectively. After the three-layer convolution operation, the upsampled image is Add together to get the restored image of the metadata-guided blind super-resolution network (2.5) Calculate the total loss Loss after this round of training; Loss=Loss ssim +Loss mse Among them, Loss mse is the mean square error loss, Loss ssim is the structural consistency loss; MSE() means calculating the mean square error between each pixel of the two images; SSIM() means calculating the similarity index between the two images in terms of brightness, contrast and structure; (2.6) Repeat steps (2.1) to (2.5) to iteratively train the metadata-guided blind super-resolution network until the metadata-guided blind super-resolution network converges; (3) Real-time acquisition of a low-resolution image And record the corresponding metadata Then the image Metadata Input the trained metadata to guide the blind super-resolution network, thereby outputting the restored high-resolution image 2. The metadata-guided camera raw image blind super-resolution method according to claim 1, characterized in that: The process of cross attention operation in the multi-head cross attention module is: First, the feature map I feature With the eigenvector Through different fully connected layers, the new feature vector Q and Then the attention weight is calculated to obtain the channel attention weight W, and finally the channel attention operation is performed Get new feature map 3. The metadata-guided camera raw image blind super-resolution method according to claim 1, characterized in that: The process of the multi-head self-attention module performing self-attention operation is: The new feature map Mapping is performed through three different fully connected layers to form three new feature vectors Then calculate the attention weights of the three new feature vectors to obtain new channel attention weights Finally, perform channel attention operation Get feature map with channel attention