Raw domain multi-frame image video super-resolution method, system and device and storage medium

Abstract

The invention discloses a Raw domain multi-frame image video super-resolution method, system and device and a storage medium, for a Raw domain sequence continuously shot by an imaging device at a high speed, multi-frame features in a Raw domain image sequence are extracted by using a window spatial domain attention-based Transformer architecture, and the multi-frame features are aligned to a unified position by using multi-scale optical flow-based deformable convolution, so that the super-resolution of the Raw domain multi-frame image video is realized. Multiple aligned frames are fused by using an inter-channel attention-based Transform architecture, super-resolution enhancement is performed on features obtained by fusion based on a convolutional neural network architecture, original information on a Raw domain is fully utilized, the imaging quality is superior to that of an RGB domain, the noise intensity in an image can be greatly reduced, and the definition and sharpness can be remarkably improved. And natural and rich details are recovered.

Description

Technical Field

[0001] The present invention relates to the technical field of image and video super-resolution, and in particular to a Raw domain multi-frame image and video super-resolution method, system, device and medium. Background Art

[0002] Super-resolution technology (SR) is an important direction in the field of computer vision and image processing, which aims to generate high-resolution images or videos from low-resolution images or videos through algorithms. Super-resolution technology has a wide range of applications in many scenarios, such as medical imaging, satellite imaging, security monitoring, video enhancement, etc.

[0003] Raw images are uncompressed and unprocessed image data directly output from digital cameras, mobile phone cameras or image sensors. Unlike common JPEG or PNG format images, Raw images retain the original data captured by the sensor, so they are usually richer in information and more complete in details than compressed images.

[0004] Burst Photography is a photography technique that captures instantaneous action or scenes by taking multiple photos in rapid succession, such as Figure 1 The figure shows a schematic diagram of continuously taking multiple photos.

[0005] There are currently two main types of super-resolution solutions.

[0006] Solution 1: Use bilinear interpolation and bicubic interpolation to estimate unknown pixel values ​​and interpolate directly on the image grid. However, this solution usually leads to blurred images and cannot effectively restore details.

[0007] Solution 2: Single Image Super-Resolution (SISR) based on deep learning. This solution recovers a high-resolution version of a single low-resolution image. SISR is a fundamental problem in the study of super-resolution technology, and it solves the challenge of how to infer high-resolution details from a single low-resolution image. However, this solution is difficult to restore good details. The authenticity of the details restored based on the generation method is often not guaranteed, and often generates artifacts that are not intuitive. Summary of the invention

[0008] The purpose of the present invention is to provide a Raw domain multi-frame image video super-resolution method, system, device and storage medium, which can greatly improve the image clarity and sharpness and restore real and rich details.

[0009] The objective of the present invention is achieved through the following technical solutions:

[0010] (corresponding to the claims, omitted for now)

[0011] It can be seen from the technical solution provided by the present invention that for the Raw domain sequence of high-speed continuous shooting of the imaging device, the Transformer architecture based on window spatial attention is used to extract multi-frame features in the Raw domain image sequence, and the multi-frame features are aligned to the same position using multi-scale deformable convolution based on optical flow. The aligned multi-frames are fused using the Transformer architecture based on inter-channel attention, and the fused features are super-resolution enhanced based on the convolutional neural network architecture. The original information in the Raw domain is fully utilized, and the imaging quality is better than that in the RGB domain. The noise intensity in the image can be greatly reduced, the clarity and sharpness can be significantly improved, and the natural and rich details can be restored. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0013] Figure 1 A schematic diagram of continuously taking multiple photos provided as the background technology of the present invention;

[0014] Figure 2 A flowchart of a Raw domain multi-frame image video super-resolution method provided by an embodiment of the present invention;

[0015] Figure 3 A schematic diagram of a Transformer architecture based on window spatial attention provided in an embodiment of the present invention;

[0016] Figure 4 A schematic diagram of the architecture of a deformable convolution provided by an embodiment of the present invention;

[0017] Figure 5 A schematic diagram of a Transformer architecture based on inter-channel attention provided by an embodiment of the present invention;

[0018] Figure 6 A flowchart of generating a simulated Raw domain image through inverse operation provided by an embodiment of the present invention;

[0019] Figure 7 A schematic diagram of a Raw domain multi-frame image video super-resolution system provided by an embodiment of the present invention;

[0020] Figure 8A schematic diagram of a processing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.

[0022] First, the terms that may be used in this article are explained as follows:

[0023] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y means both “X” or “Y” and “X and Y”.

[0024] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.

[0025] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.

[0026] The following is a detailed description of a Raw domain multi-frame image video super-resolution method, system, device and storage medium provided by the present invention. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professional and technical personnel in the field. If the specific conditions are not indicated in the embodiments of the present invention, they are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. The reagents or instruments used in the embodiments of the present invention that do not indicate the manufacturer are all conventional products that can be purchased commercially.

[0027] Embodiment 1

[0028] The embodiment of the present invention provides a Raw domain multi-frame image video super-resolution method, such as Figure 2As shown, it mainly includes the following steps:

[0029] Step 1: Obtain a Raw domain image sequence.

[0030] Step 2: Obtain a super-resolution image by extracting multi-frame features and performing alignment, fusion and super-resolution enhancement.

[0031] In an embodiment of the present invention, a Transformer architecture based on window spatial attention is used to extract multi-frame features in a Raw domain image sequence, a multi-scale optical flow-based deformable convolution is used to align the multi-frame features to a unified position, a Transformer architecture based on inter-channel attention is used to fused the aligned multi-frames, and a convolutional neural network architecture is used to perform super-resolution enhancement on the fused features to obtain a super-resolution image.

[0032] In the embodiment of the present invention, the specific number of multiple frames can be set as needed, for example, 3 frames can be selected. Of course, the user can also make adjustments according to actual conditions or needs, and the present invention does not limit this.

[0033] The above-mentioned scheme provided by the embodiment of the present invention can improve the image quality by extracting multi-frame features, aligning and fusion processing, and obtaining fused multi-frame information for the Raw domain sequence of high-speed continuous shooting of the imaging device, and then performing super-resolution enhancement processing on it. Compared with the original image, the noise intensity in the image is greatly reduced and the clarity and sharpness are significantly improved, thereby restoring natural and rich details.

[0034] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail with reference to specific embodiments below.

[0035] 1. Overall introduction of the plan.

[0036] 1. Multi-frame feature extraction.

[0037] In the embodiment of the present invention, a Transformer architecture based on window spatial attention is used to extract multi-frame features in a Raw domain image sequence. Figure 3 As shown in the figure, it is a schematic diagram of the Transformer architecture based on window spatial attention. H and W are the height and width of the image, 3 is the number of channels, and C is the symbol of the number of channels. The various values ​​involved are also examples of related parameters. In actual applications, users can adjust the relevant parameter values ​​according to actual conditions or experience. Figure 3Middle: Part (a) shows the principle of Swin Transformer. SWIN stands for Shifted Window. Swin Transformer is a model architecture applied to visual tasks, which originated from the Transformer architecture in NLP tasks. Swin Transformer mainly uses the Window Attention mechanism, that is, attention is only calculated within the window. Compared with the original Transformer, the computational complexity changes from quadratic to linear, which is more suitable for visual tasks; Part (a) shows the division of Window in each layer of the network. It can be seen that as the layer is increased, the receptive field of each Window gradually increases. Part (b) shows the principle of Shifted Window. The large box represents Window, and the small box represents image patch. Each patch is represented by a token. Part (c) shows the structure of two Swin Transformer Blocks connected in series. Norm stands for layer normalization, MLP stands for multi-layer perceptron, W-MAS stands for Window Multi-head Attention Strategy, and SW-MAS stands for Shifted Window Multi-head Attention Strategy. Part (d) shows the structure of the Transformer architecture based on window spatial attention. The Transformer architecture based on window spatial attention mainly includes: a block partition unit (Patch Partition) and multiple feature extraction units, which are connected in sequence, wherein the first feature extraction unit includes a linear embedding layer (Linear Embedding) and several Swin Transformer Blocks connected in sequence to its back end, and the remaining feature extraction units include a block merging layer (Patch Merging) and several Swin TransformerBlocks connected in sequence to its back end; the block partition unit is responsible for dividing the input single-frame Raw domain image into blocks, and processing them in sequence through each feature extraction unit, and the last feature extraction unit outputs the features of the single-frame Raw domain image.

[0038] It should be noted that Figure 3 The structure shown is only an example. In actual applications, the user can adjust the number of feature extraction units and the number of Swin Transformer Blocks according to actual conditions.

[0039] 2. Feature alignment.

[0040] In an embodiment of the present invention, an AI optical flow network (e.g., SpyNet) is used to estimate multi-scale optical flows of different resolutions from each frame to a target frame, wherein the target frame is a frame selected from the multiple frames (e.g., the first frame or the middle frame), and each frame is the other frames in the multiple frames excluding the target frame; the estimated multi-scale optical flows are respectively input into the corresponding scale deformable convolution as reference values, and each layer of deformable convolution refers to the aligned features of the previous layer, and finally the features of multiple frames are aligned to a unified position.

[0041] Those skilled in the art can understand that multi-scale means downsampling the input frame multiple times to obtain a set of resolutions, for example {multiple frames of original resolution, multiple frames of 1 / 4 resolution, multiple frames of 1 / 16 resolution}, and then performing optical flow estimation at each resolution level from bottom to top, and inputting the estimation result into the deformable convolution as a reference to obtain the final output of the level, and upsampling the output to the previous level as a reference. The idea behind this is a divide and conquer, from coarse to fine approach.

[0042] like Figure 4 , which is a schematic diagram of the architecture of deformable convolution.

[0043] 3. Feature fusion.

[0044] In the embodiment of the present invention, a Transformer architecture based on inter-channel attention is used to fuse the aligned multiple frames.

[0045] The Transformer architecture based on inter-channel attention includes: a first deep convolution layer and a second deep convolution layer, N Transformer Blocks for downsampling, N-1 Transformer Blocks for upsampling, a channel attention and gating network, and M (for example, M=2) Transformer Blocks connected to the back end of the Nth Transformer Block for upsampling; the input features are processed by the N Transformer Blocks for downsampling in sequence after passing through the first deep convolution layer, the output of the Nth Transformer Block for downsampling is processed by the first Transformer Block for upsampling, and then connected in series with the output of the Nth Transformer Block for downsampling, and then enters the second Transformer Block for upsampling for processing, and each Transformer Block processing result for upsampling is connected in series with the output of the corresponding Transformer Block for downsampling and enters the next Transformer Block for upsampling for processing; the output of the N-1th Transformer Block for upsampling is connected in series with the output of the second Transformer Block for downsampling. The outputs of the blocks are connected in series, processed by sampling on the channel attention and gating network, and connected in series with the output of the first Transformer Block for downsampling. After being processed by M sequentially connected Transformer Blocks and the second deep convolutional layer, they are added to the input features to obtain fused features.

[0046] In the embodiment of the present invention, each Transformer Block in the Transformer architecture based on inter-channel attention is stacked with several Transformer Blocks (transformation neural network modules). Figure 5 L1, L2, L3, L4, L after each Transformer Block r It indicates the number of stacks. The F marked at the arrow is the output feature of the previous module. The subscript of F is a marking symbol used to distinguish the outputs of different modules.

[0047] In an embodiment of the present invention, the gated attention network includes: a multi-headed deep separable convolutional transposed attention network (MDTA) and a deep separable convolutional feedforward gated network (GDFN) connected in sequence; wherein:

[0048] like Figure 5As shown in part (a), the multi-head deep separable convolutional transposed attention network includes: a layer normalization layer (Norm), four convolutional layers and three deep convolutional layers, and a transposed attention module; the input information enters three branches after the layer normalization layer, each branch includes a convolutional layer and a deep convolutional layer set in sequence, and the three branches correspond to the QKV information of the output attention mechanism. The QKV information is processed by the transposed attention module for attention operation, and then processed by a convolution layer and added to the input information to obtain the output information of the multi-variable convolutional head transposed attention module;

[0049] like Figure 5 As shown in part (b), the depth-separable convolutional forward propagation gating network includes: a layer normalization layer, three convolutional layers and two deep convolutional layers, and an activation function layer; the output information of the multi-variable convolutional head transposed attention module enters two branches after the layer normalization layer, the first branch is a convolutional layer and a deep convolutional layer arranged in sequence, and the second branch is a convolutional layer, a deep convolutional layer and an activation function layer arranged in sequence; the outputs of the two branches are multiplied element by element and then processed by the convolution layer and added to the input information to obtain the output information of the gated variable convolutional feedforward network.

[0050] like Figure 5 As shown, a structural example of a Transformer architecture based on inter-channel attention is provided. In this example, N=3. In addition, the various numerical values ​​involved in this example are also examples of related parameters. In actual applications, users can adjust the relevant parameter values ​​according to actual conditions or experience.

[0051] 4. Super-resolution enhancement.

[0052] Super-resolution enhancement can be implemented based on a convolutional neural network architecture. The relevant network architecture can refer to conventional technologies and will not be described in detail in the present invention.

[0053] 2. Preliminary preparation work.

[0054] The model architectures corresponding to the four processes involved in the first part above together constitute a super-resolution model. The present invention pre-constructs a training set to train the super-resolution model.

[0055] Specifically, we collect RGB image datasets and generate simulated Raw domain Burst (continuous shooting) multi-frames as training sets through inverse operations, such as Figure 6As shown in the figure, the processing process is as follows: for each RGB image, invert global tone mapping (Invert Global Tone-Mapping) processing, invert gamma compression (Invert Gamma Compression) processing, invert color correction (Invert Color Correction) processing, invert gain (Invert Gain) processing, blur degradation (Blur Degradation) processing, simulate low-resolution continuous shooting (Simulate LR Bursts) processing, mosaic (Mosaic) processing and noise addition (Add Noise) processing are performed in turn to generate the corresponding Raw domain image (RAW LR Bursts); finally, the generated Raw domain images constitute the training set.

[0056] During the training process, the training set is input into the super-resolution model to obtain the corresponding super-resolution image, and the loss (e.g., L1 loss) of the super-resolution image and the corresponding RGB image is calculated as the training loss of the model, and the model parameters are optimized accordingly. The training details involved can refer to conventional techniques and will not be elaborated in the present invention.

[0057] The above solution provided by the embodiment of the present invention makes full use of the original information in the Raw domain, has better imaging quality than that in the RGB domain, can restore real and rich details, and significantly improves clarity and sharpness.

[0058] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0059] Embodiment 2

[0060] The present invention also provides a Raw domain multi-frame image video super-resolution system, which is mainly used to implement the method provided in the above embodiment, such as Figure 7 As shown, the system mainly includes:

[0061] A data acquisition unit, used for acquiring a Raw domain image sequence;

[0062] The super-resolution model is used to extract multi-frame features in the Raw domain image sequence using the Transformer architecture based on window spatial attention, align the multi-frame features to the same position using multi-scale optical flow-based deformable convolution, fuse the aligned multi-frames using the Transformer architecture based on inter-channel attention, and perform super-resolution enhancement on the fused features based on the convolutional neural network architecture to obtain a super-resolution image.

[0063] Technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0064] Embodiment 3

[0065] The present invention also provides a processing device, such as Figure 8 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the methods provided in the aforementioned embodiments.

[0066] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0067] In the embodiment of the present invention, the specific types of the memory, input device and output device are not limited; for example:

[0068] The input device may be a touch screen, an image acquisition device, a physical button or a mouse, etc.;

[0069] The output device may be a display terminal;

[0070] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0071] Embodiment 4

[0072] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0073] In the embodiment of the present invention, the readable storage medium is a computer-readable storage medium and can be set in the aforementioned processing device, for example, as a memory in the processing device. In addition, the readable storage medium can also be a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk, etc., which can store program codes.

[0074] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A Raw domain multi-frame image video super-resolution method, characterized in that: include: Step 1: Obtain a Raw domain image sequence; Step 2: Use the Transformer architecture based on window spatial attention to extract multi-frame features in the Raw domain image sequence, use multi-scale optical flow-based deformable convolution to align the multi-frame features to the same position, use the Transformer architecture based on inter-channel attention to fuse the aligned multi-frames, and perform super-resolution enhancement on the fused features based on the convolutional neural network architecture to obtain a super-resolution image.

2. The Raw domain multi-frame image video super-resolution method according to claim 1, characterized in that: The Transformer architecture based on window spatial attention includes: a block division unit, and a plurality of feature extraction units, wherein the plurality of feature extraction units are connected in sequence, wherein the first feature extraction unit includes a linear embedding layer and a plurality of Swin Transformer Blocks connected in sequence to its rear end, and the remaining feature extraction units include a block merging layer and a plurality of Swin Transformer Blocks connected in sequence to its rear end; The block division unit is responsible for dividing the input single-frame Raw domain image into blocks, and processing them in turn through each feature extraction unit, and the last feature extraction unit outputs the features of the single-frame Raw domain image.

3. The Raw domain multi-frame image video super-resolution method according to claim 1, characterized in that: The method of using multi-scale optical flow-based deformable convolution to align multi-frame features to a unified position includes: Use AI optical flow network to estimate multi-scale optical flow of different resolutions from each frame to the target frame; the target frame is a frame selected from multiple frames, and each frame is the other frames in the multiple frames excluding the target frame; The estimated multi-scale optical flows are input into the corresponding scale deformable convolution as reference values. Each layer of deformable convolution refers to the aligned features of the previous layer, and finally the multi-frame features are aligned to the same position.

4. The method for super-resolution of multiple frames of images in Raw domain according to claim 1, characterized in that: The Transformer architecture based on inter-channel attention includes: The first and second deep convolutional layers, N Transformer Blocks for downsampling, N-1 Transformer Blocks for upsampling, channel attention and gating networks, and M Transformer Blocks connected to the back end of the Nth Transformer Block for upsampling; After the input features pass through the first deep convolutional layer, they are processed by N Transformer Blocks for downsampling in sequence. The output of the Nth Transformer Block for downsampling is processed by the first Transformer Block for upsampling, and then connected in series with the output of the Nth Transformer Block for downsampling, and then enters the second Transformer Block for upsampling. The processing result of each Transformer Block for upsampling is connected in series with the output of the corresponding Transformer Block for downsampling and then enters the next Transformer Block for upsampling; the output of the N-1th Transformer Block for upsampling is connected in series with the output of the second Transformer Block for downsampling, processed by the channel attention and gating network and connected in series with the output of the first Transformer Block for downsampling, and then processed by the sequentially connected M Transformer Blocks and the second deep convolutional layer, and then added to the input features to obtain fused features.

5. The Raw domain multi-frame image video super-resolution method according to claim 4, characterized in that: The gated attention network includes: a multi-head deep separable convolutional transposed attention network and a deep separable convolutional forward propagation gated network connected in sequence; wherein: The multi-head deep separable convolution transposed attention network includes: a layer normalization layer, four convolution layers and three deep convolution layers, and a transposed attention module; the input information enters three branches after the layer normalization layer, each branch includes a convolution layer and a deep convolution layer set in sequence, and the three branches correspond to the QKV information of the output attention mechanism. The QKV information is processed by the transposed attention module for attention operation, and then processed by a convolution layer and added to the input information to obtain the output information of the multi-variable convolution head transposed attention module; The depth-separable convolutional forward propagation gating network includes: a layer normalization layer, three convolutional layers and two deep convolutional layers, and an activation function layer; the output information of the multi-variable convolutional head transposed attention module enters two branches after the layer normalization layer, the first branch is a convolutional layer and a deep convolutional layer set in sequence, and the second branch is a convolutional layer, a deep convolutional layer and an activation function layer set in sequence; the outputs of the two branches are multiplied element by element and then processed by the convolution layer and added to the input information to obtain the output information of the gated variable convolutional feedforward network.

6. The Raw domain multi-frame image video super-resolution method according to claim 1, characterized in that: The method further comprises: the step 2 is implemented by a super-resolution model, and a training set is pre-constructed to train the super-resolution model; Among them, an RGB image dataset is collected, and multiple frames of simulated Raw domain Burst are generated as training sets through inverse operations.

7. The Raw domain multi-frame image video super-resolution method according to claim 1, characterized in that: Collect RGB image data sets and generate simulated Raw domain Burst multi-frames as training sets through inverse operations, including: For each RGB image, reverse global tone mapping processing, reverse gamma compression processing, reverse color correction processing, reverse gain processing, blur degradation processing, simulated low-resolution continuous shooting processing, mosaic processing and noise addition processing are performed in sequence to generate the corresponding Raw domain image; The generated Raw domain images constitute the training set.

8. A Raw domain multi-frame image video super-resolution system, characterized in that: include: A data acquisition unit, used for acquiring a Raw domain image sequence; The super-resolution model is used to extract multi-frame features in the Raw domain image sequence using the Transformer architecture based on window spatial attention, align the multi-frame features to the same position using multi-scale optical flow-based deformable convolution, fuse the aligned multi-frames using the Transformer architecture based on inter-channel attention, and perform super-resolution enhancement on the fused features based on the convolutional neural network architecture to obtain a super-resolution image.

9. A processing device, characterized in that: include: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images