Image restoration method, device, equipment and readable storage medium
Through feature extraction, alignment and fusion and balance processing of repaired images and reference images, the problem of poor image repair effect in the prior art is solved, and a higher quality image repair effect is achieved.
Patent Information
- Application Number
- CN202210494054.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-29
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2042-04-29
AI Technical Summary
In the prior art, when the image damage area is large or the expected content contains complex semantic information, it is difficult to effectively perform image repair, resulting in poor repair effect.
By inputting the image to be repaired and the reference image into the encoder, extracting and aligning the texture features and structural features, performing multi-scale filling repair, and using attention mechanism to perform feature equalization processing, and finally performing jump connection in the decoder and generating repair image.
The image repair effect is significantly improved, especially when the damaged area is large or the expected complex content is expected, the generated repair images are of higher quality, and the PSNR and SSIM values are significantly improved.
Smart Images

Figure CN114782282B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image restoration technology, and in particular to an image restoration method, device, equipment and readable storage medium. Background Art
[0002] The purpose of image restoration is to fill the missing parts of the image with reasonable content. It is mainly used to repair damaged areas or remove unwanted targets in photos. The specific idea is to find sufficient prior information (such as smoothness and self-similarity) as a reference for filling the holes.
[0003] In the related art, the restoration method based on deep learning has attracted great attention. In order to better assist in image restoration in complex situations, the method of structural prior is introduced to guide the restoration task. The structural prior effectively helps improve the quality of the complete image. However, when the damaged area of the image is large or the expected content contains complex semantic information, this method will still encounter many difficulties. For example, when the damaged area is large, it is often necessary to guess to synthesize the image of the missing hole, which leads to the phenomenon that the image at the repaired position and the surrounding image look abrupt and unreasonable, that is, there is a problem of poor restoration effect. Summary of the invention
[0004] The present application provides an image restoration method, apparatus, device and readable storage medium to solve the problem in the related art that it is impossible to effectively restore images with large damaged areas or with expected content containing complex semantic information.
[0005] In a first aspect, an image restoration method is provided, comprising the following steps:
[0006] Inputting the image to be repaired into a first encoder and the reference image corresponding to the image to be repaired into a second encoder to obtain a first feature map of the image to be repaired and a second feature map of the reference image, and performing feature extraction on the first feature map and the second feature map respectively to obtain a first texture feature and a first structural feature of the image to be repaired and a second texture feature and a second structural feature of the reference image;
[0007] Performing alignment and fusion processing on the first texture feature and the second texture feature as well as the first structure feature and the second structure feature respectively to obtain a fused texture feature and a fused structure feature;
[0008] Perform multi-scale filling and repair on the fused texture features and the fused structure features to obtain the filled texture features and the filled structure features;
[0009] The filling texture feature and the filling structure feature are connected and convolved to obtain a balanced feature;
[0010] The equalization feature is input into the decoder, so that the decoder jumps and connects the equalization feature with the first feature map input into the decoder to generate a restored image.
[0011] In some embodiments, the first feature map of the image to be repaired includes multiple first shallow feature maps and multiple first deep feature maps, and the second feature map of the reference image includes multiple second shallow feature maps and multiple second deep feature maps.
[0012] In some embodiments, the extracting features from the first feature map and the second feature map respectively to obtain the first texture feature and the first structural feature of the image to be repaired and the second texture feature and the second structural feature of the reference image includes:
[0013] Splicing the plurality of first shallow feature maps to obtain a first texture feature of the image to be repaired, and splicing the plurality of first deep feature maps to obtain a first structural feature of the image to be repaired;
[0014] The multiple second shallow feature maps are spliced to obtain the second texture features of the reference image, and the multiple second deep feature maps are spliced to obtain the second structural features of the reference image.
[0015] In some embodiments, the step of aligning and fusing the first texture feature and the second texture feature and the first structure feature and the second structure feature to obtain a fused texture feature and a fused structure feature includes:
[0016] Performing an alignment process on the first texture feature and the second texture feature to obtain an aligned texture feature, and performing a fusion process on the aligned texture feature and the first texture feature to obtain a fused texture feature;
[0017] The first structural feature and the second structural feature are aligned to obtain an aligned structural feature, and the aligned structural feature and the first structural feature are fused to obtain a fused structural feature.
[0018] In some embodiments, the aligning the first texture feature and the second texture feature to obtain the aligned texture feature includes:
[0019] Connecting the first texture feature and the second texture feature to obtain a connected texture feature;
[0020] The connected texture features are calculated and processed based on the offset estimator to obtain a texture feature offset map;
[0021] The second texture feature is subjected to deformable convolution processing based on the texture feature offset map to obtain an aligned texture feature.
[0022] In some embodiments, the connecting and convolution processing of the filling texture feature and the filling structure feature to obtain the balanced feature includes:
[0023] Based on the attention mechanism, the filling texture features and filling structure features are connected and convolved to obtain balanced features.
[0024] In some embodiments, the operating parameters of the first encoder and the second encoder are the same.
[0025] In a second aspect, an image restoration device is provided, comprising:
[0026] A feature extraction unit, which is used to input the image to be repaired into a first encoder and a reference image corresponding to the image to be repaired into a second encoder, obtain a first feature map of the image to be repaired and a second feature map of the reference image, and perform feature extraction on the first feature map and the second feature map respectively to obtain a first texture feature and a first structural feature of the image to be repaired and a second texture feature and a second structural feature of the reference image;
[0027] An alignment and fusion unit, which is used to perform alignment and fusion processing on the first texture feature and the second texture feature and the first structure feature and the second structure feature respectively, to obtain a fused texture feature and a fused structure feature;
[0028] A feature filling unit, which is used to perform multi-scale filling and repair on the fused texture features and the fused structure features to obtain filled texture features and filled structure features;
[0029] A feature equalization unit, which is used to connect and convolve the filling texture feature and the filling structure feature to obtain a balanced feature;
[0030] The image reconstruction unit is used to input the equalization feature into the decoder, so that the decoder jumps and connects the equalization feature with the first feature map input into the decoder to generate a restored image.
[0031] In a third aspect, an image restoration device is provided, comprising: a memory and a processor, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the aforementioned image restoration method.
[0032] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer storage medium stores a computer program, and when the computer program is executed by a processor, the aforementioned image restoration method is implemented.
[0033] The beneficial effects brought about by the technical solution provided by this application include: it can effectively improve the image restoration effect.
[0034] The present application provides an image restoration method, apparatus, device and readable storage medium, comprising: inputting an image to be restored into a first encoder and a reference image corresponding to the image to be restored into a second encoder, obtaining a first feature map of the image to be restored and a second feature map of the reference image, and performing feature extraction on the first feature map and the second feature map respectively, obtaining a first texture feature and a first structural feature of the image to be restored and a second texture feature and a second structural feature of the reference image; performing alignment and fusion processing on the first texture feature and the second texture feature and the first structural feature and the second structural feature respectively, obtaining a fused texture feature and a fused structural feature; performing multi-scale filling and restoration on the fused texture feature and the fused structural feature, obtaining a filling texture feature and a filling structural feature; performing connection and convolution processing on the filling texture feature and the filling structural feature, obtaining a balanced feature; and inputting the balanced feature into a decoder, so that the decoder jumps and connects the balanced feature with the first feature map input into the decoder to generate a restored image. This application introduces a reference image into the encoding and decoding network to make good use of external information to infer the structural and texture features of damaged images such as those with large damaged areas or expected content containing complex semantic information, thereby obtaining better restoration results; at the same time, it fully considers the potential differences between the image to be restored and the reference image, and effectively fuses the features between the two to improve the restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0036] Figure 1 A schematic diagram of a process of an image restoration method provided in an embodiment of the present application;
[0037] Figure 2 A schematic diagram of image restoration data processing provided in an embodiment of the present application;
[0038] Figure 3 A schematic diagram of the alignment and fusion data processing provided in an embodiment of the present application;
[0039] Figure 4 A schematic diagram of the structure of an image restoration device provided in an embodiment of the present application;
[0040] Figure 5 A schematic diagram of the structure of an image restoration device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0041] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.
[0042] The embodiments of the present application provide an image restoration method, apparatus, device and readable storage medium, which can solve the problem in the related art that it is impossible to effectively restore images with large damaged areas or with expected content containing complex semantic information.
[0043] See also Figure 1 and Figure 2 As shown, the embodiment of the present application provides an image restoration method, comprising the following steps:
[0044] Step S10: Input the image to be repaired into the first encoder and the reference image corresponding to the image to be repaired into the second encoder to obtain a first feature map of the image to be repaired and a second feature map of the reference image, and perform feature extraction on the first feature map and the second feature map respectively to obtain a first texture feature and a first structural feature of the image to be repaired and a second texture feature and a second structural feature of the reference image; wherein the operating parameters of the first encoder and the second encoder are the same.
[0045] Furthermore, the first feature map of the image to be repaired includes multiple first shallow feature maps and multiple first deep feature maps, and the second feature map of the reference image includes multiple second shallow feature maps and multiple second deep feature maps.
[0046] Furthermore, the extracting features from the first feature map and the second feature map respectively to obtain the first texture feature and the first structural feature of the image to be repaired and the second texture feature and the second structural feature of the reference image includes:
[0047] Splicing the plurality of first shallow feature maps to obtain a first texture feature of the image to be repaired, and splicing the plurality of first deep feature maps to obtain a first structural feature of the image to be repaired;
[0048] The multiple second shallow feature maps are spliced to obtain the second texture features of the reference image, and the multiple second deep feature maps are spliced to obtain the second structural features of the reference image.
[0049] Exemplarily, in the embodiment of the present application, the data set provided is the DPED10K data set compiled by using the SIFT (Scale-invariant feature transform) algorithm on the DPED data set, which can provide 10,000 pairs of input images and reference images. This implementation is implemented on the pictures provided by the DPED10K data set, where there are differences in perspective, color, etc. between the input image and its corresponding reference image.
[0050] In this embodiment, the image to be repaired is used as the input image, and the image to be repaired (i.e. Figure 2 The CorruptImage in a , the reference image (i.e. Figure 2 Reference Image) is represented as I b , then P=(I a ,I b ) will be used as an input pair. Among them, the image to be repaired I a and reference image I b Two encoders E(·) that share the same operating parameters (such as the same inertia weight, learning factor, etc.), that is, the image to be repaired I a Input to the first encoder, reference image I b Enter the second encoder.
[0051] When the image to be repaired I a and reference image I b After passing through the corresponding encoder, the feature maps corresponding to the two images can be obtained respectively, that is, the image to be repaired I a Multiple first feature maps F can be obtained ai , such as F ai =[f a1 ,f a2 ,f a3 ,f a4 ,f a5 ,f a6 ], where f ai Represents the image to be repaired I at the i-th layer a The image features of a1 、f a2 and f a3 I is the image to be repaired a The first shallow feature map, f a4 、f a5 and f a6 I is the image to be repaired a The first deep feature map of b Multiple second feature maps F can be obtained bi , such as F bi=[f b1 ,f b2 ,f b3 ,f b4 ,f b5 ,f b6 ], where f bi Represents the reference image I of the i-th layer b The image features of b1 、f b2 and f b3 is the reference image I b The second shallow feature map, f b4 、f b5 and f b6 is the reference image I b The second deep feature map of .
[0052] Then multiple first shallow feature maps are spliced to obtain the first texture feature f of the image to be repaired ate ,Right now And multiple first deep feature maps are spliced to obtain the first structural feature f of the image to be repaired ast ,Right now Similarly, multiple second shallow feature maps are spliced to obtain the second texture feature f of the reference image bte ,Right now And multiple second deep feature maps are spliced to obtain the second structural feature f of the reference image bst ,Right now
[0053] Step S20: performing alignment and fusion processing on the first texture feature and the second texture feature as well as the first structure feature and the second structure feature respectively to obtain a fused texture feature and a fused structure feature;
[0054] Furthermore, the aligning and fusing the first texture feature and the second texture feature and the first structure feature and the second structure feature respectively to obtain the fused texture feature and the fused structure feature includes:
[0055] Performing an alignment process on the first texture feature and the second texture feature to obtain an aligned texture feature, and performing a fusion process on the aligned texture feature and the first texture feature to obtain a fused texture feature;
[0056] The first structural feature and the second structural feature are aligned to obtain an aligned structural feature, and the aligned structural feature and the first structural feature are fused to obtain a fused structural feature.
[0057] Furthermore, the aligning process is performed on the first texture feature and the second texture feature to obtain the aligned texture feature, including:
[0058] Connecting the first texture feature and the second texture feature to obtain a connected texture feature;
[0059] The connected texture features are calculated and processed based on the offset estimator to obtain a texture feature offset map;
[0060] The second texture feature is subjected to deformable convolution processing based on the texture feature offset map to obtain an aligned texture feature.
[0061] For example, see Figure 2 and Figure 3 As shown, in order to better make the image to be repaired I a The first texture feature f ate and the first structural feature f ast With reference image I b The corresponding second texture feature f bte and the second structural feature f bst In this embodiment, the feature alignment module M(·) (i.e., FAM) is used to achieve this function. Specifically, the image to be repaired I a The features (i.e., Target features) and the reference image I b The features of the image to be repaired (i.e., Reference features) are connected a The first texture feature f ate and reference image I b The second texture feature f bte Connect and get the connection texture feature F te ,Right now And the image to be repaired I a The first structural feature f ast and reference image I b The second structural feature f bst Connect and obtain the connection structure feature F st ,Right now
[0062] Then, the potential similarity of the two images from near to far is calculated by the offset estimator, and then the feature offset map is obtained. Specifically, based on the offset estimator, the connected texture features F te and the connection structure feature F st The texture feature offset map P is calculated by a parallel standard convolution unit. te and structural feature offset map P st ; Then in the texture feature offset map P te and structural feature offset map P stUnder the guidance of , that is, using the interpolation algorithm and end-to-end learning through gradient back propagation, the reference image I b After the features of are passed through the deformable convolution (i.e., Deformable block), the aligned features (i.e., Aligned features) are obtained, i.e., the second texture feature f bte and the second structural feature f bst After passing through deformable convolution (i.e., Deformable block), the corresponding aligned texture features f are obtained. alignedte and alignment structure features f alignedst .
[0063] Among them, the alignment feature can be calculated by the following formula X represents input, y represents output, n and N represent the index and number of convolution kernels respectively, ω n represents the weight of the nth convolution kernel, p represents the index of the center, and p n Represents the fixed offset of the nth bit, Δp n Represents the learnable offset of the nth convolution kernel.
[0064] Finally, the alignment features are combined with the image to be repaired I a Specifically, the aligned texture feature f alignedte and the first texture feature f ate Perform fusion processing to obtain the fused texture feature f te (i.e. Fused Textures), At the same time, the alignment structure feature f alignedst and the first structural feature f ast Perform fusion processing to obtain the fusion structure feature f st (i.e. Fused Structures), i.e.
[0065] Step S30: performing multi-scale filling and repairing on the fused texture features and the fused structure features to obtain filled texture features and filled structure features;
[0066] For example, for the fused texture feature f obtained in step S20 te and fusion structural features f st In this embodiment, a texture branch and a structure branch are designed to respectively te With f st Repair and obtain the filling texture feature f mte and fill structure feature f mstEach branch (i.e. Multi-scale filling) contains multiple partial convolution paths with different convolution kernels (e.g., convolution kernels are 3x3, 5x5, 7x7, etc.) to ensure that the hole area can be fully filled in multiple ranges. That is, the damaged area on the image can be updated after each operation of partial convolution, so it can effectively handle irregular hole areas. And because of the establishment of convolution kernels of different sizes, f te and f st It can obtain filling and repairing in multiple ranges and scales. In addition, in this process, the reconstruction loss function L texture (i.e., texture loss function) and L structure (i.e., structural loss function) will be used to supervise the generation of filling texture features and filling structure features.
[0067] Step S40: Connecting and convolving the filling texture feature and the filling structure feature to obtain a balanced feature;
[0068] Furthermore, the filling texture feature and the filling structure feature are connected and convolved to obtain a balanced feature, including:
[0069] Based on the attention mechanism, the filling texture features and filling structure features are connected and convolved to obtain balanced features.
[0070] For example, the filling texture feature f after multi-scale filling obtained in step S30 mte and fill structure feature f mst , this implementation uses the feature equalization module to fill the texture feature f mte and fill structure feature f mst Perform concatenation and convolution to obtain the balanced feature f for equalization e (i.e. Feature Equalization), then Since there are always differences between structural features and texture features, this embodiment can design a balanced feature module based on the attention mechanism so that the fused features can be assigned according to certain weights.
[0071] Step S50: input the equalization feature to the decoder, so that the decoder jumps and connects the equalization feature with the first feature map input to the decoder to generate a restored image.
[0072] For example, in the embodiment of the present application, the equalization feature will be skipped and connected with the feature layers of different sizes in the decoder to generate the final repaired image I out(i.e. Output). Specifically, the image to be repaired I a After passing through the encoder, the generated f a1 、f a2 、f a3 、f a4 、f a5 and f a6 The feature map is input to the decoder and jump-connected with feature layers of different sizes in the decoder. The residual module Res block composed of dilated convolution can increase the size of the receptive field. Therefore, in this embodiment, the Res block can be used to further better perceive the f extracted by the encoder. a6 , and output the perception results; the balanced features are also jump-connected with the feature layers of different sizes in the decoder, and then the balanced features are respectively connected to each f a1 、f a2 、f a3 、f a4 、f a5 and f a6 Add (i.e., Element-wise addition) and output the repaired image; in this process, L rec (i.e., pixel reconstruction loss function), L perc (i.e., perceptual loss function), L style (i.e., style loss function) and L adv (i.e., the relativistic average LS adversarial loss function) is used to supervise the images generated during the entire training process.
[0073] In addition, the present application also compares the image restoration results obtained by this embodiment with those obtained by other image restoration methods (such as EC, GI and MEDFE, etc.). Among them, the image restoration evaluation index uses PSNR (peak signal-to-noise ratio) and SSIM value (structural similarity); PSNR is an objective standard for evaluating images. The higher the PSNR value, the higher the quality of the generated image; SSIM measures the structural similarity based on the brightness, contrast and structure between samples x and y. The higher the SSIM value, the more realistic the generated image.
[0074] The test process of this embodiment tests 200 images on the DPED10K dataset at different loss levels (for example, 10%, 20%, and then increasing to 60%), calculates the average PSNR value and average SSIM value, and compares the restoration results obtained by the three image restoration methods of EC, GI, and MEDFE (see Table 1 for specific results). From the comparison of PSNR and SSIM values in Table 1, it can be found that the image restoration method of this embodiment based on the reference image to guide the inference of structure and texture features has significantly improved performance compared with the existing image restoration algorithms.
[0075] Table 1 PSNR / SSIM values of the repaired test images with damage levels between 10% and 60%
[0076] method 10%-20% 20%-30% 30%-40% 40%-50% 50%-60% MEDFE 29.88 / 0.957 26.45 / 0.912 23.86 / 0.854 22.05 / 0.787 19.44 / 0.638 EC 29.30 / 0.950 26.37 / 0.909 24.23 / 0 / 858 22.40 / 0.794 20.10 / 0.668 GC 31.24 / 0.965 27.34 / 0.927 24.86 / 0.877 22.67 / 0.821 19.88 / 0.694 This embodiment 31.77 / 0.967 28.07 / 0.935 25.63 / 0.888 23.49 / 0.832 20.58 / 0.694
[0077] In order to solve the ill-posed problem of image restoration, this embodiment introduces a reference image with similar texture and structure into image restoration to increase external information, that is, the rich texture details of the reference image are used to compensate for the lost details in the image to be restored, thereby generating more detailed and realistic content. Considering that there are pattern differences between texture features and structural features during image restoration, such as large style differences in texture and geometric differences in structure, this embodiment introduces a reference image as a guide to infer texture features and structural features respectively. Specifically, the image to be restored and the reference image will first be encoded to generate their own texture features and structural features; then the texture features and structural features will be aligned and fused, and the fused texture features and structural features will be fused and balanced in a multi-scale filling manner to ensure that the attention in each channel is the same and the feature representation is consistent; finally, the balanced features will be connected to the decoder through a jump with the features from the encoder to generate the restored image.
[0078] It can be seen that the embodiment of the present application introduces a reference image into the encoding and decoding network to make good use of external information to infer the structural characteristics and texture characteristics of damaged images such as those with large damaged areas or expected content containing complex semantic information, thereby obtaining better repair results; at the same time, fully consider the potential differences between the image to be repaired and the reference image, and effectively fuse the features between the two to improve the repair effect.
[0079] See also Figure 4 As shown, the embodiment of the present application also provides an image restoration device, comprising:
[0080] A feature extraction unit, which is used to input the image to be repaired into a first encoder and a reference image corresponding to the image to be repaired into a second encoder, obtain a first feature map of the image to be repaired and a second feature map of the reference image, and perform feature extraction on the first feature map and the second feature map respectively to obtain a first texture feature and a first structural feature of the image to be repaired and a second texture feature and a second structural feature of the reference image;
[0081] An alignment and fusion unit, which is used to perform alignment and fusion processing on the first texture feature and the second texture feature and the first structure feature and the second structure feature respectively, to obtain a fused texture feature and a fused structure feature;
[0082] A feature filling unit, which is used to perform multi-scale filling and repair on the fused texture features and the fused structure features to obtain filled texture features and filled structure features;
[0083] A feature equalization unit, which is used to connect and convolve the filling texture feature and the filling structure feature to obtain a balanced feature;
[0084] The image reconstruction unit is used to input the equalization feature into the decoder, so that the decoder jumps and connects the equalization feature with the first feature map input into the decoder to generate a restored image.
[0085] This application introduces a reference image into the encoding and decoding network to make good use of external information to infer the structural and texture features of damaged images such as those with large damaged areas or expected content containing complex semantic information, thereby obtaining better restoration results; at the same time, it fully considers the potential differences between the image to be restored and the reference image, and effectively fuses the features between the two to improve the restoration effect.
[0086] Furthermore, the first feature map of the image to be repaired includes multiple first shallow feature maps and multiple first deep feature maps, and the second feature map of the reference image includes multiple second shallow feature maps and multiple second deep feature maps.
[0087] Furthermore, the feature extraction unit is specifically used for:
[0088] Splicing the plurality of first shallow feature maps to obtain a first texture feature of the image to be repaired, and splicing the plurality of first deep feature maps to obtain a first structural feature of the image to be repaired;
[0089] The multiple second shallow feature maps are spliced to obtain the second texture features of the reference image, and the multiple second deep feature maps are spliced to obtain the second structural features of the reference image.
[0090] Furthermore, the alignment and fusion unit is specifically used for:
[0091] Performing an alignment process on the first texture feature and the second texture feature to obtain an aligned texture feature, and performing a fusion process on the aligned texture feature and the first texture feature to obtain a fused texture feature;
[0092] The first structural feature and the second structural feature are aligned to obtain an aligned structural feature, and the aligned structural feature and the first structural feature are fused to obtain a fused structural feature.
[0093] Furthermore, the alignment and fusion unit is further used for:
[0094] Connecting the first texture feature and the second texture feature to obtain a connected texture feature;
[0095] The connected texture features are calculated and processed based on the offset estimator to obtain a texture feature offset map;
[0096] The second texture feature is subjected to deformable convolution processing based on the texture feature offset map to obtain an aligned texture feature.
[0097] Furthermore, the feature balancing unit is specifically used for:
[0098] Based on the attention mechanism, the filling texture features and filling structure features are connected and convolved to obtain balanced features.
[0099] Furthermore, the operating parameters of the first encoder and the second encoder are the same.
[0100] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device and each unit can refer to the corresponding process in the aforementioned image restoration method embodiment, and will not be repeated here.
[0101] The apparatus provided in the above embodiment may be implemented in the form of a computer program. The computer program may be Figure 5 The image shown is run on the repair device.
[0102] An embodiment of the present application also provides an image restoration device, comprising: a memory, a processor, and a network interface connected via a system bus, wherein at least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement all or part of the steps of the aforementioned image restoration method.
[0103] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0104] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.
[0105] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the computer device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a video playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as video data, image data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device or other volatile solid-state storage device.
[0106] The embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, all or part of the steps of the aforementioned image restoration method are implemented.
[0107] The embodiment of the present application implements all or part of the aforementioned process, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of each of the above methods when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electric carrier signals and telecommunication signals.
[0108] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, servers or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.
[0109] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0110] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0111] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.
Claims
1. An image restoration method, characterized in that: The following steps are involved: Inputting the image to be repaired into a first encoder and the reference image corresponding to the image to be repaired into a second encoder to obtain a first feature map of the image to be repaired and a second feature map of the reference image, and performing feature extraction on the first feature map and the second feature map respectively to obtain a first texture feature and a first structural feature of the image to be repaired and a second texture feature and a second structural feature of the reference image; Performing alignment and fusion processing on the first texture feature and the second texture feature as well as the first structure feature and the second structure feature respectively to obtain a fused texture feature and a fused structure feature; Perform multi-scale filling and repair on the fused texture features and the fused structure features to obtain the filled texture features and the filled structure features; The filling texture feature and the filling structure feature are connected and convolved to obtain a balanced feature; Inputting the equalization feature into the decoder, so that the decoder jumps and connects the equalization feature with the first feature map input into the decoder to generate a restored image; The step of aligning and fusing the first texture feature and the second texture feature and the first structure feature and the second structure feature to obtain a fused texture feature and a fused structure feature comprises: Performing an alignment process on the first texture feature and the second texture feature to obtain an aligned texture feature, and performing a fusion process on the aligned texture feature and the first texture feature to obtain a fused texture feature; Performing an alignment process on the first structural feature and the second structural feature to obtain an aligned structural feature, and performing a fusion process on the aligned structural feature and the first structural feature to obtain a fused structural feature; The aligning process of the first texture feature and the second texture feature to obtain the aligned texture feature includes: Connecting the first texture feature and the second texture feature to obtain a connected texture feature; The connected texture features are calculated and processed based on the offset estimator to obtain a texture feature offset map; The second texture feature is subjected to deformable convolution processing based on the texture feature offset map to obtain an aligned texture feature.
2. The image restoration method according to claim 1, wherein: The first feature map of the image to be repaired includes multiple first shallow feature maps and multiple first deep feature maps, and the second feature map of the reference image includes multiple second shallow feature maps and multiple second deep feature maps.
3. The image restoration method according to claim 2, characterized in that: The step of extracting features from the first feature map and the second feature map respectively to obtain a first texture feature and a first structural feature of the image to be repaired and a second texture feature and a second structural feature of the reference image includes: Splicing the plurality of first shallow feature maps to obtain a first texture feature of the image to be repaired, and splicing the plurality of first deep feature maps to obtain a first structural feature of the image to be repaired; The multiple second shallow feature maps are spliced to obtain the second texture features of the reference image, and the multiple second deep feature maps are spliced to obtain the second structural features of the reference image.
4. The image restoration method according to claim 1, wherein: The connecting and convolution processing of the filling texture feature and the filling structure feature to obtain the balanced feature includes: Based on the attention mechanism, the filling texture features and filling structure features are connected and convolved to obtain balanced features.
5. The image restoration method according to claim 1, wherein: The operating parameters of the first encoder and the second encoder are the same.
6. An image restoration device, characterized in that: include: A feature extraction unit, which is used to input the image to be repaired into a first encoder and a reference image corresponding to the image to be repaired into a second encoder, obtain a first feature map of the image to be repaired and a second feature map of the reference image, and perform feature extraction on the first feature map and the second feature map respectively to obtain a first texture feature and a first structural feature of the image to be repaired and a second texture feature and a second structural feature of the reference image; An alignment and fusion unit, which is used to perform alignment and fusion processing on the first texture feature and the second texture feature and the first structure feature and the second structure feature respectively, to obtain a fused texture feature and a fused structure feature; A feature filling unit, which is used to perform multi-scale filling and repair on the fused texture features and the fused structure features to obtain filled texture features and filled structure features; A feature equalization unit, which is used to connect and convolve the filling texture feature and the filling structure feature to obtain a balanced feature; An image reconstruction unit, which is used to input the equalization feature to the decoder, so that the decoder jumps and connects the equalization feature with the first feature map input to the decoder to generate a restored image; Wherein, the alignment and fusion unit is specifically used for: Performing an alignment process on the first texture feature and the second texture feature to obtain an aligned texture feature, and performing a fusion process on the aligned texture feature and the first texture feature to obtain a fused texture feature; Performing an alignment process on the first structural feature and the second structural feature to obtain an aligned structural feature, and performing a fusion process on the aligned structural feature and the first structural feature to obtain a fused structural feature; The alignment and fusion unit is further specifically used for: Connecting the first texture feature and the second texture feature to obtain a connected texture feature; The connected texture features are calculated and processed based on the offset estimator to obtain a texture feature offset map; The second texture feature is subjected to deformable convolution processing based on the texture feature offset map to obtain an aligned texture feature.
7. An image restoration device, characterized in that: include: A memory and a processor, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the image restoration method according to any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the image restoration method according to any one of claims 1 to 5 is implemented.
