Image restoration method, device and electronic equipment

Abstract

The present invention provides an image restoration method, device and electronic device, which relates to the technical field of image processing, including obtaining an image to be restored; inputting the image to be restored into a pre-trained image restoration network; wherein the image restoration network includes a feature extraction module and a weight calculation module based on an attention mechanism; and performing restoration processing on the image to be restored through the image restoration network to obtain a restored image corresponding to the image to be restored. The present invention can better restore the image, thereby significantly improving the effect of image restoration.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to an image restoration method, device and electronic equipment. Background Art

[0002] An image is an information carrier of an objective object. However, an image with unclear quality will affect the user's visual experience of the objective object described in the image. At present, neural networks are usually used to repair unclear images to obtain corresponding high-definition images. However, due to the various reasons for unclear image quality and the weak generalization ability of existing image repair models, it is difficult to generally use existing image repair models for repairing various types of images, resulting in poor results in the repaired high-definition images. Summary of the invention

[0003] In view of this, an object of the present invention is to provide an image restoration method, device and electronic device, which can better restore the image, thereby significantly improving the effect of image restoration.

[0004] In a first aspect, an embodiment of the present invention provides an image restoration method, comprising: obtaining an image to be restored; inputting the image to be restored into a pre-trained image restoration network; wherein the image restoration network comprises a feature extraction module and a weight calculation module based on an attention mechanism; and performing restoration processing on the image to be restored through the image restoration network to obtain a restored image corresponding to the image to be restored.

[0005] In one embodiment, the step of performing restoration processing on the image to be restored by the image restoration network to obtain a restored image corresponding to the image to be restored includes: extracting feature information of the image to be restored by the feature extraction module; extracting weight information of the target area in the image to be restored by the weight calculation module; and obtaining the restored image corresponding to the image to be restored based on the feature information and the weight information.

[0006] In one embodiment, the step of extracting the weight information of the target area in the image to be repaired through the weight calculation module includes: blurring the image to be repaired to obtain a blurred image corresponding to the image to be repaired; and extracting the weight information of the target area in the blurred image through the weight calculation module.

[0007] In one embodiment, the step of obtaining a restored image corresponding to the image to be restored based on the feature information and the weight value of the target area includes: obtaining a restored image corresponding to the image to be restored based on the blurred image, the feature information and the weight information.

[0008] In one embodiment, the step of obtaining a restored image corresponding to the image to be restored based on the blurred image, the feature information and the weight information includes: bitwise multiplying the feature information and the weight information to obtain a first bitwise multiplication result; bitwise adding the first bitwise multiplication result and the blurred image, and using the bitwise addition result as the restored image corresponding to the image to be restored.

[0009] In one embodiment, the step of obtaining a restored image corresponding to the image to be restored based on the blurred image, the feature information and the weight information includes: comparing the weight information with a preset weight, and obtaining mask information corresponding to the image to be restored based on the comparison result; bitwise multiplying the feature information and the mask information to obtain a second bitwise multiplication result; bitwise adding the second bitwise multiplication result and the blurred image, and using the bitwise addition result as the restored image of the corresponding image to be restored.

[0010] In one embodiment, the image to be restored includes a low-definition face image, and the feature extraction module and / or the weight calculation module includes a fully convolutional network.

[0011] In a second aspect, an embodiment of the present invention further provides an image restoration device, comprising: an image acquisition module, used to acquire an image to be restored; an input module, used to input the image to be restored into a pre-trained image restoration network; wherein the image restoration network comprises a feature extraction module and a weight calculation module based on an attention mechanism; and a restoration module, used to perform restoration processing on the image to be restored through the image restoration network to obtain a restored image corresponding to the image to be restored.

[0012] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory; a computer program is stored in the memory, and when the computer program is executed by the processor, the method described in any one of the first aspects is executed.

[0013] In a fourth aspect, an embodiment of the present invention further provides a computer storage medium for storing computer software instructions used for any of the methods provided in the first aspect.

[0014] The image restoration method, device and electronic device provided by the embodiment of the present invention first obtain the image to be restored, and input the image to be restored into the pre-trained image restoration network, so as to restore the image to be restored through the image restoration network to obtain the restored image corresponding to the image to be restored, wherein the above-mentioned image restoration network includes a feature extraction module and a weight calculation module based on the attention mechanism. The above-mentioned method proposes a new type of image restoration network, which includes a feature extraction module and a weight calculation module based on the attention mechanism. The embodiment of the present invention uses the image restoration network to restore the image to be restored, which can not only significantly enhance the detailed texture in the image to be restored, but also effectively remove the noise in the image to be restored, so that the image can be better restored and the effect of image restoration can be significantly improved.

[0015] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0016] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of a flow chart of an image restoration method provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of the structure of an image restoration network provided by an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of the structure of an image restoration device provided by an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] At present, neural networks are usually used to repair images. For example, the repair of images containing human faces is taken as an example. Since the Generative Adversarial Networks (GAN) has a strong generation capability and can realize the generation process from scratch, the related art proposes to use the GAN network to repair low-definition face images to generate facial details in low-definition face images. Generally speaking, the facial features need to be highlighted in the face repair task, but due to the different sources of image data, there are various types of noise in the face image, such as various compression noises, Gaussian noises, mixed noises, etc., which makes it difficult for the existing model to be generally suitable for faces of various data sources, resulting in poor high-definition image repair results. Based on this, the present invention provides an image repair method, device and electronic device, which can better repair images, thereby significantly improving the effect of image repair.

[0024] To facilitate understanding of this embodiment, firstly, an image restoration method disclosed in an embodiment of the present invention is described in detail. Figure 1 The flowchart of an image restoration method shown in FIG. 1 mainly includes the following steps S102 to S106:

[0025] Step S102, obtaining an image to be repaired. The image to be repaired may be a low-definition image, such as an image whose clarity is lower than a preset clarity threshold, or an image with noise, and may be used as the image to be repaired. Optionally, the image to be repaired may include a low-definition face image, which may include a face area.

[0026] Step S104, input the image to be repaired into the pre-trained image repair network. The image repair network includes a feature extraction module and a weight calculation module based on the attention mechanism. The input of the feature extraction module is the image to be repaired, and the output of the feature extraction module is feature information, that is, the feature extraction module is used to extract feature information of the image to be repaired. The input of the weight calculation module is the image to be repaired or the image to be repaired after blurring, and the output of the weight calculation module is the weight information of the target area in the image to be repaired, that is, the weight calculation module is used to determine the weight information corresponding to the image to be repaired based on the attention mechanism. For example, if the image to be repaired is a low-definition face image, the feature extraction module will extract the feature information of the low-definition face image, and the weight calculation module will output the weight information of the face area in the low-definition face image.

[0027] Step S106, the image to be repaired is repaired through the image repair network to obtain a repaired image corresponding to the image to be repaired. The repaired image corresponding to the image to be repaired can be understood as a high-definition image obtained after repairing the image to be repaired. At this time, the clarity should be higher than the preset clarity threshold, or there is no noise in the image. In one embodiment, the feature information of the image to be repaired can be output through the feature extraction module, and the weight information of the target area in the image to be repaired can be output through the weight calculation module, so as to obtain the repaired image corresponding to the image to be repaired based on the feature information and the weight information.

[0028] The above-mentioned image restoration method provided by the embodiment of the present invention proposes a new type of image restoration network, which includes a feature extraction module and a weight calculation module based on an attention mechanism. The embodiment of the present invention uses the image restoration network to repair the image to be restored, which can not only significantly enhance the detail texture in the image to be restored, but also effectively remove the noise in the image to be restored, thereby better restoring the image and significantly improving the effect of image restoration.

[0029] To facilitate understanding of the above step S106, an embodiment of the present invention provides an implementation method of performing a repair process on the image to be repaired through an image repair network to obtain a repaired image corresponding to the image to be repaired, which can be seen from the following steps 1 to 3:

[0030] Step 1: extract feature information of the image to be repaired through a feature extraction module. In one embodiment, the feature extraction module may include a fully convolutional network, and each convolutional layer in the feature extraction module may be used to extract features of the image to be repaired, and the feature information output by the last convolutional layer is determined as the feature information of the image to be repaired.

[0031] Step 2, extracting the weight information of the target area in the image to be repaired through the weight calculation module. In one embodiment, the weight calculation module may also include a fully convolutional network. In order to obtain more accurate weight information, the embodiment of the present invention may blur the image to be repaired before inputting the image to be repaired into the weight calculation module to obtain a blurred image corresponding to the image to be repaired, and then extract the weight information of the target area in the blurred image through the weight calculation module. In an optional embodiment, a Gaussian blur algorithm may be used to blur the image to be repaired, wherein Gaussian blur is an image processing method based on normal distribution, and then the blurred image is input into the weight calculation module, thereby outputting the weight information through the weight calculation module. In practical applications, the above-mentioned feature information and weight information can be expressed in vector form, and the dimensions of the two are the same.

[0032] Step 3, based on the feature information and the weight information, obtain the restored image corresponding to the image to be restored. In one embodiment, the restored image corresponding to the image to be restored can be obtained based on the blurred image, the feature information and the weight information. In practical applications, the restored image corresponding to the image to be restored can be obtained according to the following method 1 or 2:

[0033] Method 1: Multiply the feature information and the weight information bit by bit to obtain the first bitwise multiplication result, and add the first bitwise multiplication result and the blurred image bit by bit, and use the bitwise addition result as the repaired image corresponding to the image to be repaired. Assume that the feature information is an N-dimensional feature vector O(F), and the weight information is an N-dimensional weight matrix O(W). For each element in the weight matrix, the element represents the weight value corresponding to a pixel in the image to be repaired. The blurred image is denoted by G lr , we can calculate the product of feature information and weight information to get the first bitwise multiplication result O(F)*O(W), and then calculate the first bitwise multiplication result O(F)*O(W) and the blurred image G lr Sum value, get the bitwise addition result G lr +O(F)*O(W), the bitwise addition result G lr +O(F)*O(W) can be regarded as the repaired image.

[0034] Method 2: Compare the weight information with the preset weight, and obtain the mask information corresponding to the image to be repaired based on the comparison result, perform bitwise multiplication on the feature information and the mask information to obtain a second bitwise multiplication result, and perform bitwise addition on the blurred image, and use the bitwise addition result as the repaired image corresponding to the image to be repaired. Optionally, for each element in the weight matrix, if the weight value of the pixel corresponding to the element is less than the preset weight, the mask value of the pixel corresponding to the element is set to 0, and if the weight value of the pixel corresponding to the element is greater than the preset weight, the mask value of the pixel corresponding to the element is set to 1, so that a mask matrix (that is, the above-mentioned mask information) that can characterize the mask value of each pixel can be obtained, and the dimension of the mask information is consistent with that of the weight information, so that the target area and non-target area in the image to be repaired can be distinguished by the mask information. After determining the mask information corresponding to the image to be repaired, the product of the mask information and the feature information can be calculated to obtain the second bitwise multiplication result, and then the second bitwise multiplication result is added to the blurred image bitwise to obtain the repaired image.

[0035] To facilitate understanding of the image restoration method provided in the above embodiment, the present invention provides an application example of the image restoration method, taking a low-definition face image as an example for explanation, see Figure 2 A structural diagram of an image restoration network is shown in FIG. Figure 2 As shown, the low-definition face image lr is input into the feature extraction module F, and the corresponding N-dimensional feature information O(F) can be obtained. The low-definition face image lr is Gaussian blurred to obtain the blurred image G lr , blur the image G lr Input to the weight calculation module W, we can get the high-frequency weight value O(W) of the face part in the low-definition face image lr, and then use the following formula: G lr + O(F)*O(W) to obtain the restored high-definition face image hr. The high-frequency weight value is also the aforementioned weight information.

[0036] To sum up, the above-mentioned image restoration method provided by the embodiment of the present invention obtains the residual value O(F)*O(W) based on the feature extraction module and the weight calculation module, obtains the high-frequency information in the low-definition face image, and enhances the high-frequency information through the attention mechanism of the weight calculation module, thereby enhancing the texture details (such as eyebrows, tooth edges, etc.) in the low-definition face image to a certain extent, so that the image can be better restored and the effect of image restoration can be significantly improved.

[0037] For the image restoration method provided in the above embodiment, the present invention provides an image restoration device, see Figure 3 The schematic diagram of the structure of an image restoration device shown in FIG. 1 mainly includes the following parts:

[0038] The image acquisition module 302 is used to acquire the image to be restored.

[0039] The input module 304 is used to input the image to be repaired into the pre-trained image repair network; wherein the image repair network includes a feature extraction module and a weight calculation module based on the attention mechanism.

[0040] The restoration module 306 is used to perform restoration processing on the image to be restored through an image restoration network to obtain a restored image corresponding to the image to be restored.

[0041] The above-mentioned image restoration device provided by the embodiment of the present invention proposes a new type of image restoration network, which includes a feature extraction module and a weight calculation module based on an attention mechanism. The embodiment of the present invention uses the image restoration network to repair the image to be restored, which can not only significantly enhance the detailed texture in the image to be restored, but also effectively remove the noise in the image to be restored, so that the image can be better restored and the effect of image restoration can be significantly improved.

[0042] In one embodiment, the repair module 306 is also used to: extract feature information of the image to be repaired through the feature extraction module; extract weight information of the target area in the image to be repaired through the weight calculation module; and obtain a repaired image corresponding to the image to be repaired based on the feature information and the weight information.

[0043] In one implementation, the restoration module 306 is further configured to: perform blur processing on the image to be restored to obtain a blurred image corresponding to the image to be restored; and extract weight information of the target area in the blurred image through the weight calculation module.

[0044] In one implementation, the restoration module 306 is further configured to obtain a restored image corresponding to the image to be restored based on the blurred image, the feature information and the weight information.

[0045] In one embodiment, the repair module 306 is further used to: bitwise multiply the feature information and the weight information to obtain a first bitwise multiplication result; bitwise add the first bitwise multiplication result and the blurred image, and use the bitwise addition result as the repaired image corresponding to the image to be repaired.

[0046] In one embodiment, the repair module 306 is also used to: compare the weight information with the preset weight, and obtain the mask information corresponding to the image to be repaired based on the comparison result; bit-wise multiply the feature information and the mask information to obtain a second bit-wise multiplication result; bit-wise add the second bit-wise multiplication result and the blurred image, and use the bit-wise addition result as the repaired image corresponding to the image to be repaired.

[0047] In one embodiment, the image to be restored includes a low-definition face image, and the feature extraction module and / or the weight calculation module includes a fully convolutional network.

[0048] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0049] An embodiment of the present invention provides an electronic device. Specifically, the electronic device includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned implementation methods.

[0050] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present invention, the electronic device 100 includes: a processor 40, a memory 41, a bus 42 and a communication interface 43, wherein the processor 40, the communication interface 43 and the memory 41 are connected via the bus 42; the processor 40 is used to execute an executable module stored in the memory 41, such as a computer program.

[0051] The memory 41 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 43 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0052] The bus 42 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0053] Among them, the memory 41 is used to store programs, and the processor 40 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 40 or implemented by the processor 40.

[0054] The processor 40 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 40. The above processor 40 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor to execute, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 41, and the processor 40 reads the information in the memory 41 and completes the steps of the above method in combination with its hardware.

[0055] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0056] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0057] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. An image restoration method, It is characterized in that include: Obtain the image to be repaired; Inputting the image to be repaired into a pre-trained image repair network; wherein the image repair network comprises a feature extraction module and a weight calculation module based on an attention mechanism; Performing a restoration process on the image to be restored by using the image restoration network to obtain a restored image corresponding to the image to be restored; The step of performing a restoration process on the image to be restored by the image restoration network to obtain a restored image corresponding to the image to be restored comprises: Extracting feature information of the image to be repaired by the feature extraction module; Extracting weight information of the target area in the image to be repaired by the weight calculation module; Based on the feature information and the weight information, obtaining a restored image corresponding to the image to be restored; The step of extracting weight information of the target area in the image to be repaired by the weight calculation module includes: Performing blur processing on the image to be restored to obtain a blurred image corresponding to the image to be restored; The weight information of the target area in the blurred image is extracted by the weight calculation module.

2. The method according to claim 1, It is characterized in that The step of obtaining the restored image corresponding to the image to be restored based on the feature information and the weight value of the target area includes: Based on the blurred image, the feature information and the weight information, a restored image corresponding to the image to be restored is obtained.

3. The method according to claim 2, It is characterized in that The step of obtaining a restored image corresponding to the image to be restored based on the blurred image, the feature information and the weight information comprises: Multiply the feature information and the weight information bit by bit to obtain a first bitwise multiplication result; The first bitwise multiplication result and the blurred image are bitwise added, and the bitwise addition result is used as a restored image corresponding to the image to be restored.

4. The method according to claim 2, It is characterized in that The step of obtaining a restored image corresponding to the image to be restored based on the blurred image, the feature information and the weight information comprises: Comparing the weight information with a preset weight, and obtaining mask information corresponding to the image to be repaired based on the comparison result; Performing bitwise multiplication on the feature information and the mask information to obtain a second bitwise multiplication result; The second bitwise multiplication result and the blurred image are bitwise added, and the bitwise addition result is used as the restored image corresponding to the image to be restored.

5. The method according to any one of claims 1 to 4, It is characterized in that The image to be restored includes a low-definition face image, and the feature extraction module and / or the weight calculation module includes a fully convolutional network.

6. An image restoration device, It is characterized in that include: An image acquisition module, used for acquiring an image to be repaired; An input module, used to input the image to be repaired into a pre-trained image repair network; wherein the image repair network includes a feature extraction module and a weight calculation module based on an attention mechanism; A restoration module, used to perform restoration processing on the image to be restored through the image restoration network to obtain a restored image corresponding to the image to be restored; The repair module is further used to: extract feature information of the image to be repaired through the feature extraction module; extract weight information of the target area in the image to be repaired through the weight calculation module; and obtain a repaired image corresponding to the image to be repaired based on the feature information and the weight information; The repair module is also used for: performing blur processing on the image to be repaired to obtain a blurred image corresponding to the image to be repaired; and extracting weight information of the target area in the blurred image through the weight calculation module.

7. An electronic device, It is characterized in that including a processor and a memory; The memory stores a computer program, which, when executed by the processor, performs the method according to any one of claims 1 to 5.

8. A computer storage medium, It is characterized in that Used to store computer software instructions used for the method according to any one of claims 1 to 5.

Images