Image Processing Method and Apparatus, Electronic Device, Storage Medium

Abstract

Embodiments of the present disclosure relate to an image processing method, apparatus, electronic device, and storage medium, and relate to the field of image processing technologies. The image processing method includes: decomposing a to-be-repaired image with a first resolution to obtain multiple high-frequency to-be-repaired images and a to-be-repaired image with a second resolution; repairing the to-be-repaired image with the second resolution to obtain a repaired image with the second resolution; and performing image super-resolution reconstruction on each of the high-frequency to-be-repaired images according to the repaired image with the second resolution and the prior information corresponding to each of the high-frequency to-be-repaired images to generate a target image corresponding to the to-be-repaired image with the first resolution. The technical solution of the present disclosure can improve the accuracy of image repair and the image quality.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to an image processing method, an image processing apparatus, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the process of image processing, there may be partial missing defective images, so image repair is required to repair the defective images into complete images.

[0003] In the related art, image repair usually includes performing image content repair on a low-resolution image and super-resolving the low-resolution image into an original-resolution image. When super-resolving the low-resolution image into an original-resolution image, a similarity matrix between two image patches can be learned from the repaired low-resolution image, and a high-resolution image can be obtained according to the similarity matrix. It is also possible to upsample the low-resolution image in the repaired area and fill the high-resolution image to be repaired, and then perform image reconstruction.

[0004] In the above methods, the fineness of the relationship matrix between image patches is low, so the quality of the repaired image is poor, and the accuracy of the obtained repaired image is poor. In addition, in the above methods, only the network itself can be used for learning, which has certain limitations and is difficult.

[0005] It should be noted that the information disclosed in the above background art section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to those of ordinary skill in the art. Summary of the Invention

[0006] The purpose of the present disclosure is to provide an image processing method, an apparatus, an electronic device, and a storage medium, so as to at least to some extent overcome the problem of poor accuracy of image repair caused by the limitations and defects of the related art.

[0007] Other features and advantages of the present disclosure will become apparent through the following detailed description, or will be partially learned through the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, there is provided an image processing method, including: decomposing an image to be repaired with a first resolution to obtain a plurality of high-frequency images to be repaired and an image to be repaired with a second resolution; repairing the image to be repaired with the second resolution to obtain a repaired image with the second resolution; and performing image super-resolution reconstruction on each of the high-frequency images to be repaired according to the repaired image with the second resolution and the prior information corresponding to each of the high-frequency images to be repaired, to generate a target image corresponding to the image to be repaired with the first resolution.

[0009] According to one aspect of the present disclosure, there is provided an image processing apparatus, including: an image decomposition module configured to decompose a to-be-patched image with a first resolution to obtain a plurality of high-frequency to-be-patched images and a to-be-patched image with a second resolution; an image patching module configured to patch the to-be-patched image with the second resolution to obtain a patched image with the second resolution; and an image reconstruction module configured to perform image super-resolution reconstruction on each of the high-frequency to-be-patched images according to the patched image with the second resolution and the prior information corresponding to each of the high-frequency to-be-patched images, and generate a target image corresponding to the to-be-patched image with the first resolution.

[0010] According to one aspect of the present disclosure, there is provided an electronic device, including: a processor; and a memory configured to store executable instructions of the processor; wherein the processor is configured to execute the image processing method according to any one of the above by executing the executable instructions.

[0011] According to one aspect of the present disclosure, there is provided a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image processing method according to any one of the above is implemented.

[0012] In the image processing method, image processing apparatus, electronic device, and computer-readable storage medium provided in the embodiments of the present disclosure, after decomposing a to-be-patched image with a first resolution to obtain a plurality of high-frequency to-be-patched images and a to-be-patched image with a second resolution, image super-resolution reconstruction is performed according to the patched image with the second resolution and each high-frequency to-be-patched image, and a target image corresponding to the to-be-patched image with the first resolution is generated. On the one hand, the prior information corresponding to the residuals of the high-frequency to-be-patched images is used for image super-resolution reconstruction. Since the residuals can be accurately obtained, and then patching is performed according to the prior information of the residuals, it is possible to avoid phenomena such as blurring or unnatural details in the patched high-definition image, improve the quality of the patched target image, and accurately patch the image, thereby improving the accuracy of image patching. On the other hand, since the to-be-patched image with the first resolution is decomposed and the prior information corresponding to the residuals of the high-frequency to-be-patched images is used for image super-resolution reconstruction, due to the existence of prior information as a reference, the limitation of only using a network for super-resolution is avoided, the application range is increased, the difficulty of image reconstruction is reduced, and the image reconstruction efficiency is improved.

[0013] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. Description of the Drawings

[0014] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0015] Figure 1 A schematic diagram showing an application scenario of an image processing method or an image processing apparatus to which embodiments of the present disclosure can be applied.

[0016] Figure 2 A schematic diagram showing the structure of an electronic device suitable for implementing embodiments of the present disclosure.

[0017] Figure 3 A schematic diagram schematically showing an image processing method in an embodiment of the present disclosure.

[0018] Figure 4 A schematic diagram schematically showing the process of image reconstruction in an embodiment of the present disclosure.

[0019] Figure 5 A schematic diagram schematically showing the specific process of image processing in an embodiment of the present disclosure.

[0020] Figure 6 A schematic diagram schematically showing the structure of a network model in an embodiment of the present disclosure.

[0021] Figure 7 A schematic diagram schematically showing the process of model training in an embodiment of the present disclosure.

[0022] Figure 8 A schematic diagram schematically showing the process of determining a loss function in an embodiment of the present disclosure.

[0023] Figure 9 A schematic diagram schematically showing an image inpainting in an embodiment of the present disclosure.

[0024] Figure 10 A block diagram schematically showing an image processing apparatus in an embodiment of the present disclosure. Detailed implementation manners

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present disclosure. However, those skilled in the art will realize that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or may be implemented using other methods, components, devices, steps, etc. In other cases, well-known technical solutions are not shown or described in detail to avoid obscuring the various aspects of the present disclosure.

[0026] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus repeated descriptions thereof will be omitted. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0027] The steps of the current high-definition image inpainting method can generally be divided into two steps: the first step is to perform image content inpainting on the low-resolution image; the second step is to super-resolve the low-resolution image into the original resolution image. For the first step, ComodGAN based on the generative model, HiFill or ProFill based on feature similarity filling can be used. The method in this exemplary embodiment is used to restore the low-resolution image with inpainted content into a high-resolution image with clear details.

[0028] Compared with general image super-resolution, high-definition image inpainting has the following differences: general super-resolution usually super-resolves into a fixed multiple, while the super-resolution multiple of high-definition image inpainting is usually not fixed; the input information of general super-resolution is only the low-resolution image, and the input of high-definition image inpainting super-resolution in addition to the low-resolution image also includes the high-definition image outside the inpainting area, which is equivalent to providing additional prior information of high-definition resolution. Therefore, general super-resolution is not commonly used in this problem.

[0029] In the related art, for high-definition image inpainting, there are mainly the following two types of methods for super-resolving a low-resolution image into an original-resolution image. The first type of method is to learn a similarity matrix between pairwise image patches (an image patch refers to a collection of several adjacent pixels) from the inpainted low-resolution image (the number of divided image patches is usually fixed, such as 32×32), and then apply this similarity matrix to the residual between the high-definition image and the inpainted low-resolution image to learn the residual of the image in the area to be inpainted, and then add it back to the low-resolution image to obtain the high-definition image. The fineness of the relationship matrix between image patches in this method is relatively low (since the number of image patches is much lower than the number of image pixels), and generally, the higher the resolution, the worse the restoration effect. Moreover, the mapping relationship of the texture and the mapping relationship of the residual do not exactly correspond, so the learned residual is not accurate enough. The second type of method is to upsample the low-resolution image in the inpainted area and then fill the high-definition image to be inpainted (at this time, a complete image is obtained, composed of the high-definition image + the low-resolution image), and then use this image as the input to a network similar to the Unet encoder-decoder network for high-definition image restoration. This method does not utilize the prior information provided by the difference between the high-definition image and the low-resolution image, but relies entirely on the network to learn by itself, so the effect is poor.

[0030] To solve the above technical problems, an image processing method is provided in an embodiment of the present disclosure, which can be applied to scenarios that require high-definition image inpainting and reconstruction.

[0031] Figure 1 The schematic diagram shows an application scenario to which the image processing method or the image processing apparatus according to the embodiment of the present disclosure can be applied.

[0032] This image processing method can be applied to an application scenario of inpainting high-definition images. Refer to Figure 1 As shown in, specifically, it can be applied to the process of inpainting the to-be-inpainted image 102 received on the client by the client 101. Among them, the client 101 can be various types of devices with computing functions, such as a smart phone, a tablet computer, a desktop computer, a vehicle-mounted device, a wearable device, etc. The to-be-inpainted image 102 can be any type of high-definition image in various scenarios, which can be a captured image or an image obtained from the network or other terminals, and the type of the image is not limited. Moreover, the to-be-inpainted image can be a high-definition image that has partial missing parts and needs to be inpainted. The client 101 can use the neural network model of the Laplacian pyramid to decompose the high-definition to-be-inpainted image into multiple high-frequency to-be-inpainted images and a low-resolution to-be-inpainted image. Further, inpainting is performed according to the prior information corresponding to the low-resolution inpainted image and the residuals of the multiple high-frequency to-be-inpainted images to achieve image reconstruction, and a complete image corresponding to the high-definition to-be-inpainted image is obtained.

[0033] It should be noted that the image processing method provided by the embodiments of the present disclosure can be entirely executed by the client. Correspondingly, the image processing device can be disposed in the client.

[0034] Figure 2 FIG. shows a schematic diagram of an electronic device suitable for implementing the exemplary embodiments of the present disclosure. The terminal of the present disclosure can be configured in the form of the electronic device as Figure 2 shown. However, it should be noted that Figure 2 the electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present disclosure.

[0035] The electronic device of the present disclosure at least includes a processor and a memory. The memory is used to store one or more programs. When the one or more programs are executed by the processor, the processor can implement the methods of the exemplary embodiments of the present disclosure.

[0036] Specifically, as Figure 2 shown, the electronic device 200 may include: a processor 210, an internal memory 221, an external memory interface 222, a Universal Serial Bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 271, a receiver 272, a microphone 273, a headphone interface 274, a sensor module 280, a display screen 290, a camera module 291, an indicator 292, a motor 293, a key 294, and a Subscriber Identification Module (SIM) card interface 295, etc. The sensor module 280 may include a depth sensor, a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0037] It can be understood that the structure schematically shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown, or combine certain components, or split certain components, or have different component arrangements. The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0038] The processor 210 may include one or more processing units. For example, the processor 210 may include an Application Processor (AP), a modem processor, a Graphics Processing Unit (GPU), an Image Signal Processor (ISP), a controller, a video codec, a Digital Signal Processor (DSP), a baseband processor, and / or a Neural-network Processing Unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processors. In addition, a memory may be provided in the processor 210 for storing instructions and data.

[0039] The USB interface 230 is an interface compliant with the USB standard specification. Specifically, it may be a Mini USB interface, a Micro USB interface, a USB Type-C interface, etc. The USB interface 230 can be used to connect a charger to charge the electronic device 200, and can also be used for data transmission between the electronic device 200 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. This interface can also be used to connect other electronic devices, etc.

[0040] The charging management module 240 is used to receive a charging input from a charger. Among them, the charger may be a wireless charger or a wired charger. The power management module 241 is used to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives inputs from the battery 242 and / or the charging management module 240 to supply power to the processor 210, the internal memory 221, the display screen 290, the camera module 291, the wireless communication module 260, etc.

[0041] The wireless communication function of the electronic device 200 can be implemented through antenna 1, antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor, and the baseband processor, etc.

[0042] The mobile communication module 250 can provide solutions for wireless communications such as 2G / 3G / 4G / 5G applied to the electronic device 200.

[0043] The wireless communication module 260 may provide solutions for wireless communications applied to the electronic device 200, including wireless local area networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), Frequency Modulation (FM), Near Field Communication (NFC), Infrared (IR), etc.

[0044] The electronic device 200 implements a display function through a GPU, a display screen 290, an application processor, etc. The GPU is a microprocessor for image blurring, connected to the display screen 290 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.

[0045] The electronic device 200 may implement a shooting function through an ISP, a camera module 291, a video codec, a GPU, a display screen 290, an application processor, etc. In some embodiments, the electronic device 200 may include one or N camera modules 291, where N is a positive integer greater than 1. If the electronic device 200 includes N cameras, one of the N cameras is the main camera, and the others may be secondary cameras, such as telephoto cameras.

[0046] The internal memory 221 may be used to store computer-executable program code, and the executable program code includes instructions. The internal memory 221 may include a program storage area and a data storage area. The external memory interface 222 may be used to connect an external memory card, such as a Micro SD card, to implement the storage capacity expansion of the electronic device 200.

[0047] The electronic device 200 may implement an audio function through an audio module 270, a speaker 271, a receiver 272, a microphone 273, a headphone interface 274, and an application processor, etc. For example, music playback, recording, etc.

[0048] The audio module 270 is used to convert digital audio information into an analog audio signal for output, and is also used to convert analog audio input into digital audio signals. The audio module 270 may also be used to encode and decode audio signals. In some embodiments, the audio module 270 may be disposed in the processor 210, or some functional modules of the audio module 270 may be disposed in the processor 210.

[0049] The speaker 271 is used to convert an audio electrical signal into a sound signal. The electronic device 200 can listen to music or a hands-free call through the speaker 271. The receiver 272, also known as the "earpiece", is used to convert an audio electrical signal into a sound signal. When the electronic device 200 answers a call or a voice message, the voice can be listened to by bringing the receiver 272 close to the human ear. The microphone 273, also known as the "microphone" or "transmitter", is used to convert a sound signal into an electrical signal. When making a call or sending a voice message, the user can speak close to the microphone 273 with the mouth to input the sound signal into the microphone 273. The electronic device 200 can be provided with at least one microphone 273. The headphone jack 274 is used to connect a wired headphone.

[0050] Regarding the sensors included in the electronic device 200, the depth sensor is used to obtain the depth information of a scene. The pressure sensor is used to sense a pressure signal and can convert the pressure signal into an electrical signal. The gyroscope sensor can be used to determine the motion posture of the electronic device 200. The barometric pressure sensor is used to measure the barometric pressure. The magnetic sensor includes a Hall sensor. The electronic device 200 can use the magnetic sensor to detect the opening and closing of a flip leather case. The acceleration sensor can detect the magnitude of the acceleration of the electronic device 200 in various directions (generally three axes). The distance sensor is used to measure the distance. The proximity light sensor can include, for example, a light-emitting diode (LED) and a light detector, such as a photodiode. The fingerprint sensor is used to collect fingerprints. The temperature sensor is used to detect the temperature. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. A visual output related to the touch operation can be provided through the display screen 290. The ambient light sensor is used to sense the ambient light brightness. The bone conduction sensor can obtain a vibration signal.

[0051] The keys 294 include a power-on key, volume keys, etc. The keys 294 can be mechanical keys or touch keys. The motor 293 can generate a vibration prompt. The motor 293 can be used for an incoming call vibration prompt or for touch vibration feedback. The indicator 292 can be an indicator light and can be used to indicate the charging state, the change in battery level, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 295 is used to connect a SIM card. The electronic device 200 interacts with the network through the SIM card to implement functions such as calls and data communication.

[0052] This application also provides a computer-readable storage medium. The computer-readable storage medium can be included in the electronic device described in the above embodiments; or it can exist alone without being assembled into the electronic device.

[0053] A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0054] The computer-readable storage medium can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the above.

[0055] The computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to implement the methods described in the following embodiments.

[0056] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code that contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0057] The units described in the embodiments of the present disclosure can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not, in some cases, constitute a limitation on the unit itself. Next, the image processing method in the embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0058] Please refer to Figure 3 and Figure 5 , in step S310, the to-be-patched image with the first resolution is decomposed to obtain multiple high-frequency to-be-patched images and a to-be-patched image with the second resolution.

[0059] In the embodiments of the present disclosure, the to-be-patched image with the first resolution may be a to-be-patched image with a high resolution, for example, a defective high-definition image. The to-be-patched image with the first resolution may include a complete region and a defective region. The complete region includes high-frequency information and low-frequency information, and there is no high-frequency information in the defective region, that is, the detailed information is lost. Based on this, the to-be-patched image with the first resolution can be decomposed into multiple layers by the super-resolution network model of the Laplacian pyramid to obtain multiple high-frequency to-be-patched images and a to-be-patched image with the second resolution. Among them, the first resolution is greater than the second resolution. The multiple high-frequency to-be-patched images may be high-frequency to-be-patched images with different resolutions, and the to-be-patched image with the second resolution may be a low-frequency and low-resolution to-be-patched image.

[0060] The Laplacian pyramid decomposition is used to decompose an image into different spatial frequency bands respectively. The super-resolution network model of the Laplacian pyramid can be used for image decomposition and image reconstruction. Refer to Figure 6 shown in, the super-resolution network model of the Laplacian pyramid may include three parts: Laplacian pyramid decomposition, residual patching network, and Laplacian pyramid reconstruction, which are mainly used to implement the Laplacian pyramid decomposition process, image patching, and Laplacian pyramid reconstruction process. The number of convolutional layers of the image patching network can be adjusted accordingly according to the resolution to be repaired. The higher the resolution, the more convolutional layers can be used to obtain a larger network capacity. For example, Figure 6 the convolutional layer of the image patching network in may be 3 layers.

[0061] On this basis, the to-be-patched image with the first resolution can be decomposed into multiple layers according to the number of layers of the super-resolution network model of the Laplacian pyramid to obtain multiple high-frequency to-be-patched images and a to-be-patched image with the second resolution. The number of layers of the super-resolution network model of the Laplacian pyramid refers to the number of pyramid layers included in the super-resolution network model, which can be represented by L. For example, the number of pyramid layers can be two or three, etc., and can be adjusted and set according to specific requirements. The more layers of the Laplacian pyramid, the higher the resolution it can repair. Therefore, by changing the number of layers of the Laplacian pyramid, image restoration with different resolutions can be realized, avoiding the limitation in the related art that only image restoration with a fixed resolution can be performed, and improving the application range and flexibility. And, among the number of layers of the super-resolution network model of the Laplacian pyramid, the resolution decreases sequentially from top to bottom. For example, the resolution represented by the first layer is greater than the resolution represented by the second layer. Refer toFigure 6 As shown, when the number of layers L of the super-resolution network model of the Laplacian pyramid is 2, according to the Laplacian decomposition method, the image to be repaired at the first resolution (i.e., the high-definition image to be repaired I 0 ) can be decomposed into 2 high-frequency images to be repaired and one low-resolution image to be repaired I L . Specifically, the 2 high-frequency images to be repaired can be represented by h 0 and h L-1 . Then, the decomposed images can be repaired through the residual repair network, thereby realizing the Laplacian pyramid reconstruction.

[0062] The high-frequency image to be repaired refers to the high-resolution image to be repaired. However, the resolution of the high-frequency image to be repaired is less than the first resolution and greater than the second resolution. Moreover, the resolutions of multiple high-frequency images to be repaired are also different, and the resolutions of multiple high-frequency images to be repaired decrease with the increase of the decomposition times. For the resolutions of multiple high-frequency images to be repaired, they can be decreased according to a preset ratio, and the preset ratio can be 50% or the like. That is, the resolution can be decreased according to the preset ratio, and the image to be repaired at the first resolution can be decomposed according to the decreased resolution. For example, when the image to be repaired at the first resolution is decomposed into 2 high-frequency images to be repaired and 1 low-resolution image to be repaired at the second resolution, the resolution of the first high-frequency image to be repaired is greater than that of the second high-frequency image to be repaired. The resolution of the first high-frequency image to be repaired is half of the first resolution, and the resolution of the second high-frequency image to be repaired is half of the resolution of the first high-frequency image to be repaired. In the embodiments of the present disclosure, each high-frequency image to be repaired can be a residual image, and there are incomplete regions and complete regions in each high-frequency image to be repaired. Therefore, the high-frequency image to be repaired can also be called a residual image, and the corresponding residual can be obtained according to the high-frequency image to be repaired.

[0063] Specifically, for the image to be repaired I at the first resolution 0Performing Laplace decomposition can obtain L high-frequency images to be repaired and one image to be repaired with the second resolution (low-frequency image to be repaired). When performing the decomposition, the high-frequency images to be repaired can be obtained according to the difference between the current image to be repaired and the upsampled image obtained by upsampling the current image to be repaired. Specifically, the current image to be repaired can be downsampled to obtain a downsampled image, and then upsampled to obtain an upsampled image corresponding to the resolution of the current image to be repaired. The residual image corresponding to the high-frequency image to be repaired is determined according to the residual between the current image to be repaired and the upsampled image corresponding to the current image to be repaired, so as to determine the residual. Among them, the current image to be repaired refers to the image to be repaired in the current layer, which can be obtained by downsampling the image to be repaired in the previous layer, or can be obtained by downsampling the original image to be repaired with the first resolution by a preset ratio. The preset ratio of downsampling for each current image to be repaired is different. Among them, the upsampling can be 2-fold upsampling or upsampling by other multiples to obtain the same size as the original Figure 1 sample. Specifically, formula (1) can be referred to to determine multiple high-frequency images to be repaired to represent the residual image:

[0064] h i =I i -up(dn(I i )) Formula (1)

[0065] Among them, dn(·) represents 2-fold downsampling, updn(·) represents 2-fold upsampling, and I i represents the current image to be repaired, and h i represents the high-frequency image to be repaired.

[0066] Based on this, the original image to be repaired with the first resolution can be downsampled at different resolutions, and then the upsampled image obtained by upsampling the downsampled image with a lower resolution subtracted from the downsampled image with a higher resolution to obtain the residual image. It should be noted that the image to be repaired with the smallest resolution is the image to be repaired with the second resolution, and the residual does not need to be calculated.

[0067] At the same time, the image to be repaired with the second resolution I 0 can also be obtained by downsampling the image to be repaired with the first resolution I L . The downsampling can be 2-fold downsampling for L times, and is specifically determined according to formula (2):

[0068] I L =dn(I 0 ) L Formula (2)

[0069] Among them, dn(·) L represents 2-fold downsampling for L times.

[0070] By taking the image to be repaired of the first resolution I 0 Perform Laplace multi-layer decomposition to obtain multiple high-frequency images to be repaired (h 0 ,……,h L-1 ) and an image I to be repaired at a second resolution L , thereby calculating the residuals corresponding to multiple high-frequency images to be repaired, which can avoid the problem of inaccurate residuals caused by the incomplete correspondence between the mapping relationship of the texture and the mapping relationship of the residual in the related technology, and can make the residual learning more hierarchical, thereby reducing the difficulty of network learning and improving the accuracy of the residual.

[0071] In step S320, the image to be repaired with the second resolution is repaired to obtain a repaired image with the second resolution.

[0072] In the embodiment of the present disclosure, the image to be repaired at the second resolution may be referred to as a low-frequency image to be repaired, and the second resolution may be a low resolution and smaller than the first resolution.

[0073] For the image to be repaired at the second resolution, a repair operation can be performed on the image to be repaired at the second resolution according to the image repair model to obtain a repaired image at the second resolution. Accordingly, the repaired image at the second resolution can be a low-resolution repaired image, that is, a low-frequency image. The image repair model can be various types of models that can be used to repair images, such as a Comodgan model or other models.

[0074] For the Comodgan model, it includes a generator and a discriminator. The generator is used to generate a patched image, and the discriminator is used to identify whether a certain image is a real image or a generated image. Through the generator and the discriminator, the generated patched image can be made more consistent with the real image. Based on this, the image to be patched at the second resolution can be input into the Comodgan model, and the generator therein can generate a patched image of the second resolution corresponding to the image to be patched at the second resolution.

[0075] In step S330, image super-resolution reconstruction is performed on each of the high-frequency images to be repaired according to the repaired image of the second resolution and the prior information of each of the high-frequency images to be repaired, so as to generate a target image corresponding to the image to be repaired of the first resolution.

[0076] In the disclosed embodiment, the residual refers to the difference between different images, and is specifically used here to represent the residual corresponding to the high-frequency image to be repaired. The residual between the current resolution image, the current resolution image downsampled to half the resolution, and then upsampled to the original resolution image can be used. The high-frequency image to be repaired includes a defective area and a complete area. Prior information refers to the known part of each layer of the high-frequency image to be repaired that does not need to be repaired, that is, the complete area in each decomposed high-frequency image to be repaired. Super-resolution reconstruction refers to restoring the completed low-resolution image to a high-resolution image. For the same image, its super-resolution mode is the same. Therefore, when the image includes a defective area and a complete area, the super-resolution mode of the defective area and the complete area is exactly the same, so the prior information can provide reference information for the repair process of the defective area in the high-frequency image to be repaired. Based on this, the residual high-frequency information of the complete area of ​​the high-frequency image to be repaired can be used to repair the high-frequency information of the high-frequency image to be repaired step by step, so as to perform image super-resolution reconstruction, thereby obtaining a target image. The target image can be a complete image corresponding to the image to be repaired at the first resolution, that is, a repaired image. Based on this, the resolution of the target image can also be the first resolution.

[0077] Figure 4 A flowchart for image reconstruction is schematically shown in FIG. Figure 4 As shown in , it mainly includes the following steps:

[0078] In step S410, the inpainted image of the second resolution is cascaded with the current image among the plurality of high-frequency images to be inpainted to obtain a inpainted current image.

[0079] In this step, the current image can be the image to be repaired with the lowest resolution among multiple high-frequency images to be repaired. The repaired image with the second resolution can be upsampled to obtain an upsampled image. Further, the upsampled image and the current image can be concatenated and input into the super-resolution network model of the Laplacian pyramid for convolution operation to obtain the repaired current image. Specifically, the upsampling can be 2-fold upsampling. The repaired image with the second resolution and the current image among multiple high-frequency images to be repaired can be concatenated at the channel level, that is, the repaired image with the second resolution and the same channels of the current image are concatenated. Further, the upsampled image and the current image can be concatenated and input into the convolution layer in the super-resolution network model of the Laplacian pyramid for convolution operation to obtain the repaired current image. The convolution layer can be in the image repair network of the super-resolution network model of the Laplacian pyramid. The convolution layer can be 3 gated convolution layers, or a general convolution layer or a dilation convolution layer. Since the image to be repaired with the first resolution is decomposed by Laplacian to obtain multiple high-frequency images to be repaired and a low-frequency image to be repaired, and there are complete regions that do not need to be repaired in each high-frequency image to be repaired, and these complete regions can be used as prior information of the residual. Therefore, when performing image repair, the prior information corresponding to the residual of the high-frequency image to be repaired is utilized, improving the accuracy.

[0080] In step S420, the repaired current image and the repaired image with the second resolution are fused to obtain a fused image, and the fused image and the previous adjacent image of the current image are concatenated for image repair.

[0081] In this step, after obtaining the repaired current image, the repaired image with the second resolution can be upsampled to obtain an upsampled repaired image. Further, the upsampled repaired image and the repaired current image can be fused to obtain a fused image. Among them, the fusion process refers to an addition operation, that is, an addition operation is performed on the pixels of the upsampled image and the repaired current image. After obtaining the fused image, the fused image can be upsampled again to obtain an upsampled fused image, and the upsampled fused image and the adjacent image of the current image are concatenated and input into the convolution layer of the super-resolution network model of the Laplacian pyramid for convolution operation for high-frequency image repair. The previous adjacent image of the current image refers to the previous layer of high-frequency image to be repaired adjacent to the current image among multiple high-frequency images to be repaired, and the resolution of the previous adjacent image is greater than the resolution of the current image. It should be added that for the Laplacian pyramid, the resolution corresponding to each layer decreases in order from top to bottom.

[0082] During the process of image repair, according to the repaired image I with the second resolution L The first reconstructed image I' after upsampling iWith the high-frequency image h to be repaired i-1 Perform cascading to obtain the repaired high-frequency image h' i-1 , specifically as shown in formula (3), that is, for h' i-1 and I' i Perform cascading to obtain the repaired high-frequency image h' i-1 . Further, the first reconstructed image I' i Can be upsampled and added to the repaired high-frequency image h' i-1 To obtain the second reconstructed image I' i-1 , as shown in formula (4) for example. Among them, h' i Represents the repaired high-frequency image, I' i Represents the reconstructed image, and I' 0 Represents the finally output high-definition reconstructed image. Among them, the value of i ranges from 0 to L. It can be understood that the first reconstructed image I' i Corresponds to the (i - 1)-th one among the multiple high-frequency images to be repaired (h 0 , ……, h L-1 ) obtained in the Laplacian pyramid decomposition. The second reconstructed image I' i-1 Corresponds to the (i - 2)-th one among the multiple high-frequency images to be repaired (h 0 , ……, h L-1 ) obtained in the Laplacian pyramid decomposition.

[0083] h' i-1 = G(h i-1 , up(I' L )) Formula (3)

[0084] I' i-1 = up(I' i ) + h' i-1 Formula (4)

[0085] Figure 5 Schematically shows the specific flowchart of image processing. Refer to Figure 5 As shown in, the input is the image to be repaired and the mask 501 for indicating the repair area. The image to be repaired refers to the image to be repaired at the first resolution (high-definition image to be repaired). First, downsample the high-definition image to be repaired to obtain the low-resolution image to be repaired (image to be repaired at the second resolution) and the corresponding mask 502. Then, input the low-resolution image to be repaired and the corresponding mask into an image repair model 503 to obtain the low-resolution repaired image 504. The image repair model can be, for example, the Comodgan model. Further, the low-resolution repaired image and the prior information of the high-frequency image to be repaired are input into the Laplacian pyramid super-resolution network model 505 for super-resolution processing to obtain the high-definition repaired image (target image) 506 as the output.Figure 5 The technical solution in [reference] utilizes the prior information of the high-frequency image to be repaired and the low-resolution repaired image to perform image super-resolution reconstruction on the high-frequency image to be repaired, which can improve the accuracy of image reconstruction.

[0086] Figure 6 The structural diagram of the network model is schematically shown in [reference]. Referring to Figure 6 as shown in [reference], when the number of layers L of the super-resolution network model of the Laplacian pyramid is 2, the image to be repaired with the first resolution (i.e., the high-definition image to be repaired I 0 ) is decomposed into multiple high-frequency images to be repaired h 0 and h L-1 , and a low-resolution image to be repaired I L . And the low-resolution image to be repaired I L can be repaired to obtain a low-resolution repaired image I' L .

[0087] Further, when i is L, the low-resolution repaired image I' L is first upsampled by 2 times to obtain a first reconstructed image. The first reconstructed image is cascaded with the high-frequency image to be repaired h L-1 and input into a 3-layer gated convolutional layer in the residual repair network to obtain a repaired high-frequency image h' L-1 . Further, the repaired high-frequency image h' L-1 is added to the upsampled first reconstructed image, and then cascaded with the adjacent upper-layer high-frequency image to be repaired h L-2 and input into a 3-layer gated convolutional layer sharing weights for high-frequency image repair, so as to obtain a repaired image corresponding to the high-definition image to be repaired, that is, the target image I' 0 .

[0088] The technical solution provided by the embodiments of the present disclosure utilizes the prior information corresponding to the residual of the high-frequency image to be repaired to perform image super-resolution reconstruction, which can improve the quality of the repaired target image and the accuracy of image repair. In addition, using the prior information corresponding to the residual of the high-frequency image to be repaired for image super-resolution reconstruction reduces the difficulty of image reconstruction and improves the efficiency of image reconstruction.

[0089] In the embodiments of the present disclosure, in order to improve the accuracy, the neural network model can be trained with a reference image of the first resolution to obtain a super-resolution network model of the Laplacian pyramid.

[0090] Figure 7 The flowchart of model training is schematically shown in [reference]. Referring to Figure 7 as shown in [reference], it mainly includes the following steps:

[0091] In step S710, the reference image with the first resolution is decomposed into a high-frequency reference image and a reference image with the second resolution.

[0092] In this step, the reference image with the first resolution can also be a complete image with the first resolution, that is, an image without defective areas and does not need to be repaired. The reference image with the first resolution can also be a defective image to be repaired. The reference image with the first resolution can be the same as or different from the type of the image to be repaired with the first resolution. The reference image with the first resolution can be decomposed into multiple high-frequency reference images and a reference image with the second resolution through a super-resolution network model of the Laplacian pyramid. The resolution of the high-frequency reference image is less than the first resolution and greater than the second resolution. Moreover, the resolutions of the multiple high-frequency reference images are also different, and the resolutions of the multiple high-frequency reference images decrease as the number of decomposition times increases. For the resolution, it can be reduced according to a preset ratio, and the preset ratio can be 50% or the like. That is, the resolution can be reduced according to a preset ratio, and the reference image with the first resolution is decomposed according to the reduced resolution.

[0093] Since in the embodiments of the present disclosure, when performing image super-resolution and image repair on the image to be repaired with the first resolution, the input of the model is the defective image to be repaired. Therefore, when the reference image with the first resolution is a complete image, the high-frequency image obtained by decomposing the reference image with the first resolution can be processed to obtain a defective reference image to be repaired as the high-frequency reference image. Specifically, the reference image with the first resolution can be multiplied by a mask representing the repair area to obtain the reference image to be repaired.

[0094] In step S720, a loss function is determined according to the reference image with the first resolution and the reference repaired image with the first resolution.

[0095] In this step, the reference repaired image with the first resolution refers to the complete image corresponding to the reference image with the first resolution obtained by image reconstruction of the reference image with the first resolution. The loss function is used to describe the gap between the reference image with the first resolution and the reference repaired image with the first resolution.

[0096] Figure 8 The flowchart of calculating the loss function is schematically shown in Figure 8 As shown in

[0097] In step S810, a pixel-level reconstruction function is determined according to the gradient information of the reference image with the first resolution and the reference repaired image with the first resolution.

[0098] In this step, the pixel-level reconstruction function refers to the pixel domain and the gradient domain, which is specifically determined according to the difference between the reference image at the first resolution and the reference patched image at the first resolution, and the difference between the gradient information of the reference image at the first resolution and the reference patched image at the first resolution. The calculation formula is shown in Formula (5):

[0099]

[0100] In step S820, logical processing is performed on the discrimination results corresponding to the reference image at the first resolution and the reference patched image at the first resolution to obtain an adversarial generation loss function.

[0101] In this step, the discrimination result is the result of processing using a discriminator, and the discrimination result can be used to indicate whether the reference image at the first resolution and the reference patched image at the first resolution belong to real images or generated images. The logical processing includes absolute value calculation and average value calculation. Specifically, the difference between the discrimination result of the reference image at the first resolution and the reference patched image at the first resolution can be calculated first, and then the absolute value corresponding to the difference is further calculated, and the average value of the absolute value is calculated to obtain an adversarial generation loss function. Specifically, refer to Formula (6) shown below:

[0102] L gan =mean(abs(D(I 0 )-D(I' 0 ))) Formula (6)

[0103] Among them, D(·) represents the discriminator, abs(·) represents taking the absolute value, and mean(·) represents taking the average value.

[0104] In step S830, the loss function is determined according to the pixel-level reconstruction function and the adversarial generation loss function.

[0105] In this step, an addition operation can be performed on the pixel-level reconstruction function and the adversarial generation loss function to obtain a target function as the loss function.

[0106] For the discriminator, its loss function is obtained by performing logical operations on the discrimination results of the reference image at the first resolution and the reference patched image at the first resolution. Its purpose is to distinguish real images and reconstructed images as much as possible. Specifically, it can be determined according to Formula (7):

[0107] L D =mean(abs(1-D(I 0 )))+mean(abs(1+D(I' 0 ))) Formula (7)

[0108] After determining the loss function, the model parameters can be adjusted according to the loss function. It should be noted that during the process of training the model according to the loss function, a training method can be adopted for model training. Specifically, the training method includes: the batch size representing the number of data samples for one training is 8, that is, 8 samples are taken from the training set for training each time. The learning rate directly affects the convergence state of the model, and the initial learning rate can be set to 0.0001. Every 10 epochs, the learning rate decays by 0.5. When a complete data set passes through the neural network once and returns once, this process is called one epoch. One epoch is the process of training all training samples once, that is, all training samples have undergone a forward propagation and a backward propagation in the neural network.

[0109] In step S730, the model parameters of the neural network model are adjusted according to the loss function to obtain the super-resolution network model of the Laplacian pyramid.

[0110] In this step, the minimum of the loss function can be used as the training target, and the above training method can be adopted to adjust the model parameters until the training target of the minimum loss function is reached, and then the training process ends. Thus, the super-resolution network model of the Laplacian pyramid is determined according to the model parameters at the end of training.

[0111] After completing the model training, the patched image at the second resolution, the prior information corresponding to the residual between the image to be patched at the first resolution and the image to be patched at the second resolution can be input into the super-resolution network model of the Laplacian pyramid for image reconstruction to generate the target image corresponding to the image to be patched at the first resolution.

[0112] In the process of high-definition image patching, first perform Laplacian decomposition on the high-definition image, and then input the decomposed low-frequency image into the image patching model to obtain a low-resolution patched image. Then, input the low-resolution patched image and the high-frequency image into the super-resolution network model LPSR of the Laplacian pyramid for reconstruction to obtain a high-definition patched image. Refer to Figure 9 the schematic diagram of image patching shown in . Decompose the high-definition image A, and use HiFill as the image patching model to patch the obtained low-frequency image to obtain a low-resolution patched image B. Further, input the low-resolution patched image B and the decomposed high-frequency image into the super-resolution network model LPSR of the Laplacian pyramid for reconstruction to obtain a high-definition patched image C.

[0113] It should be added that when using the super-resolution network model of the Laplacian pyramid for image inpainting, the number of layers of the Laplacian pyramid can be positively correlated with the resolution of the image to be inpainted that can be processed. That is, the more layers the Laplacian pyramid has, the higher the resolution of the image to be inpainted. Therefore, by simply adjusting the number of layers of the Laplacian pyramid, image restoration at different resolutions can be achieved, avoiding the limitation in the related art that only image inpainting at a single resolution can be realized, and increasing flexibility and the scope of application.

[0114] The technical solution in the embodiments of the present disclosure is mainly used for the process of restoring the obtained low-resolution inpainted image to a high-resolution image during the high-definition image inpainting process. The input includes not only the low-resolution image but also the incomplete high-definition image to be inpainted. After embedding the Laplacian pyramid in the network, both the Laplacian pyramid decomposition and reconstruction are used. The Laplacian pyramid decomposition utilizes the prior information of high-definition image inpainting, and the reconstruction is to generate the high-frequency information of the area to be inpainted. The high-frequency residual of the high-frequency image to be inpainted in the complete area is calculated through the Laplacian pyramid decomposition method, and the prior information corresponding to the high-frequency residual is used to restore the residual information in multiple levels, which helps the inpainting learning of high-frequency information, can improve the accuracy of image inpainting, improve the quality of the inpainted image, and improve the accuracy of image reconstruction. The Laplacian multi-level decomposition method makes the residual learning more hierarchical. In addition to using the step-by-step restoration, it also utilizes the prior information. Since the super-resolution mode of the image is the same during the super-resolution process, when there is prior information provided by the super-resolution mode in the area that does not need to be inpainted, the learning difficulty of the network can be reduced. By sharing the weights between different layers of the pyramid and changing the number of layers of the Laplacian pyramid, image restoration at different resolutions can be achieved, image inpainting at multiple high-definition resolutions can be realized, the resolution can be flexibly adjusted, and it has high portability.

[0115] The method in the embodiments of the present disclosure can be used in other high-definition image editing algorithms, such as defogging and de-reflection. First, image editing is performed on the low-resolution image, and then the super-resolution network model of the Laplacian pyramid is used to perform high-definition image restoration. It can also be applied to some high-resolution cameras to achieve functions such as removing pedestrians and debris in high-resolution photos of mobile phones.

[0116] An image processing apparatus is provided in the embodiments of the present disclosure. Referring to Figure 10 as shown, the image processing apparatus 1000 may include:

[0117] An image decomposition module 1001, configured to decompose the image to be inpainted with the first resolution to obtain multiple high-frequency images to be inpainted and an image to be inpainted with the second resolution;

[0118] An image inpainting module 1002 for inpainting the to-be-inpainted image of the second resolution to obtain an inpainted image of the second resolution;

[0119] An image reconstruction module 1003 for performing image super-resolution reconstruction on each of the high-frequency to-be-inpainted images according to the inpainted image of the second resolution and the prior information corresponding to each of the high-frequency to-be-inpainted images, and generating a target image corresponding to the to-be-inpainted image of the first resolution.

[0120] In an exemplary embodiment of the present disclosure, the image decomposition module includes: a decomposition control module for decomposing the to-be-inpainted image of the first resolution using a super-resolution network model of a Laplacian pyramid to obtain multiple high-frequency to-be-inpainted images and a to-be-inpainted image of the second resolution; wherein the sum of the number of the multiple high-frequency to-be-inpainted images and the number of the to-be-inpainted images of the second resolution is equal to the number of layers of the pyramid in the super-resolution network model of the Laplacian pyramid.

[0121] In an exemplary embodiment of the present disclosure, the number of layers of the pyramid in the super-resolution network model of the Laplacian pyramid is positively correlated with the first resolution.

[0122] In an exemplary embodiment of the present disclosure, the image reconstruction module includes: an image concatenation module for concatenating the inpainted image of the second resolution with the current image among the multiple high-frequency to-be-inpainted images to obtain a current inpainted image; an image fusion module for performing a fusion process on the current inpainted image and the inpainted image of the second resolution to obtain a fusion image, and concatenating the fusion image with the previous adjacent image of the current image for image inpainting.

[0123] In an exemplary embodiment of the present disclosure, the image concatenation module is configured to: upsample the inpainted image of the second resolution, concatenate the upsampled image with the current image and perform a convolution operation to obtain a current inpainted image.

[0124] In an exemplary embodiment of the present disclosure, the image fusion module includes: an upsampling module for upsampling the inpainted image of the second resolution to obtain an upsampled inpainted image; a fusion control module for adding the current inpainted image and the upsampled inpainted image to obtain a fusion image; a convolution processing module for upsampling the fusion image again, and performing a convolution operation on the upsampled fusion image and the adjacent image of the current image for image inpainting.

[0125] In an exemplary embodiment of the present disclosure, the apparatus further includes: a reference image decomposition module, configured to decompose a reference image with a first resolution into a plurality of high-frequency reference images and a reference image with a second resolution; a loss function determination module, configured to determine a loss function according to the reference image with the first resolution and a reference repaired image with the first resolution; and a model training module, configured to adjust model parameters of a neural network model according to the loss function to obtain a super-resolution network model of a Laplacian pyramid.

[0126] In an exemplary embodiment of the present disclosure, the loss function determination module includes: a first determination module, configured to determine a pixel-level reconstruction function according to gradient information of the reference image with the first resolution and the reference repaired image with the first resolution; a second determination module, configured to perform logical processing on the reference image with the first resolution and the reference repaired image with the first resolution to obtain an adversarial generation loss function; and a loss function generation module, configured to determine the loss function according to the pixel-level reconstruction function and the adversarial generation loss function.

[0127] It should be noted that the specific details of each module in the above image processing apparatus have been described in detail in the corresponding image processing method, and thus will not be elaborated herein.

[0128] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions for causing a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0129] In addition, the above drawings are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, rather than for limiting purposes. It is easy to understand that the processes shown in the above drawings do not indicate or limit the time sequence of these processes. Additionally, it is also easy to understand that these processes can be executed synchronously or asynchronously in, for example, multiple modules.

[0130] It should be noted that although several modules or units of a device for action execution are mentioned in the above detailed description, such a division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0131] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and examples are only to be considered as exemplary, and the true scope and spirit of the present disclosure are pointed out by the claims.

[0132] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. An image processing method, characterized in that, comprising: decomposing the image to be repaired with the first resolution to obtain multiple high-frequency images to be repaired and an image to be repaired with the second resolution; repairing the image to be repaired with the second resolution to obtain a repaired image with the second resolution; performing image super-resolution reconstruction on each of the high-frequency images to be repaired according to the repaired image with the second resolution and the prior information corresponding to each of the high-frequency images to be repaired, and generating a target image corresponding to the image to be repaired with the first resolution; the prior information is the complete area that does not need to be repaired in the high-frequency image to be repaired; wherein, the performing image super-resolution reconstruction on each of the high-frequency images to be repaired according to the repaired image with the second resolution and the prior information corresponding to each of the high-frequency images to be repaired, and generating a target image corresponding to the image to be repaired with the first resolution, includes: cascading the repaired image with the second resolution with the current image among the multiple high-frequency images to be repaired to obtain a repaired current image; performing a fusion process on the repaired current image and the repaired image with the second resolution to obtain a fusion image, and cascading the fusion image with the previous adjacent image of the current image for image repair.

2. The image processing method according to claim 1, characterized in that, the decomposing the image to be repaired with the first resolution to obtain multiple high-frequency images to be repaired and an image to be repaired with the second resolution, includes: using a super-resolution network model of the Laplacian pyramid to decompose the image to be repaired with the first resolution to obtain multiple high-frequency images to be repaired and an image to be repaired with the second resolution; wherein, the sum of the number of the multiple high-frequency images to be repaired and the number of the image to be repaired with the second resolution is equal to the number of layers of the pyramid in the super-resolution network model of the Laplacian pyramid.

3. The image processing method according to claim 2, characterized in that, the number of layers of the pyramid in the super-resolution network model of the Laplacian pyramid is positively correlated with the first resolution.

4. The image processing method according to claim 1, characterized in that, the cascading the repaired image with the second resolution with the current image among the multiple high-frequency images to be repaired to obtain a repaired current image, includes: performing upsampling on the repaired image with the second resolution, cascading the upsampled image with the current image and performing a convolution operation to obtain a repaired current image.

5. The image processing method according to claim 1, characterized in that, the performing a fusion process on the repaired current image and the repaired image with the second resolution to obtain a fusion image, and cascading the fusion image with the previous adjacent image of the current image for image repair, includes: performing upsampling on the repaired image with the second resolution to obtain an upsampled repaired image; adding the repaired current image and the upsampled repaired image to obtain a fusion image; performing upsampling on the fusion image again, and performing a convolution operation on the upsampled fusion image and the previous adjacent image of the current image for image repair.

6. The image processing method according to claim 2, wherein, the method further includes: decomposing a reference image with a first resolution into multiple high-frequency reference images and a reference image with a second resolution; determining a loss function according to the reference image with the first resolution and the reference repaired image with the first resolution; adjusting the model parameters of the neural network model according to the loss function to obtain a super-resolution network model of the Laplacian pyramid.

7. The image processing method according to claim 6, wherein, the determining of the loss function according to the reference image with the first resolution and the reference repaired image with the first resolution includes: determining a pixel-level reconstruction function according to the gradient information of the reference image with the first resolution and the reference repaired image with the first resolution; performing a logical process on the reference image with the first resolution and the reference repaired image with the first resolution to obtain an adversarial generation loss function; determining the loss function according to the pixel-level reconstruction function and the adversarial generation loss function.

8. An image processing apparatus, wherein, it includes: an image decomposition module, configured to decompose a to-be-repaired image with a first resolution to obtain multiple high-frequency to-be-repaired images and a to-be-repaired image with a second resolution; an image repair module, configured to repair the to-be-repaired image with the second resolution to obtain a repaired image with the second resolution; an image reconstruction module, configured to perform image super-resolution reconstruction on each of the high-frequency to-be-repaired images according to the repaired image with the second resolution and the prior information corresponding to each of the high-frequency to-be-repaired images, and generate a target image corresponding to the to-be-repaired image with the first resolution; the prior information is a complete area that does not need to be repaired in the high-frequency to-be-repaired image; wherein, the performing of image super-resolution reconstruction on each of the high-frequency to-be-repaired images according to the repaired image with the second resolution and the prior information corresponding to each of the high-frequency to-be-repaired images, and generating a target image corresponding to the to-be-repaired image with the first resolution includes: cascading the repaired image with the second resolution with the current image among the multiple high-frequency to-be-repaired images to obtain a repaired current image; performing a fusion process on the repaired current image and the repaired image with the second resolution to obtain a fusion image, and cascading the fusion image with the previous adjacent image of the current image for image repair.

9. An electronic device, wherein, it includes: a processor; and a memory, configured to store executable instructions of the processor; wherein, the processor is configured to execute the image processing method according to any one of claims 1-7 by executing the executable instructions.

10. A computer-readable storage medium, on which a computer program is stored, wherein, the computer program, when executed by a processor, implements the image processing method according to any one of claims 1-7.

Images