Image restoration method, device, equipment and medium

Abstract

The present application discloses an image restoration method, apparatus, device, and medium, belonging to the field of image restoration. The method comprises: obtaining a defective image and a mask image, wherein the mask image is used to indicate the missing region on the defective image; performing image restoration on the missing region on the defective image at at least two scales, fusing the restoration results at the at least two scales to obtain a completed image; the at least two scales include resolution scales after upsampling or downsampling the defective image; and outputting the completed image. This method performs image restoration on the missing region on the defective image at at least two scales, and fusing the restoration results at the at least two scales, which can better prevent information loss, preserve the information of the original defective image, and make the restored completed image clearer and more complete.

Description

Technical Field

[0001] The present application relates to the field of image restoration, and in particular to an image restoration method, apparatus, device, and medium. Background Art

[0002] Image inpainting refers to the process of completing missing regions in digital images. The goal is to fill in missing pixels with semantically plausible and visually realistic content that is stylistically consistent with the rest of the image. Image inpainting techniques can be used as image editing tools, for example, to remove unwanted objects from images or restore damaged areas in paintings.

[0003] Related image inpainting methods primarily extract features from the valid regions of the original image to create an intermediate image, which is then used to fill in the missing regions of the original image, ultimately yielding the completed image. Supplementing the missing regions with information from the valid regions offers a compromise between inpainting effectiveness and computational cost.

[0004] However, the restoration method that supplements the missing area with information from the valid area inevitably cannot accurately restore the image. The missing area after the final restoration is only based on the information of the original image, which makes the restored image significantly different from the restored image in the ideal state. Summary of the Invention

[0005] This application provides an image restoration method, apparatus, device, and medium that can improve the clarity and accuracy of completed images. The technical solution is as follows:

[0006] According to one aspect of the present application, there is provided an image restoration method, the method comprising:

[0007] Acquire a defective image and a mask image, wherein the mask image is used to indicate a missing area on the defective image;

[0008] Performing image restoration on the missing region of the defective image at at least two scales, and fusing the restoration results at the at least two scales to obtain a completed image; the at least two scales include resolution scales after upsampling or downsampling the defective image;

[0009] The completed image is output.

[0010] According to another aspect of the present application, an image restoration device is provided, comprising:

[0011] an acquisition module, configured to acquire a defective image and a mask image, wherein the mask image is used to indicate a missing area on the defective image;

[0012] a restoration module, configured to perform image restoration on the missing region of the defective image at at least two scales, and fuse the restoration results at the at least two scales to obtain a completed image; the at least two scales include resolution scales after upsampling or downsampling the defective image;

[0013] An output module is used to output the completed image.

[0014] According to one aspect of the present application, a computer device is provided, comprising: a processor and a memory, wherein the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the image restoration method described above.

[0015] According to another aspect of the present application, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the image restoration method described above.

[0016] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described image restoration method.

[0017] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0018] By performing image restoration on the missing areas of the defective image at at least two scales and fusing the restoration results at the at least two scales, information loss can be better prevented, the information of the original defective image can be preserved, and the restored complement image can be made clearer and more complete. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0020] Figure 1 is a schematic diagram of an image restoration system according to an exemplary embodiment;

[0021] Figure 2 is a flow chart of an image restoration method according to an exemplary embodiment of the present application;

[0022] Figure 3is a schematic diagram of an image restoration method according to an exemplary embodiment of the present application;

[0023] Figure 4 is a schematic diagram of an image restoration method according to an exemplary embodiment of the present application;

[0024] Figure 5 is a flow chart of an image restoration method according to an exemplary embodiment of the present application;

[0025] Figure 6 is a schematic diagram of an image restoration model of an exemplary embodiment of the present application;

[0026] Figure 7 is a schematic diagram comparing an image restoration method according to an exemplary embodiment of the present application and other image restoration methods;

[0027] Figure 8 is a schematic diagram comparing an image restoration method according to an exemplary embodiment of the present application and other image restoration methods;

[0028] Figure 9 is a flow chart of a training method for an image restoration model according to an exemplary embodiment of the present application;

[0029] Figure 10 Schematic diagrams of several residual filters of an exemplary embodiment of the present application;

[0030] Figure 11 1 is a schematic diagram of FFT visualization results of 30 residual filters of an exemplary embodiment of the present application;

[0031] Figure 12 is a schematic diagram comparing an image restoration method according to an exemplary embodiment of the present application and other image restoration methods;

[0032] Figure 13 is a schematic diagram comparing an image restoration method according to an exemplary embodiment of the present application and other image restoration methods;

[0033] Figure 14 is a schematic diagram comparing an image restoration method according to an exemplary embodiment of the present application and other image restoration methods;

[0034] Figure 15 is a schematic diagram comparing an image restoration method according to an exemplary embodiment of the present application and other image restoration methods;

[0035] Figure 16 is a schematic diagram of an image restoration result of an exemplary embodiment of the present application;

[0036] Figure 17 is a structural block diagram of an image restoration device provided by an exemplary embodiment of the present application;

[0037] Figure 18 A structural block diagram of an electronic device provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0038] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0039] First, a brief introduction to the terms involved in the embodiments of this application is given:

[0040] Image restoration: refers to the use of information about the neighborhood of missing or redundant parts of an image and the overall structural information of the image to fill in missing areas or remove redundant areas of an image according to certain restoration rules.

[0041] Image restoration model: refers to the algorithmic model and / or machine learning model that can be used for image restoration.

[0042] Traditional image restoration models can be divided into structure-based and texture-based models. With the continuous development of computer vision technology, image restoration models also include deep learning-based image restoration models.

[0043] More specifically, deep learning-based image restoration models include auto-encoders (AE), convolutional neural networks (CNN), and generative adversarial neural networks (GAN).

[0044] Computer Vision (CV): Computer vision is the study of how machines can "see." Specifically, it refers to machine vision techniques such as using cameras and computers to replace the human eye in identifying, tracking, and measuring objects. Further image processing is performed to transform the images into images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping, and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0045] PEN (Pyramid-context Encoder): A pyramid context encoder network that fills in missing regions by gradually learning regional attention. For more information, see: Yanhong Zeng, Jianlong Fu, Hongyang Chao, and Baining Guo. Learning pyramid-context encoder network for high-quality image inpainting. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1486–1494, 2019.

[0046] Gconv (Gated Convolution): A two-stage generative network from coarse to fine, introducing gated convolution. For more information, see: Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. In Int. Conf. Comput. Vis., pages 4471–4480, 2019.

[0047] MEDFE (Mutual Encoder-Decoder with Feature Equalizations): Jointly recovering structure and texture. For more information, see: Hongyu Liu, Bin Jiang, Yibing Song, Wei Huang, and Chao Yang. Rethinking image inpainting via amutual encoder-decoder with feature equalizations. In Eur. Conf. Comput. Vis., 2020.

[0048] GDN (Detection-based Generative Network): A generative framework based on pixel-wise dense detectors for image inpainting. For more information, please refer to the paper: Ruisong Zhang, Weize Quan, Baoyuan Wu, Zhifeng Li, and Dong-Ming Yan. Pixel-wise dense detector for image inpainting. Comput. Graph. Forum, 39(7), 2020.

[0049] RFR (Recurrent Feature Reasoning): Uses recursive feature reasoning and knowledge-consistent attention in feature space for progressive image inpainting. For more information, see: Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, and Dacheng Tao. Recurrent feature reasoning for image inpainting. In IEEE Conf. Comput. Vis. Pattern Recog., pages 7757–7765, 2020.

[0050] Mask image: refers to an image used to block the target image. The mask image is used to control the area or processing process of the image.

[0051] In one embodiment of the present application, the mask image is used to indicate the missing area of ​​the defective image, and the mask image is a binary mask, where 0 represents a valid pixel and 1 represents a missing pixel.

[0052] Sample image set: The image restoration model provides sample missing images required for training. In some embodiments, the sample image set includes at least one of the Large-Scale CelebFaces Attributes (CelebA-HQ) dataset, the Places2 dataset, and the Paris StreetView dataset.

[0053] Predicted completed image: refers to the image output by the image restoration model after the missing image is completed.

[0054] The solution of the embodiment of the present application includes a model training stage and an image restoration stage. Figure 1 FIG. 1 is a schematic diagram of an image restoration system according to an exemplary embodiment. Figure 1As shown, in the model training stage, the model training device 110 trains an image restoration model with higher accuracy through a pre-set sample image set. In the image restoration stage, the image restoration device 120 outputs the restored predicted completed image based on the trained image restoration model and the input defective image.

[0055] Among them, the above-mentioned model training device 110 and image restoration device 120 can be computer devices with machine learning capabilities, for example, the computer device can be a terminal or a server.

[0056] Optionally, the model training device 110 and the image restoration device 120 may be the same computer device, or the model training device 110 and the image restoration device 120 may be different computer devices. Furthermore, when the model training device 110 and the image restoration device 120 are different devices, they may be the same type of device, such as both the model training device 110 and the image restoration device 120 may be servers; or, the model training device 110 and the image restoration device 120 may be different types of devices. The server may be an independent physical server, or a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited thereto. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in this application.

[0057] Figure 2 This is a flowchart of an image restoration method according to an exemplary embodiment of the present application. Figure 1 For example, the method is performed by the image restoration device 120 shown in FIG. 1 , and includes:

[0058] Step 220: Acquire a defective image and a mask image, where the mask image is used to indicate a missing area on the defective image.

[0059] A defective image refers to an image with a defective area. Figure 3 The defective image 32 is repaired on the missing area of ​​the defective image 32 at two scales, scale 1 and scale 2, to obtain a completed image 33.

[0060] A mask image refers to an image used to block a target image. The mask image is used to control the area or process of image processing.

[0061] Exemplarily, the pixel values ​​of the pixels of the mask image include at least one of 0 to 255. Exemplarily, the pixel values ​​of the pixels on the mask image include 0 and 1, where 0 represents a valid pixel and 1 represents a missing pixel. The mask image is used to indicate the missing area on the defective image. Figure 3 . Figure 3 A mask image 31 is shown. The mask image 31 is used to indicate missing areas of the defect image 32.

[0062] The missing region refers to a defective region in an image. Optionally, the missing region can be obtained by manually marking the defective image, or by detecting the defective image using a current automatic detection algorithm.

[0063] Exemplarily, the defect image and the mask image have the same size and resolution.

[0064] Step 240: performing image restoration on the missing region of the defective image at at least two scales, and fusing the restoration results at at least two scales to obtain a completed image.

[0065] Image restoration is performed on the missing region of the defective image in parallel at at least two scales, and the restoration results at at least two scales are fused to obtain a completed image.

[0066] The at least two scales include resolution scales after upsampling or downsampling the defective image;

[0067] Scale refers to the resolution of an image. In this application, the image obtained by upsampling or downsampling a defective image has at least two resolutions, and the resulting image resolution is the scale. For example, the resolution of the image obtained by upsampling or downsampling is 256*256, 128*128, 64*64, or 32*32.

[0068] Upsampling is a sampling method that increases the resolution of the same image, and downsampling is a sampling method that decreases the resolution of the same image.

[0069] Fusion refers to combining at least two images of the same size. For example, if the pixel values ​​of the two binary-masked images are both 0 or 1, fusion refers to adding the pixel values ​​of the two images.

[0070] Step 260: output the completed image.

[0071] The completed image is the image obtained by the image restoration method of the defective image.

[0072] Exemplary references Figure 3 , Figure 3 It shows that the computer device performs image restoration on the missing area of ​​the defective image 32 at two scales, and fuses the restoration results at at least two scales to finally obtain the completed image 33.

[0073] In summary, by performing image restoration on the missing areas of the defective image at at least two scales and fusing the restoration results at the at least two scales, information loss can be better prevented, the information of the original defective image can be preserved, and the restored complement image can be made clearer and more complete.

[0074] Schematically, the above-mentioned image restoration method includes inputting the defective image into the image restoration model to obtain a completed image. Figure 4 The schematic diagram of the structure of an image restoration model 400 is shown. A computer device receives a defective image 42 as input and sends a mask image 41 and the defective image 42 to the image restoration model 400. The defective image 42 is restored by the image restoration model 400 to form a completed image 43, and the computer device outputs the completed image 43.

[0075] The image restoration model 400 includes n first network layers 420 and n second network layers 440 .

[0076] Combined with reference Figure 4 and Figure 5 , Figure 5 This is a flowchart of an image restoration method according to an exemplary embodiment of the present application. Figure 1 For example, the method is performed by the image restoration device 120 shown in FIG. 1 , and includes:

[0077] Step 502: Acquire a defective image and a mask image, where the mask image is used to indicate a missing area on the defective image.

[0078] A defective image refers to an image with defective areas.

[0079] A mask image is an image used to block out a target image. It is used to control the region or process of image processing. For example, the pixel values ​​of a mask image can only be 0 or 1, where 0 indicates a valid pixel and 1 indicates a missing pixel. The mask image is used to indicate missing regions in a defective image.

[0080] The missing region refers to a defective region in an image. Optionally, the missing region can be obtained by manually marking the defective image, or by detecting the defective image using a current automatic detection algorithm.

[0081] Exemplarily, the defect image and the mask image have the same size and resolution.

[0082] Step 504: Input the defective image and the mask image into an image restoration model, where the image restoration model includes: n first network layers and n second network layers, where n is a positive integer.

[0083] Schematically, the defective image is input to the first network layer, the first network layer outputs the repaired defective image, and the repaired defective image is input to the second network layer, and the second network layer outputs the repaired completed image.

[0084] Step 506: The defective image is scaled up layer by layer from the original scale through the n-layer first network layer, and the missing area is repaired to obtain intermediate images of n scales;

[0085] The original scale refers to the size of the defective image input to the first layer of the n-layer network. The inpainting process refers to the predicted inpainting of the missing area. The intermediate image refers to the image output after the defective image passes through the first layer of the n-layer network.

[0086] The input image at each scale of the latter first network layer in two adjacent first network layers is a fused image obtained by cross-fusion of the intermediate images output at at least two scales of the former first network layer.

[0087] Increasing the scale layer by layer means that the defective image of the original scale passes through n first network layers layer by layer, and finally obtains an intermediate image of n scales. Schematically, the original scale of the defective image is R, the scale of the intermediate image obtained after passing through the first first network layer includes 1 / 2R, and the scale of the intermediate image obtained after passing through the second first network layer includes 1 / 4R, and the scale is increased layer by layer until the nth first network layer, the intermediate scale obtained is 1 / 2 n R.

[0088] Step 508: Using n second network layers, the intermediate image is scaled down layer by layer from n scales and the missing areas are repaired to obtain a completed image of the original scale.

[0089] The input image at each scale of the latter second network layer in two adjacent second network layers is a fused image obtained by cross-fusion of the intermediate images output at at least two scales of the former second network layer.

[0090] Layer-by-layer scale reduction means that the intermediate images output by n first network layers pass through n second network layers layer by layer, and finally obtain the completed image of the original scale. Schematically, the scale of the intermediate images output by n first network layers is 1 / 2 n R, the scale of the intermediate image obtained after the first and second network layers includes 1 / 2 n R, and then the scale of the intermediate image obtained by the second network layer includes 1 / 2 n-1R, the scale is reduced layer by layer until the nth second network layer, and the original scale of the completed image is R.

[0091] Step 510: output the completed image.

[0092] In summary, the missing areas on the defective image are repaired through n-layer first network layers and n-layer second network layers, and the information of the defective image of n-layer first network layers and n-layer second network layers is finally obtained to obtain the information of the completed image, which can better prevent information loss and preserve the information of the original defective image, so that the repaired completed image is closer to the predicted completed image.

[0093] In an exemplary embodiment, the above Figure 4 The i-th first network layer in the n-layer first network layer 420 includes i scale modules, where i scale modules correspond to different scales, and i is a positive integer not greater than n. The j-th second network layer in the n-layer second network layer 440 includes n-j+1 scale modules, where n-j+1 scale modules correspond to different scales, and j is a positive integer not greater than n. Specifically:

[0094] Figure 6 The schematic diagram shows a structure of an image restoration model 600. The image restoration model 600 includes: four first network layers 620 and four second network layers 640.

[0095] The four first network layers 620 include: first network layer P11, first network layer P12, first network layer P13, and first network layer P14. First network layer P11 includes one scale module corresponding to the original scale; first network layer P12 includes two scale modules, one corresponding to the original scale and the 1 / 2 scale; first network layer P13 includes three scale modules, one corresponding to the original scale, the 1 / 2 scale, and the 1 / 4 scale; first network layer P14 includes four scale modules, one corresponding to the original scale, the 1 / 2 scale, the 1 / 4 scale, and the 1 / 8 scale.

[0096] The four second network layers 640 include: second network layer P21, second network layer P22, second network layer P23, and second network layer P24. Second network layer P21 includes one scaling module corresponding to the original scale; second network layer P22 includes two scaling modules, one corresponding to the original scale and the 1 / 2 scale; second network layer P23 includes three scaling modules, one corresponding to the original scale, the 1 / 2 scale, and the 1 / 4 scale; second network layer P24 includes four scaling modules, one corresponding to the original scale, the 1 / 2 scale, the 1 / 4 scale, and the 1 / 8 scale.

[0097] Schematically, the scale module repairs the fused image of the input intermediate image to obtain an image with the same scale as the fused image. The scale of the fused image corresponding to the first scale module of the first network layer P11 is R, and the scale of the output image is R; the scale of the fused image corresponding to the first scale module of the first network layer P12 is R, and the scale of the output image is R, the scale of the fused image corresponding to the second scale module of the first network layer P12 is 1 / 2R, and the scale of the output image is 1 / 2R; the scale of the fused image corresponding to the first scale module of the first network layer P13 is R, and the scale of the output image is R, the scale of the fused image corresponding to the second scale module of the first network layer P13 is 1 / 2R, and the scale of the output image is 1 / 2R. The scale of the fused image corresponding to the third scale module of 13 is 1 / 4R, so the scale of the output image is 1 / 4R; the scale of the fused image corresponding to the first scale module of the first network layer P14 is R, so the scale of the output image is R, the scale of the fused image corresponding to the second scale module of the first network layer P14 is 1 / 2R, so the scale of the output image is 1 / 2R, the scale of the fused image corresponding to the third scale module of the first network layer P14 is 1 / 4R, so the scale of the output image is 1 / 4R, the scale of the fused image corresponding to the fourth scale module of the first network layer P14 is 1 / 8R, so the scale of the output image is 1 / 8R.

[0098] The scale of the fused image corresponding to the first scale module of the second network layer P21 is R, so the output image scale is R, the scale of the fused image corresponding to the second scale module of the second network layer P21 is 1 / 2R, so the output image scale is 1 / 2R, the scale of the fused image corresponding to the third scale module of the second network layer P21 is 1 / 4R, so the output image scale is 1 / 4R, the scale of the fused image corresponding to the fourth scale module of the second network layer P21 is 1 / 8R, so the output image scale is 1 / 8R; the scale of the fused image corresponding to the first scale module of the second network layer P22 is R, so the output image scale is R, The scale of the fused image corresponding to the second scale module of the second network layer P22 is 1 / 2R, so the scale of the output image is 1 / 2R. The scale of the fused image corresponding to the third scale module of the second network layer P22 is 1 / 4R, so the scale of the output image is 1 / 4R. The scale of the fused image corresponding to the first scale module of the second network layer P23 is R, so the scale of the output image is R. The scale of the fused image corresponding to the second scale module of the first network layer P23 is 1 / 2R, so the scale of the output image is 1 / 2R. The scale of the fused image corresponding to the first scale module of the second network layer P24 is R, so the scale of the output image is R.

[0099] Based on Figure 6In an optional embodiment, the above step 506 may include the following sub-steps:

[0100] Step 506-1: Call the first first network layer to repair the missing area in the defective image of the original scale to obtain an intermediate image of the original scale;

[0101] Step 506-2: When i is greater than 1, the computer device extracts features of each scale of the intermediate image output by the (i-1)th first network layer into feature images of scale i;

[0102] In step 506-3, the computer device cross-fuses the feature images of the same scale into a fused image, and inputs the fused image into the scale module corresponding to the scale in the i-th first network layer for repair, thereby obtaining an intermediate image of the i-th scale.

[0103] Based on Figure 6 In an optional embodiment, the above step 508 may include the following sub-steps:

[0104] Step 508-1: Call the jth second network layer to repair the missing areas in the intermediate images of the n-j+1 scales to obtain n-j+1 intermediate images;

[0105] Step 508-2: When j is not equal to n, the computer device extracts features from the intermediate image of each scale output by the j-th second network layer into feature images of nj scales. The computer device cross-fuses the feature images of the same scale to form a fused image, which is then input into the scale module corresponding to the scale in the j+1-th second network layer for restoration, thereby obtaining intermediate images of j scales.

[0106] In step 508-3, when j is equal to n, the computer device determines the intermediate image of the original scale output by the nth second network layer as the completed image.

[0107] In the above embodiment, there are two situations in which feature images of the same scale are cross-fused:

[0108] First, the i-th first network layer includes i scale modules, and each scale module outputs i+1 feature images. When i is less than n, the fused image obtained by fusing the k-th feature image output by the m-th scale module of the i-th first network layer with the k-th feature image output by other scale modules of the i-th first network layer is input to the k-th scale module of the i+1-th first network layer, where k is a positive integer not greater than i+1 and m is a positive integer not greater than i. Therefore, this process can be expressed by the equation:

[0109]

[0110] The left side of the equation is the fusion image of the k-th scale module input to the i+1-th first network layer, and the right side of the equation is the sum of the k-th feature image output by the m-th scale module of the i-th first network layer and the k-th feature image output by other scale modules of the i-th first network layer.

[0111] It is worth noting that in the above equation Where m can refer to any number from 1 to i, that is, There is no regular addition operation in , that is, the image fusion of the k-th scale module of the i+1-th first network layer can select the k-th feature image output by several scale modules of the i-th first network layer for addition operation.

[0112] When i equals n, the first second network layer includes n scale modules, each of which outputs n intermediate images. Suppose the fused image obtained by fusing the kth intermediate image output by the mth scale module of the nth first network layer with the kth intermediate image output by other scale modules of the nth first network layer is input to the kth scale module of the first second network layer, where k is a positive integer not greater than n and m is a positive integer not greater than n. Therefore, this process can be expressed by the equation:

[0113]

[0114] Where j = 1, i = n, the left side of the equation is the fused image input to the kth scale module of the first second network layer, and the right side of the equation is the sum of the kth intermediate image output by the mth scale module of the nth first network layer and the kth intermediate images output by other scale modules of the nth first network layer.

[0115] It is worth noting that in the above equation Here, m can refer to any number from 1 to n, that is, the image fusion of the k-th scale module of the 1st second network layer can select the k-th feature image output by several scale modules of the n-th first network layer for addition operation.

[0116] Second, the jth second network layer includes n-j+1 scale modules, and each scale module outputs nj feature images. Let the fused image obtained by fusing the kth feature image output by the mth scale module of the jth second network layer with the kth feature image output by other scale modules of the jth second network layer be input to the kth scale module of the j+1th second network layer, where k is a positive integer not greater than nj, and m is a positive integer not greater than n-j+1. Therefore, this process can be expressed by the equation:

[0117]

[0118] The left side of the equation is the fusion image of the k-th scale module input to the j+1-th second network layer, and the right side of the equation is the sum of the k-th feature image output by the m-th scale module of the j-th second network layer and the k-th feature image output by other scale modules of the j-th second network layer.

[0119] It is worth noting that in the above equation Where m can refer to any number from 1 to n-j+1, that is, There is no regular addition operation in , that is, the fusion image of the k-th scale module of the j+1-th second network layer can select the k-th feature image output by several scale modules of the j-th second network layer for addition operation.

[0120] Schematic reference Figure 6 , n=4, the four scales of the input image r1 are 256*256, r2 is 128*128, r3 is 64*64 and r4 is 32*32. Figure 6 It is a schematic diagram of an image restoration model of an exemplary embodiment of the present application.

[0121] The four first network layers include P11 to P14, and the four second network layers include P21 to P24. The four resolutions of the intermediate images obtained by the fusion image through the four scaling modules are R, 1 / 2R, 1 / 4R and 1 / 8R.

[0122] In this embodiment of the present application, intensive information exchange is performed between the four first network layers. For example, the second first network layer includes two scale modules, each of which outputs three feature images. The fused image obtained by fusing the first feature image output by the first scale module of the second first network layer with the first feature image output by the other scale modules of the second first network layer is input to the first scale module of the third first network layer. This process can be expressed by the equation:

[0123]

[0124] Where i = 2, the left side of the equation is the fusion image of the first scale module input to the third first network layer, and the right side of the equation is the sum of the first feature image output by the first scale module of the second first network layer and the first feature image output by other scale modules of the second first network layer.

[0125] In this embodiment of the present application, the fourth first network layer and the first second network layer perform intensive information exchange. The fused image obtained by fusing the first feature image output by the first scale module of the fourth first network layer with the first feature image output by the other scale modules of the fourth first network layer is input to the first scale module of the first second network layer. Therefore, this process can be expressed by the equation:

[0126]

[0127] Where j = 1, i = 4, the left side of the equation is the fusion image of the first scale module input to the first second network layer, and the right side of the equation is the sum of the first feature image output by the first scale module of the fourth first network layer and the first feature image output by other scale modules of the fourth first network layer.

[0128] In this embodiment of the present application, intensive information exchange is performed between the four second network layers. For example, the second second network layer includes three scale modules, each of which outputs two feature images. The fused image obtained by fusing the first feature image output by the first scale module of the second second network layer with the first feature image output by the other scale modules of the second first network layer is input to the first scale module of the third second network layer. This process can be expressed by the equation:

[0129]

[0130] Where j = 2, the left side of the equation is the fusion image of the first scale module input to the third second network layer, and the right side of the equation is the sum of the first feature image output by the first scale module of the second second network layer and the first feature image output by other scale modules of the second second network layer.

[0131] In the figure, upsampling and downsampling are used to sample the intermediate image into the feature image.

[0132] In one embodiment, the scaling modules in the first and second network layers include at least one basic module. Illustratively, the number of basic modules included in the scaling modules of P1-8 is 1, 1, 4, 2, 1, 4, 1, and 1, respectively. Each basic module includes two layers of basic residual blocks; alternatively, each basic module includes three layers of ResNet (a type of residual network) modules.

[0133] In one embodiment, the number of channels of a module of the same scale doubles as the scale decreases. For four different scales, the corresponding numbers of channels are C, 2C, 4C, and 8C. Optionally, C=18.

[0134] The following will compare and analyze the image restoration method of the embodiment of the present application and the existing image restoration method from the perspectives of quantitative analysis and qualitative analysis.

[0135] The quantitative analysis is as follows:

[0136] Figure 7 This is a comparative diagram of an image restoration method according to an exemplary embodiment of the present invention and other image restoration methods. Figure 7The characters in are explained. l1(%) refers to the L1 norm loss, also known as the minimum absolute deviation. The lower the value, the better the image restoration method. The higher the PSNR (Peak Signal to Noise Ratio) and SSIM (Structural Similarity Index Measurement), the better the image restoration method. PEN refers to a pyramid context encoder network that fills in missing areas by gradually learning regional attention. Gonv refers to a two-stage generative network from coarse to fine, and introduces gated convolution. MEDFE refers to an inter-encoder-decoder network with feature balance for joint restoration of structure and texture. GDN refers to a generative framework based on pixel-level dense detectors for image restoration. RFR refers to progressive image restoration using recursive feature inference and knowledge-consistent attention in feature space. Ours refers to an image restoration method of an exemplary embodiment of the present application. Masks refers to the area occupied by the defective area in the defective image. Datasets refers to a sample image set, 'C' refers to the CelebA-HQ dataset, and 'P' refers to the Places2 dataset. 1%-10% indicates the percentage of the missing area to the total image area.

[0137] Comparing the values ​​of l1(%), PSNR, and SSIM, the image restoration method of an exemplary embodiment of the present application is superior to other image restoration methods.

[0138] The qualitative analysis is as follows:

[0139] Figure 8 A comparative diagram of an image restoration method according to an exemplary embodiment of the present application and other image restoration methods. Figure 8 The characters in the text are explained below. Gt refers to the ideal completed image, and Input refers to the defective image input to the image restoration model.

[0140] There are six rows from top to bottom, each group of three rows, and the two groups correspond to the Places2 and Paris StreetView datasets respectively. It can be seen from the figure that the PEN method has relatively poor visual quality. Due to the application of contextual attention, the results of the GConv method show some unreasonable repetitive patterns, such as the windows in the 4th and 6th rows. Compared with existing restoration techniques, the image restoration method of an exemplary embodiment of the present application can better restore structural and texture details. For example, the structure of the window (the second to last row) can be successfully restored. In addition, the image restoration method of an exemplary embodiment of the present application realistically restores the wood texture (the last image in the second row) based on contextual information, and also obtains a clear boundary (the last image in the third row).

[0141] Comparing the output image with the ideal completed image, the image restoration method according to an exemplary embodiment of the present application outperforms other image restoration methods.

[0142] To sum up, the image restoration method provided in this embodiment performs image restoration on the missing area of ​​the defective image through n-layer first network layers and n-layer second network layers, and fuses the information of the defective image of n-layer first network layers and n-layer second network layers to finally obtain the information of the completed image, which can better prevent information loss, preserve the information of the original defective image, and make the restored completed image closer to the predicted completed image.

[0143] The above image restoration model is trained using a training method. Figure 9 This is a flowchart of a training method for an image restoration model of an exemplary embodiment of the present application.

[0144] Step 901: Obtain a sample missing image from a sample image set. The sample missing image corresponds to a sample mask image and a sample completion image. The sample mask image is used to indicate a missing area on the sample missing image.

[0145] The sample image set provides target sample images required for training the image restoration model. In some embodiments, the sample image set includes at least one of the Large-Scale CelebFaces Attributes (CelebA-HQ) dataset, the Places2 dataset, and the Paris StreetView dataset.

[0146] The sample missing image refers to a target sample image with missing areas obtained from the sample image set.

[0147] The sample mask image consists of a binary mask M (where 0 represents a valid pixel and 1 represents a missing pixel) used to indicate the missing area.

[0148] Step 902: Input the sample missing image and the sample mask image into the image restoration model;

[0149] Step 903: Obtain the predicted completed image output by the image restoration model;

[0150] The predicted completed image refers to the image output by the image restoration model after the sample missing image is restored. The process of predicting and completing the image can refer to the contents described in the above embodiments, and will not be repeated in this embodiment.

[0151] Step 904, calculating the loss difference between the sample completed image and the predicted completed image;

[0152] Step 905: Optimize the network parameters of the image restoration model according to the loss difference.

[0153] The network parameters of the image restoration model are used to adjust the restoration performance of the image restoration model. The network parameters of the image restoration model in the exemplary embodiment of the present application have been reflected in the above embodiments, such as the number of basic modules contained in the scale module of P11-P24 and the number of channels of the scale module.

[0154] The model training device substitutes the sample-completed image and the predicted-completed image into a loss function to obtain the difference in loss. This loss function is typically a joint loss function, where commonly used terms include at least one of the following: reconstruction loss, adversarial loss, and perceptual loss. In image restoration models, the loss function design typically combines or improves upon these four common terms based on practical needs and the characteristics of the network architecture.

[0155] In one embodiment, the loss difference includes a weighted sum of a valid pixel reconstruction loss and a missing pixel reconstruction loss, wherein the valid pixel reconstruction loss is used to indicate the loss of valid pixels in the sample-completed image, and the missing pixel reconstruction loss is used to indicate the loss of missing pixels in the sample-completed image.

[0156] Schematically, the effective pixel reconstruction loss is expressed as:

[0157]

[0158] in, is the effective pixel reconstruction loss, Ι gt is the predicted completed image, Ι out is the completed image, ⊙ is the element-wise multiplication operation, and sum(Μ) is the number of non-zero elements in M.

[0159] The missing pixel reconstruction loss is expressed as:

[0160]

[0161] in, is the missing pixel reconstruction loss.

[0162] The loss function is expressed as:

[0163]

[0164] where α is the balancing factor, represents the loss function.

[0165] In this embodiment, It does not reflect the loss of high-frequency components in the missing area.

[0166] In one embodiment, the loss difference includes a weighted sum of valid pixel reconstruction loss, missing pixel reconstruction loss, and high-frequency reconstruction loss. The valid pixel reconstruction loss indicates the loss of valid pixels in the sample-completed image, the missing pixel reconstruction loss indicates the loss of missing pixels in the sample-completed image, and the high-frequency reconstruction loss indicates the loss of high-frequency components in the sample-completed image.

[0167] Among them, the calculation process of high-frequency reconstruction loss includes the following three steps:

[0168] First, calculate the fusion image of the sample completion image and the predicted completion image;

[0169] The fused image is expressed as:

[0170] Ι mer = Ι gt ⊙(1-M)+Ι out ⊙Μ;

[0171] Among them, I mer Refers to the fusion image of the sample completion image and the predicted completion image.

[0172] Second, a high-frequency residual filter is used to extract features from the sample completion image to obtain a first feature map; a high-frequency residual filter is used to extract features from the fused image to obtain a second feature map;

[0173] Use φ(Ι gt ) represents the first feature map obtained by extracting features from the sample completion image using a high-frequency residual filter. mer ) represents the second feature map obtained by extracting features from the fused image using a high-frequency residual filter.

[0174] The filter kernel weights of the high-frequency residual filter use 30 residual filters. Figure 10 Several residual filters of an exemplary embodiment of the present application are shown. Figure 11 FIG. 4 shows the FFT (Fast Fourier Transformation) visualization results of 30 residual filters of an exemplary embodiment of the present application.

[0175] From a signal processing perspective, these residual filters limit the high-frequency components of the image. From a statistical perspective of natural images, these filters have different shapes (sizes) and orientations, thereby simulating statistical information at the local patch level.

[0176] In this embodiment, color channel by color channel is used to identify mer / Ι gt Perform a convolution operation and finally obtain a tensor with 90 channels.

[0177] Third, based on the difference between the second feature map and the first feature map, a high-frequency reconstruction loss is calculated.

[0178] The high-frequency reconstruction loss can be expressed as:

[0179]

[0180] Among them, φ(·) is a feature extraction function. represents the high-frequency reconstruction loss function.

[0181] In summary, the loss function used to train the image restoration model is expressed as:

[0182]

[0183] Where α and β are balancing factors. Optional: α = 6, β = 200.

[0184] In summary, the high-frequency loss function directly focuses on the high-frequency information calculated from local image patches of varying shapes and orientations, further enhancing the naturalness and authenticity of the inpainted result. This image inpainting method utilizes an image inpainting model trained with the high-frequency loss function. The resulting inpainted image has richer high-frequency component information, better restoring the structure and texture details of the missing regions of the defective image, resulting in a clear and complete inpainted image.

[0185] The following will compare and analyze the image restoration models trained with the above two loss functions from the perspectives of quantitative analysis and qualitative analysis.

[0186] Quantitative analysis description:

[0187] Figure 12 3 is a schematic diagram comparing an image restoration method according to an exemplary embodiment of the present application and other image restoration methods. Figure 12 Figure 3 compares the performance of three image inpainting methods on the CelebA-HQ and Places2 datasets, and Figure 4 compares the performance of the three inpainting methods using inpainting models trained with and without high-frequency loss functions. PEN_HF, Gconv_HF, and Ours_HF refer to inpainting models trained with high-frequency loss functions, while PEN, Gconv, and Ours refer to inpainting models not trained with high-frequency loss functions. Figure 12 The value displayed is PSNR.

[0188] Depend on Figure 12 It can be directly seen that the image restoration method of the image restoration model trained with the high-frequency loss function is better and its PSNR value is higher.

[0189] Qualitative analysis description:

[0190] Figure 13 This is a schematic diagram comparing an image restoration method according to an exemplary embodiment of the present application with other image restoration methods. The second column shows an image restoration method that has not been trained with a high-frequency loss function. The third column shows an image restoration method that has been trained with a high-frequency loss function.

[0191] from Figure 13 It can be directly seen that the image restoration method trained with a high-frequency loss function can significantly enhance the restoration of textures, such as pillars (the second row). In addition, it can achieve clearer boundaries and more complete structures (the anchors in the last row).

[0192] Figure 14 3 is a schematic diagram comparing an image restoration method according to an exemplary embodiment of the present application and other image restoration methods. Figure 15 3 is a schematic diagram comparing an image restoration method according to an exemplary embodiment of the present application and other image restoration methods.

[0193] First of all, Figure 14 and Figure 15 Explain the characters in the image. EdNet and U-Net are commonly used image inpainting networks for image generation and inpainting. Both image inpainting networks have an "hourglass" shape. HRNet is an image inpainting network based on multiple downsampling and direct upsampling.

[0194] Figure 14 Shows PSNR comparison of four different network architectures on the CelebA-HQ and Places2 datasets.

[0195] Figure 15 The datasets used from top to bottom are Places2 and Paris StreetView.

[0196] Figure 14 and Figure 15 The four image restoration models shown are all based on Trained.

[0197] Compare Figure 14 and Figure 15 Whether analyzed from a quantitative perspective or a qualitative perspective, the image restoration method of an exemplary embodiment of the present application is superior to other image restoration methods.

[0198] The image restoration method of the embodiment of the present application is further applied to delete the object from the selected real image, and the corresponding result is as follows: Figure 16 shown. Figure 16: This is a schematic diagram of the image restoration results of an exemplary embodiment of the present application. The image restoration model of an exemplary embodiment of the present application is trained on Places2 and Paris StreetView respectively. Figure 16 As shown, the image repaired by the image repair method of this exemplary embodiment can achieve a visually realistic effect, and the shadowed tree and flagpole are successfully removed.

[0199] In one embodiment, the image inpainting method was implemented using PyTorch 1.3.1 (an open-source Python machine learning library). The GPU (Graphics Processing Unit) was an NVIDIA TITAN RTX graphics card. The image inpainting network was trained using the Adam optimizer, with β1 = 0.5 and β2 = 0.999.

[0200] The image inpainting method sets the initial value of the preset learning rate to 0.0002 in the first 100 epochs and linearly decays it to zero in the next 100 epochs.

[0201] The sizes of the defective image and mask image in this image restoration method are both 256×256.

[0202] Application scenarios:

[0203] The image restoration method of the present application can be applied to an application on a terminal. For example, through the image restoration application, missing areas in an image can be compensated, or redundant areas in an image can be removed. In one embodiment, an image restoration application is present on a smartphone. In response to a user inputting a defective image into the image restoration application, the smartphone automatically identifies the missing areas of the defective image and sends the missing areas to the backend. The backend receives the defective image and sends the restored image to the smartphone, which then displays the restored image on the smartphone interface.

[0204] Figure 17 This is a structural block diagram of an image restoration device provided by an exemplary embodiment of the present application. Figure 18 As shown, the device includes:

[0205] An acquisition module 1720 is configured to acquire a defective image and a mask image, wherein the mask image is used to indicate a missing area on the defective image;

[0206] The restoration module 1740 is configured to perform image restoration on the missing region of the defective image at at least two scales, and fuse the restoration results at the at least two scales to obtain a completed image; the at least two scales include resolution scales after upsampling or downsampling the defective image;

[0207] The output module 1760 is configured to output the completed image.

[0208] In an optional embodiment, the repair module 1740 includes: an input submodule 1741 , a first repair submodule 1742 , and a second repair submodule 1743 .

[0209] In an optional embodiment, the input submodule 1741 is configured to input the defective image and the mask image into an image restoration model, wherein the image restoration model includes: n first network layers and n second network layers, where n is a positive integer greater than 1;

[0210] In an optional embodiment, the first restoration module 1742 is configured to scale the defective image layer by layer from the original scale through the n first network layers and perform restoration processing on the missing region to obtain intermediate images of n scales;

[0211] In an optional embodiment, the second restoration module 1743 is configured to reduce the scale of the intermediate image from the n scales layer by layer through the n second network layers and perform restoration processing on the missing area to obtain a completed image of the original scale.

[0212] Among them, the input image at each scale of the latter first network layer in two adjacent first network layers is a fusion image cross-fused with the intermediate image output at at least two scales of the former first network layer; the input image at each scale of the latter second network layer in two adjacent second network layers is a fusion image cross-fused with the intermediate image output at at least two scales of the former second network layer.

[0213] In an optional embodiment, the i-th first network layer in the n-layer first network layer includes i scale modules, the i scale modules correspond to different scales, and i is a positive integer not greater than n.

[0214] In an optional embodiment, the first repair module 1742 is further configured to call the first first network layer to repair the missing region in the defective image at the original scale to obtain an intermediate image at the original scale;

[0215] In an optional embodiment, the first restoration module 1742 is further configured to, when i is greater than 1, extract features of each scale of the intermediate image output by the (i-1)th first network layer into feature images of scale i;

[0216] In an optional embodiment, the first restoration module 1742 is further used to input the fused image obtained by cross-fusion of feature images belonging to the same scale into the scale module corresponding to the scale in the i-th first network layer for restoration, so as to obtain intermediate images of i scales.

[0217] In one embodiment, the j-th second network layer in the n-layer second network layers includes n-j+1 scale modules, and the n-j+1 scale modules correspond to different scales, where j is a positive integer not greater than n.

[0218] In an optional embodiment, the second restoration module 1743 is further configured to call the jth second network layer to perform restoration processing on the missing region in the intermediate image at the n-j+1 scales, to obtain n-j+1 intermediate images;

[0219] In an optional embodiment, the second restoration module 1743 is further configured to, when j is not equal to n, extract features of each scale of the intermediate image output by the j-th second network layer into feature images of nj scales;

[0220] In an optional embodiment, the second restoration module 1743 is further configured to input a fused image obtained by cross-fusion of feature images of the same scale into a scale module corresponding to the scale in the j+1th second network layer for restoration, thereby obtaining an intermediate image of j scales.

[0221] In an optional embodiment, the second restoration module 1743 is further configured to, when j is equal to n, determine the intermediate image of the original scale output by the n-th second network layer as the completed image.

[0222] In an optional embodiment, the image restoration model is trained by a training module:

[0223] The training module is used to obtain a sample missing image from a sample image set, wherein the sample missing image corresponds to a sample mask image and a sample completion image, and the sample mask image is used to indicate a missing area on the sample missing image;

[0224] The training module is also used to input the sample missing image and the sample mask image into the image restoration model; obtain the predicted completed image output by the image restoration model; calculate the loss difference between the sample completed image and the predicted completed image; and optimize the network parameters of the image restoration model based on the loss difference.

[0225] In an optional embodiment, the loss difference includes:

[0226] The weighted sum of the valid pixel reconstruction loss and the missing pixel reconstruction loss;

[0227] Among them, the effective pixel reconstruction loss is used to indicate the loss of effective pixel points in the predicted completed image relative to the effective pixel points of the sample completed image, and the missing pixel reconstruction loss is used to indicate the loss of missing pixel points in the predicted completed image relative to the missing pixel points of the sample completed image.

[0228] In an optional embodiment, the loss difference includes:

[0229] The weighted sum of valid pixel reconstruction loss, missing pixel reconstruction loss, and high-frequency reconstruction loss;

[0230] Among them, the effective pixel reconstruction loss is used to indicate the loss of effective pixel points in the predicted completed image relative to the effective pixel points of the sample completed image, the missing pixel reconstruction loss is used to indicate the loss of missing pixel points in the predicted completed image relative to the missing pixel points of the sample completed image, and the high-frequency reconstruction loss is used to indicate the loss of high-frequency components in the predicted completed image relative to the high-frequency components of the sample completed image.

[0231] In an optional embodiment, the calculation process of the high-frequency reconstruction loss includes:

[0232] Calculating a fused image of the sample completed image and the predicted completed image;

[0233] Performing feature extraction on the sample completion image using a high-frequency residual filter to obtain a first feature map; performing feature extraction on the fused image using the high-frequency residual filter to obtain a second feature map;

[0234] The high-frequency reconstruction loss is calculated based on a difference between the second feature map and the first feature map.

[0235] It should be noted that the image restoration device provided in the above embodiment is merely an example of the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image restoration device provided in the above embodiment is based on the same concept as the image restoration method embodiment. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0236] In summary, the apparatus provided in this embodiment, by performing image restoration on the missing regions of a defective image at at least two scales and fusing the restoration results at said at least two scales, can better prevent information loss, preserve the information of the original defective image, and make the restored image clearer and more complete. The apparatus provided in this embodiment utilizes an image restoration model trained with a high-frequency loss function. The restored image obtained by this image restoration apparatus has richer high-frequency component information, better restoring the structural and texture details of the missing regions of the defective image, making the restored image clearer and more complete.

[0237] Figure 18 The following is a block diagram of an electronic device 1800 according to an exemplary embodiment of the present application. The electronic device 1800 may be a portable mobile terminal, such as a smartphone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, or a desktop computer. The electronic device 1800 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other similar names.

[0238] Typically, the electronic device 1800 includes a processor 1801 and a memory 1802 .

[0239] The processor 1801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 1801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 1801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0240] Memory 1802 may include one or more computer-readable storage media, which may be non-transitory. Memory 1802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in memory 1802 is used to store at least one instruction, which is executed by processor 1801 to implement the image restoration method provided in the method embodiments of this application.

[0241] In some embodiments, electronic device 1800 may optionally include a peripheral device interface 1803 and at least one peripheral device. The processor 1801, memory 1802, and peripheral device interface 1803 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 1803 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 1804, a display screen 1805, a camera assembly 1806, an audio circuit 1807, and a power supply 1808.

[0242] The peripheral device interface 1803 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 1801 and the memory 1802. In some embodiments, the processor 1801, the memory 1802, and the peripheral device interface 1803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1801, the memory 1802, and the peripheral device interface 1803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0243] The RF circuit 1804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1804 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 1804 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 1804 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1804 may also include circuitry related to Near Field Communication (NFC), although this application does not limit this.

[0244] The display screen 1805 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1805 is a touch screen display, the display screen 1805 also has the ability to collect touch signals on the surface or above the surface of the display screen 1805. The touch signal can be input as a control signal to the processor 1801 for processing. At this time, the display screen 1805 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, there can be one display screen 1805, which is set on the front panel of the electronic device 1800; in other embodiments, there can be at least two display screens 1805, which are respectively set on different surfaces of the electronic device 1800 or in a folding design; in other embodiments, the display screen 1805 can be a flexible display screen, which is set on the curved surface or folding surface of the electronic device 1800. Even the display screen 1805 can be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1805 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0245] The camera assembly 1806 is used to capture images or videos. Optionally, the camera assembly 1806 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1806 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0246] The audio circuit 1807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into the processor 1801 for processing, or input into the radio frequency circuit 1804 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there can be multiple microphones, each located in different parts of the electronic device 1800. The microphone can also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signal from the processor 1801 or the radio frequency circuit 1804 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as distance measurement. In some embodiments, the audio circuit 1807 may also include a headphone jack.

[0247] Power supply 1808 is used to power the various components of electronic device 1800. Power supply 1808 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1808 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0248] In some embodiments, the electronic device 1800 further includes one or more sensors 1809 , including but not limited to an acceleration sensor 1810 , a gyroscope sensor 1811 , a pressure sensor 1812 , an optical sensor 1813 , and a proximity sensor 1814 .

[0249] The accelerometer 1810 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the electronic device 1800. For example, the accelerometer 1810 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 1801 can control the display screen 1805 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 1810. The accelerometer 1810 can also be used to collect game or user motion data.

[0250] The gyroscope sensor 1811 can detect the orientation and rotation angle of the electronic device 1800. It can also work with the accelerometer 1810 to collect 3D motions of the user on the electronic device 1800. Based on the data collected by the gyroscope sensor 1811, the processor 1801 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0251] The pressure sensor 1812 can be set on the side frame of the electronic device 1800 and / or the lower layer of the display screen 1805. When the pressure sensor 1812 is set on the side frame of the electronic device 1800, it can detect the user's grip signal of the electronic device 1800, and the processor 1801 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 1812. When the pressure sensor 1812 is set on the lower layer of the display screen 1805, the processor 1801 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 1805. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0252] Optical sensor 1813 is used to detect ambient light intensity. In one embodiment, processor 1801 can control the display brightness of display screen 1805 based on the ambient light intensity detected by optical sensor 1813. Specifically, when the ambient light intensity is high, the display brightness of display screen 1805 is increased; when the ambient light intensity is low, the display brightness of display screen 1805 is decreased. In another embodiment, processor 1801 can also dynamically adjust the shooting parameters of camera assembly 1806 based on the ambient light intensity detected by optical sensor 1813.

[0253] Proximity sensor 1814, also known as a distance sensor, is typically located on the front panel of electronic device 1800. Proximity sensor 1814 is used to detect the distance between the user and the front of electronic device 1800. In one embodiment, when proximity sensor 1814 detects that the distance between the user and the front of electronic device 1800 is gradually decreasing, processor 1801 controls display screen 1805 to switch from the screen-on state to the screen-off state. When proximity sensor 1814 detects that the distance between the user and the front of electronic device 1800 is gradually increasing, processor 1801 controls display screen 1805 to switch from the screen-off state to the screen-on state.

[0254] Those skilled in the art will understand that Figure 18 The structure shown in the figure does not constitute a limitation on the electronic device 1800, and the electronic device 1800 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0255] The present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set. The at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the image restoration method provided by the above method embodiment.

[0256] The present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the image restoration method provided by the above-mentioned method embodiment.

[0257] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0258] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0259] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An image restoration method, characterized in that: The method comprises: Acquire a defective image and a mask image, wherein the mask image is used to indicate a missing area on the defective image; Inputting the defective image and the mask image into an image restoration model, wherein the image restoration model comprises: n first network layers and n second network layers, where n is a positive integer greater than 1; The defective image is scaled up layer by layer from the original scale through the n-layer first network layer, and the missing area is repaired to obtain intermediate images of n scales; The intermediate image is scaled down layer by layer from the n scales and the missing area is repaired through the n layers of second network layers to obtain a completed image of the original scale; wherein the input image at each scale of the latter first network layer in two adjacent first network layers is a fused image obtained by cross-fusion of the intermediate images output at at least two scales of the former first network layer; the input image at each scale of the latter second network layer in two adjacent second network layers is a fused image obtained by cross-fusion of the intermediate images output at at least two scales of the former second network layer; The completed image is output.

2. The method according to claim 1, characterized in that The i-th first network layer in the n-layer first network layer includes i scale modules, the i scale modules correspond to different scales, and i is a positive integer not greater than n; The step of increasing the scale of the defective image layer by layer from the original scale and repairing the missing area through the n-layer first network layer to obtain intermediate images of n scales includes: Calling the first network layer to repair the missing area in the defective image of the original scale to obtain an intermediate image of the original scale; When i is greater than 1, the intermediate image of each scale output by the i-1th first network layer is feature extracted into a feature image of scale i; The fused image obtained by cross-fusion of feature images belonging to the same scale is input into the scale module corresponding to the current scale in the i-th first network layer for repair, and the intermediate image of the i-scale is obtained.

3. The method according to claim 1, characterized in that The j-th second network layer in the n-layer second network layers includes n-j+1 scale modules, wherein the n-j+1 scale modules correspond to different scales, and j is a positive integer not greater than n; The step of reducing the scale of the intermediate image from the n scales layer by layer through the n second network layers and repairing the missing area to obtain the completed image of the original scale includes: Calling the j-th second network layer to repair the missing areas in the intermediate images of the n-j+1 scales to obtain n-j+1 intermediate images; When j is not equal to n, the intermediate image of each scale output by the j-th second network layer is feature extracted into feature images of nj scales; The fused image obtained by cross-fusion of feature images of the same scale is input into the scale module corresponding to the current scale in the j+1th second network layer for repair, and the intermediate image of j scales is obtained; When j is equal to n, the intermediate image of the original scale output by the nth second network layer is determined as the completed image.

4. The method according to claim 2 or 3, characterized in that The scale module includes at least one basic module; The basic module includes two layers of basic residual blocks; or, the basic module includes a three-layer Resnet module.

5. The method according to any one of claims 1 to 3, characterized in that: The image restoration model is trained using the following training method: Acquire a sample missing image from a sample image set, wherein the sample missing image corresponds to a sample mask image and a sample completion image, and the sample mask image is used to indicate a missing area on the sample missing image; Inputting the sample missing image and the sample mask image into the image restoration model; Obtaining a predicted completed image output by the image restoration model; Calculating a loss difference between the sample completed image and the predicted completed image; According to the loss difference, the network parameters of the image restoration model are optimized.

6. The method according to claim 5, characterized in that The loss difference includes: The weighted sum of the valid pixel reconstruction loss and the missing pixel reconstruction loss; Among them, the effective pixel reconstruction loss is used to indicate the loss of effective pixel points in the predicted completed image relative to the effective pixel points of the sample completed image, and the missing pixel reconstruction loss is used to indicate the loss of missing pixel points in the predicted completed image relative to the missing pixel points of the sample completed image.

7. The method according to claim 5, characterized in that The loss difference includes: The weighted sum of valid pixel reconstruction loss, missing pixel reconstruction loss, and high-frequency reconstruction loss; Among them, the effective pixel reconstruction loss is used to indicate the loss of effective pixel points in the predicted completed image relative to the effective pixel points of the sample completed image, the missing pixel reconstruction loss is used to indicate the loss of missing pixel points in the predicted completed image relative to the missing pixel points of the sample completed image, and the high-frequency reconstruction loss is used to indicate the loss of high-frequency components in the predicted completed image relative to the high-frequency components of the sample completed image.

8. The method according to claim 7, characterized in that The calculating the loss difference between the sample completed image and the predicted completed image includes: Calculating a fused image of the sample completed image and the predicted completed image; Performing feature extraction on the sample completion image using a high-frequency residual filter to obtain a first feature map; performing feature extraction on the fused image using the high-frequency residual filter to obtain a second feature map; The high-frequency reconstruction loss is calculated based on a difference between the second feature map and the first feature map.

9. An image restoration device, characterized in that: The device comprises: an acquisition module, configured to acquire a defective image and a mask image, wherein the mask image is used to indicate a missing area on the defective image; An input submodule, configured to input the defective image and the mask image into an image restoration model, wherein the image restoration model comprises: n first network layers and n second network layers, where n is a positive integer greater than 1; A first restoration submodule is configured to increase the scale of the defective image layer by layer from the original scale through the n first network layers and to restore the missing area to obtain intermediate images of n scales; A second restoration submodule is configured to reduce the scale of the intermediate image layer by layer from the n scales through the n second network layers and to restore the missing area to obtain a completed image at the original scale; An output module, configured to output the completed image; Among them, the input image at each scale of the latter first network layer in two adjacent first network layers is a fusion image cross-fused with the intermediate image output at at least two scales of the former first network layer; the input image at each scale of the latter second network layer in two adjacent second network layers is a fusion image cross-fused with the intermediate image output at at least two scales of the former second network layer.

10. The device according to claim 9, characterized in that The i-th first network layer in the n-layer first network layer includes i scale modules, the i scale modules correspond to different scales, and i is a positive integer not greater than n; The first repair submodule is used to call the first first network layer to repair the missing area in the defective image of the original scale to obtain an intermediate image of the original scale; when i is greater than 1, the intermediate image of each scale output by the i-1th first network layer is feature extracted into a feature image of i scales; the fused image obtained by cross-fusion of the feature images belonging to the same scale is input into the scale module corresponding to the current scale in the i-th first network layer for repair to obtain an intermediate image of i scales.

11. The device according to claim 9, characterized in that The j-th second network layer in the n-layer second network layers includes n-j+1 scale modules, wherein the n-j+1 scale modules correspond to different scales, and j is a positive integer not greater than n; The second restoration submodule is configured to call the jth second network layer to perform restoration processing on the missing areas in the intermediate images of the n-j+1 scales, thereby obtaining n-j+1 intermediate images; when j is not equal to n, extract features of each scale of the intermediate images output by the jth second network layer into feature images of nj scales; cross-fuse the feature images of the same scale to obtain a fused image, which is input into the scale module corresponding to the current scale in the j+1th second network layer for restoration, thereby obtaining intermediate images of the j scale; When j is equal to n, the intermediate image of the original scale output by the nth second network layer is determined as the completed image.

12. A computer device, characterized in that: The computer device includes: a processor and a memory, the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the image restoration method according to any one of claims 1 to 8.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is loaded and executed by a processor to implement the image restoration method according to any one of claims 1 to 8.

14. A computer program product, characterized in that The computer program product includes computer instructions, which are loaded and executed by a processor to implement the image restoration method according to any one of claims 1 to 8.

Images