Image inpainting method, terminal and computer storage medium

Abstract

The embodiment of the application discloses an image repairing method, a terminal determines a target region in an image to be repaired, and generates a first image and a second image corresponding to the image to be repaired based on the target region; and based on a full-scale convolutional neural network, the image repairing processing is performed according to the first image and the second image, so that a repaired image corresponding to the image to be repaired is obtained.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an image restoration method, a terminal, and a computer storage medium. Background Technology

[0002] With the rapid development of mobile terminal device technology and image and video media social networks, people have an increasing demand for image and video editing technologies. Image restoration technology is one of the technologies with important application value and research prospects.

[0003] Currently, among the various solutions for high-definition image restoration, the approach of first restoring the downsampled image and then upsampling it has high computational complexity and introduces significant structural distortion, resulting in poor restoration performance. Restoration solutions that introduce network structures with a larger sensing range either still have a limited sensing range or introduce structural distortion during computation, both of which affect the restoration outcome.

[0004] It is evident that current common image restoration methods all suffer from poor restoration results. Summary of the Invention

[0005] This application provides an image restoration method, a terminal, and a computer storage medium, which can improve the restoration effect of image restoration.

[0006] The technical solution of this application embodiment is implemented as follows:

[0007] In a first aspect, embodiments of this application provide an image restoration method, the method comprising:

[0008] A target region is determined in the image to be repaired, and a first image and a second image corresponding to the image to be repaired are generated based on the target region;

[0009] Based on a full-scale convolutional neural network, image inpainting processing is performed on the first image and the second image to obtain the inpainted image corresponding to the image to be inpainted.

[0010] Secondly, embodiments of this application provide a terminal, the terminal comprising: a determining unit, a generating unit, and a repairing unit.

[0011] The determining unit is used to determine the target region in the image to be repaired;

[0012] The generation unit is used to generate a first image and a second image corresponding to the image to be repaired based on the target region.

[0013] The repair unit is used to perform image repair processing based on the first image and the second image using a full-scale convolutional neural network to obtain the repaired image corresponding to the image to be repaired.

[0014] Thirdly, this application provides a terminal, which includes a processor and a memory storing processor-executable instructions. When the instructions are executed by the processor, the image restoration method described above is implemented.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a program applied in a terminal, wherein when the program is executed by a processor, it implements the image restoration method described above.

[0016] This application provides an image restoration method, a terminal, and a computer storage medium. The terminal determines a target region in the image to be restored and generates a first image and a second image corresponding to the image to be restored based on the target region. Based on a full-scale convolutional neural network, image restoration processing is performed on the first image and the second image to obtain a restored image corresponding to the image to be restored. Therefore, in this application's embodiment, the terminal can generate a first image (hole image) and a second image (mask image) of the image to be restored based on the target region, and then use a full-scale convolutional neural network to combine the first image and the second image for image restoration processing. Specifically, by using a full-scale convolutional neural network, multiple levels and multiple perceptual scales of convolutional operations are introduced during image restoration, which can effectively model the local and global contextual relationships between the hole region and the known region, resulting in better restoration operations and thus improving the restoration effect. Attached Figure Description

[0017] Figure 1 A schematic diagram of the implementation process of the image restoration method. Figure 1 ;

[0018] Figure 2 A schematic diagram of the target area;

[0019] Figure 3 A schematic diagram of the implementation process of the image restoration method. Figure 2 ;

[0020] Figures 4a-4d For comparison purposes Figure 1 ;

[0021] Figures 5a-5d For comparison purposes Figure 2 ;

[0022] Figure 6 This is a schematic diagram illustrating the implementation principle of the image restoration proposed in the embodiments of this application. Figure 1 ;

[0023] Figure 7 This is a schematic diagram illustrating the implementation principle of the image restoration proposed in the embodiments of this application. Figure 2 ;

[0024] Figure 8 This is a schematic diagram of the structure of a full-scale convolutional module;

[0025] Figure 9 This is a schematic diagram of the structure of a global-scale convolutional network;

[0026] Figure 10 This is a schematic diagram illustrating the training method for an image restoration model.

[0027] Figure 11 Diagram of the terminal's structural composition Figure 1 ;

[0028] Figure 12 Diagram of the terminal's structural composition Figure 2 . Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It is understood that the specific embodiments described herein are merely for explaining the relevant application and not for limiting the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the application are shown in the accompanying drawings.

[0030] With the rapid development of mobile terminal device technology and image / video media social networks, the demand for image and video editing technologies is increasing. Image restoration technology is one of the technologies with significant application value and research prospects. Applications of image restoration technology include removing unwanted objects from user-captured images and restoring old video data.

[0031] The development of deep learning technology has greatly promoted the progress of image restoration technology; however, high-resolution image restoration technology still has considerable room for development. Currently, the mainstream high-resolution image restoration technology mainly involves first downsampling the high-resolution image to be restored, then using a convolutional neural network for restoration, and finally upsampling the restored image in some way. For example, one restoration method uses a pyramid-shaped restoration structure, first restoring the image downsampled by 32 times, then enlarging it and using it together with the image downsampled by 16 times as input for restoration, and so on, until the original size image is restored. Another restoration method directly inputs the downsampled and restored image into a super-resolution neural network for upsampling. Yet another restoration method introduces similar blocks as input during upsampling, calculating the relationship between the known regions and the hole regions of the restored downsampled image, introducing the features of the known regions of the high-resolution image into the hole regions to improve texture details. Another restoration method, building on the former, proposes a residual aggregation-based restoration network (High-resolution image fill network, HiFill), which eliminates the upsampling network calculation and directly adds the residuals of the known regions of the high-resolution image to the hole regions to improve texture details.

[0032] In addition, another type of technology addresses the problem of excessively large hole regions in high-resolution images by introducing network structures with a larger receptive range. Among these, an Aggregated Contextual Transformations GAN (AOTGAN) is proposed, which simultaneously introduces dilated convolutions of different sizes within a single module, thereby increasing its receptive range. Furthermore, the receptive range can be further enhanced by introducing a transformer architecture with self-attention capabilities. A transformer is a self-attention mechanism capable of calculating the direct relationships between all blocks in an image, possessing global receptive power.

[0033] Currently, among the aforementioned schemes for high-resolution image restoration, the approach of first restoring the downsampled image and then upsampling it has two drawbacks: firstly, the computational complexity is high due to the presence of multiple networks; secondly, directly fusing similar blocks for upsampling introduces significant structural distortion, resulting in poor restoration performance. While restoration schemes incorporating network structures with larger receptive ranges still suffer from limitations, the receptive range of aggregation context transform networks remains insufficient relative to the size of high-resolution images, leading to poor restoration results. Transformer-based image restoration segments the image into blocks for computation, but the random shapes of holes in the image introduce structural distortion during computation, also affecting restoration performance.

[0034] It is evident that current common image restoration methods all suffer from poor restoration results.

[0035] To address the aforementioned shortcomings, the terminal identifies a target region in the image to be repaired and generates a first image and a second image corresponding to the image to be repaired based on the target region. Then, using a full-scale convolutional neural network, image inpainting processing is performed on the first and second images to obtain the repaired image. Therefore, in the embodiments of this application, the terminal can generate a first image (hole image) and a second image (mask image) of the image to be repaired based on the target region, and then use a full-scale convolutional neural network to combine the first and second images for image inpainting processing. Specifically, by using a full-scale convolutional neural network, multiple levels and multiple perceptual scales of convolutional operations are introduced during image inpainting, which can effectively model the local and global contextual relationships between the hole region and the known region, resulting in better inpainting operations and thus improving the image inpainting effect.

[0036] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0037] One embodiment of this application provides an image restoration method. Figure 1 A schematic diagram of the implementation process of the image restoration method. Figure 1 ,like Figure 1 As shown in the embodiments of this application, the method for performing image restoration processing by the terminal may include the following steps:

[0038] Step 101: Determine the target region in the image to be repaired, and generate a first image and a second image corresponding to the image to be repaired based on the target region.

[0039] In the embodiments of this application, the terminal can first determine the target region in the image to be repaired, and then further generate a first image and a second image corresponding to the image to be repaired based on the target region.

[0040] It should be noted that, in the embodiments of this application, the terminal can be any terminal with storage function, such as: tablet computer, mobile phone, e-reader, remote control, personal computer (PC), laptop computer, in-vehicle equipment, smart TV, wearable device, personal digital assistant (PDA), portable media player (PMP), navigation device, etc.

[0041] Furthermore, in the embodiments of this application, the image to be repaired can be an image pre-stored by the terminal, for example, the image to be repaired can be an image in the mobile phone's photo album; the image to be repaired can also be an image captured by the terminal in real time, for example, the image to be repaired can be a preview image obtained by the mobile phone; the image to be repaired can also be a frame of a video recorded by the terminal in real time, for example, the image to be repaired can be a frame of a surveillance video recorded by a monitoring device.

[0042] In other words, in the embodiments of this application, the image to be repaired can be pre-stored by the terminal, or it can be collected by the terminal in real time, or it can be sent by the terminal to other devices. This application does not make any specific limitations.

[0043] Furthermore, in embodiments of this application, the target region can be a portion of the image to be repaired, wherein the target region can be a region corresponding to one or more target objects in the image to be repaired. For example, Figure 2 A schematic diagram of the target area, such as Figure 2 As shown, in the image to be repaired, the region a corresponding to the target object A is the target region.

[0044] It is understood that, in the embodiments of this application, the terminal can determine the target region in the image to be repaired in various ways. Specifically, the terminal can identify the target object in the image to be repaired through machine learning, and then determine the region corresponding to the target as the target region; the terminal can also determine the target object in the image to be repaired through a received selection instruction, and then determine the region corresponding to the target as the target region; the terminal can also directly determine the target region in the image to be repaired through a received selection instruction. This application does not impose specific limitations.

[0045] Furthermore, in the embodiments of this application, after the target region is determined in the image to be repaired, the terminal can generate a first image and a second image corresponding to the image to be repaired based on the target region. The first image can be a hole image after removing the target object from the image to be repaired, while the second image can be a mask image generated for the target region corresponding to the target object.

[0046] It should be noted that, in the embodiments of this application, when determining the first image corresponding to the image to be repaired based on the target region, the terminal can remove pixels of the target region from the image to be repaired, thereby generating the first image; and / or, the terminal can set the pixel value corresponding to the target region in the image to be repaired to a first value, thereby generating the first image. The first value can be any value in the range [0, 255].

[0047] In other words, in the embodiments of this application, after the terminal deletes or obscures the target object in the image to be repaired, it can remove the original pixel values ​​in the target area of ​​the image to be repaired, thereby obtaining a first image corresponding to the image to be repaired, in which the target object has been removed.

[0048] It should be noted that, in the embodiments of this application, when determining the second image corresponding to the image to be repaired based on the target region, the terminal can set the pixel value corresponding to the target region in the image to be repaired as a second value, and simultaneously set the pixel values ​​corresponding to other regions in the image to be repaired that are outside the target region as a third value, thereby generating the second image. The second value can be 255, and the third value can be 0.

[0049] In other words, in the embodiments of this application, the second image can be a mask image corresponding to the image to be repaired. In the second image, the pixel value of the target area corresponding to the target object is 255, which is displayed as white, and the pixel value of the remaining areas outside the target area is 0, which is displayed as black. That is, the second image is a black and white image. The resolution of the second image and the image to be repaired can be the same.

[0050] Step 102: Based on a full-scale convolutional neural network, perform image inpainting processing on the first image and the second image to obtain the inpainted image corresponding to the image to be inpainted.

[0051] In the embodiments of this application, after the terminal determines the target region in the image to be repaired and generates a first image and a second image corresponding to the image to be repaired based on the target region, it can further perform image repair processing based on the first image and the second image using a full-scale convolutional neural network, thereby generating a repaired image corresponding to the image to be repaired.

[0052] Furthermore, in the embodiments of this application, the full-scale convolutional neural network can be used to implement convolution operations at multiple levels and multiple perceptual scales, thereby effectively modeling the local and global contextual relationships of the hole region and the known region, and performing better repair operations.

[0053] It should be noted that, in the embodiments of this application, when performing image inpainting processing based on the first image and the second image to obtain the repaired image corresponding to the image to be repaired, the terminal can first generate a first repair feature map based on the first image; then the first image, the second image and the first repair map can be input into the full-scale convolutional neural network to generate the repaired image corresponding to the image to be repaired.

[0054] In other words, in the embodiments of this application, the process of repairing the image to be repaired may include two branches. The first branch is to generate a first repair feature map based on the first image, and the second branch is to use a full-scale convolutional neural network to combine the first image, the second image and the first repair feature map generated by the first branch to perform repair processing, and finally generate the repaired image.

[0055] It is understood that, in the embodiments of this application, when generating the first repair feature map based on the first image, the terminal can first perform reduction processing and smoothing processing on the first image to obtain the reduced smooth map corresponding to the first image; then the reduced smooth map can be input into the preset repair network, and finally the first repair feature map can be output.

[0056] In other words, in the embodiments of this application, the first repair feature map can be a feature image obtained by repairing the reduced smoothed map after the reduction smoothing process.

[0057] It should be noted that, in the embodiments of this application, the terminal can reduce the size of the first image according to a first preset scale. The first preset scale may include a first height scale and a first width scale, which can be any value less than 1, and the first height scale and the first width scale can be the same or different.

[0058] For example, in an embodiment of this application, it is assumed that the size of the image to be repaired is (H, W), that is, the height is H and the width is W. After being reduced by a first preset scale of 1 / 8 for both the first height scale and the first width scale, the size of the reduced image is (1 / 8 × H, 1 / 8 × W).

[0059] It is understood that, in the embodiments of this application, the terminal can perform smoothing processing on the reduced image, so that the image maintains good structural information while smoothing the texture details, and finally obtains the corresponding reduced smooth image.

[0060] It should be noted that, in the embodiments of this application, the terminal can use various methods to reduce the size of the image to be repaired, including but not limited to local mean interpolation, nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc. The terminal can use various methods to smooth the reduced image, including but not limited to L0 norm smoothing algorithms, smoothing filtering algorithms, etc.

[0061] Furthermore, in the embodiments of this application, after completing the reduction and smoothing processing of the image to be repaired to obtain a reduced smooth map, the reduced smooth map can be repaired by a preset repair network to generate a first repair feature map. The preset repair network may include a downsampling convolutional network, and / or a residual convolutional network, and / or an upsampling deconvolutional network, and / or a dilated convolutional network, and / or a full-scale convolutional network.

[0062] For example, in an embodiment of this application, the preset repair network may include a three-layer downsampling convolutional module (downsampling convolutional network), a six-layer residual convolutional module (residual convolutional network), and a three-layer upsampling deconvolutional module (upsampling deconvolutional network). The feature map output by the three-layer downsampling convolutional module is 1 / 4 × 1 / 4 the size of the input image, with 256 channels. The processing of the six-layer residual convolutional module maintains the size of the feature map. The feature map output by the three-layer upsampling deconvolutional module is the same size as the input image, with 256 channels.

[0063] It should be noted that, in the embodiments of this application, the full-scale convolutional neural network may include an encoding network, a fusion network, a repair network, and a decoding network.

[0064] Furthermore, in the embodiments of this application, in the process of inputting the first image, the second image, and the first repaired image into a full-scale convolutional neural network to generate the repaired image, the first image and the second image can first be input into the encoding network to generate the first feature map; then the first repaired feature map and the first feature map can be fused based on the fusion network to generate the second feature map; then the second feature map can be repaired based on the repair network to generate the second repaired feature map; finally, the second repaired feature map can be input into the decoding network to generate the repaired image.

[0065] It should be noted that, in the embodiments of this application, the terminal can first concatenate the first image and the second image, and then input the concatenated image into the encoding network to output the first feature map. The encoding network can be used to perform reduction processing and feature extraction processing on the input concatenated image, wherein the scale of the reduction processing can be the aforementioned first preset scale, and correspondingly, the size of the first repaired feature map and the first feature map can be the same.

[0066] On the other hand, in the embodiments of this application, the number of channels of the first feature map can be the sum of the number of channels of the first image and the second image, that is, the number of channels of the first feature map can be twice the number of channels of the original image to be repaired. For example, assuming that the number of channels of the image to be repaired is 256, then the corresponding number of channels of the first image and the second image is both 256, the number of channels of the first repair feature map generated based on the first image is 256, and the number of channels of the first feature map generated after concatenation processing is 512.

[0067] For example, in an embodiment of this application, the terminal can first concatenate a first image (hole image) and a second image (mask image) with a size of (H, W) and a number of channels of 256, and then input the concatenated image into the encoding network to obtain a first feature map with a size of (1 / 8×H, 1 / 8×W) and a number of channels of 512.

[0068] For example, in an embodiment of this application, the encoding network may include a four-layer convolutional module (convolutional network), the size of the output first feature map is 1 / 8 × 1 / 8 of the input image, and the number of channels may be the sum of the number of channels of the input image.

[0069] Furthermore, in the embodiments of this application, after obtaining the first feature map through the encoding network, the terminal can perform fusion processing on the first repaired feature map and the first feature map based on the fusion network, thereby generating a second feature map. The fusion network can be used for feature fusion processing. The terminal can first concatenate the first repaired feature map and the second feature map, and then input the concatenated image into the fusion network to output the concatenated fused second feature map.

[0070] It should be noted that, in the embodiments of this application, the size and number of channels of the first feature map and the second feature map can be the same.

[0071] For example, in an embodiment of this application, after concatenating a first repair feature map of size (1 / 8×H, 1 / 8×W) with 256 channels and a first feature map of size (1 / 8×H, 1 / 8×W) with 512 channels, the first feature map is input into a fusion network for feature fusion processing, and finally a second feature map of size (1 / 8×H, 1 / 8×W) with 512 channels can be output.

[0072] For example, in an embodiment of this application, the fusion network may include a convolutional module (convolutional network) that concatenates the first repair feature map output by the preset repair network in the first branch and the first feature map output by the encoding network in the second branch as the input of the fusion network. Assuming the size of the image to be repaired is (H, W), and the first height scale and the first width scale in the first preset scale are both 1 / 8, then the size of the input first repair feature map and the first feature map are both (1 / 8×H, 1 / 8×W). Assuming the number of channels of the first repair feature map output by the preset repair network is 256 and the number of channels of the first feature map is 512, that is, the number of channels of the image input to the fusion network after concatenation is 768, and the number of channels of the output second feature map is 512, and the size is (1 / 8×H, 1 / 8×W).

[0073] Furthermore, in embodiments of this application, the repair network may include a first-scale convolutional network, a second-scale convolutional network, and a global-scale convolutional network. The first-scale convolutional network can be used to perform smaller-scale convolutional operations, the second-scale convolutional network can be used to perform larger-scale convolutional operations, and the global-scale convolutional network can be used to perform convolutional operations in the frequency domain after transforming the spatial domain to the frequency domain.

[0074] It is understood that, in the embodiments of this application, when performing convolution operations, the scales corresponding to the first-scale convolutional network, the second-scale convolutional network, and the global-scale convolutional network increase sequentially.

[0075] It should be noted that, in the embodiments of this application, for the repair network, the terminal can pre-set at least one first-scale convolutional network, at least one second-scale convolutional network, and at least one global-scale convolutional network. That is, the number of the first-scale convolutional network, the second-scale convolutional network, and the global-scale convolutional network can be set to any size, and this application does not impose any specific limitations.

[0076] Furthermore, in the embodiments of this application, the first-scale convolutional network and the second-scale convolutional network have the same convolutional kernel; the different networks in the first-scale convolutional network have different sensing ranges and dilation coefficients; the different networks in the second-scale convolutional network have different sensing ranges and dilation coefficients.

[0077] In other words, in the embodiments of this application, different first-scale convolutional networks have different sensing ranges and dilation coefficients, and different second-scale convolutional networks also have different sensing ranges and dilation coefficients. For example, the repair network includes two first-scale convolutional networks, Network 1 and Network 2, two second-scale convolutional networks, Network 3 and Network 4, and a global-scale convolutional network, Network 5. Network 1 can be used to perform small-scale convolutional operations with a kernel size of 3×3 and a sensing range of 3×3; Network 2 can be used to perform small-scale convolutional operations with a kernel size of 3×3, a dilation coefficient of 2, and a sensing range of 5×5; Network 3 can be used to perform medium-scale convolutional operations with a kernel size of 3×3, a dilation coefficient of 4, and a sensing range of 9×9; Network 4 can be used to perform medium-scale convolutional operations with a kernel size of 3×3, a dilation coefficient of 8, and a sensing range of 17×17; and Network 5 can be used to transform the spatial domain to the frequency domain and perform convolutional operations in the frequency domain, thus having a global sensing range.

[0078] Furthermore, in the embodiments of this application, when repairing the second feature map based on the repair network to generate the second repaired feature map, the terminal can first group the feature information of the second feature map according to the number of channels of the second feature map to obtain multiple feature groups; then, the feature information of the second feature map can be input into the first-scale convolutional network, the second-scale convolutional network and the global-scale convolutional network according to the multiple feature groups, so as to generate multiple sets of repaired feature information respectively; finally, the multiple sets of repaired feature information can be fused to further generate the second repaired feature map.

[0079] It should be noted that, in the embodiments of this application, when the terminal groups the feature information of the second feature map according to the number of channels of the second feature map, it can use any grouping rule to group the feature information. The number of channels in each feature group of the multiple feature groups obtained can be the same or different. This application does not impose any specific limitations.

[0080] For example, in an embodiment of this application, assuming the number of channels in the second feature map input to the repair network is 512, the terminal can divide the feature information of the second feature map into 16 parts and 5 feature groups according to the number of channels, with 32 channels in each part. The first four feature groups, namely feature group 1, feature group 2, feature group 3, and feature group 4, each include 3 parts of channels, that is, each includes 96 channels, while the fifth feature group, namely feature group 2, includes 4 parts of channels, that is, includes 128 channels.

[0081] For example, in an embodiment of this application, the repair network may include an eight-layer full-scale convolutional module (full-scale convolutional network). For the input feature map, such as the second feature map, the feature information of the second feature map is first divided into 16 equal parts according to the number of channels of the second feature map and then divided into 5 feature groups. Each of the first four feature groups includes 3 channels, and the fifth feature group includes 4 channels. The first feature group, such as feature group 1, can be input into a first-scale convolutional network, such as network 1, to perform a small-scale convolution operation with a kernel size of 3×3 and a receptive range of 3×3, outputting a set of repaired feature information. The second feature group, such as feature group 2, can be input into another first-scale convolutional network, such as network 2, to perform a small-scale convolution operation with a kernel size of 3×3, a dilation factor of 2, and a receptive range of 5×5, outputting a set of repaired feature information. The third feature group, such as feature group 3, can be input into a second-scale convolutional network, such as network 3, to perform a convolution operation with a kernel size of 3×3... The first feature group, such as feature group 4, can be input into another second-scale convolutional network, such as network 4, where a medium-scale convolutional operation with a kernel size of 3×3, a dilation factor of 8, and a receptive range of 17×17 is performed, outputting a set of repaired feature information. The second feature group, such as feature group 5, can be input into a global-scale convolutional network, such as network 5, which transforms the spatial domain to the frequency domain, performs convolutional operations in the frequency domain, has a global receptive range, and outputs a set of repaired feature information.

[0082] It should be noted that, in the embodiments of this application, the number of full-scale convolutional modules in the repair network can be adjusted as needed. Within the allowable range of complexity, increasing the number of modules helps to improve the repair capability of the model.

[0083] Furthermore, in the embodiments of this application, a global-scale convolutional network can be used to transform the input feature information to obtain the corresponding frequency domain feature map; then, the frequency domain feature map is convolved to obtain the convolution result; finally, the convolution result can be transformed to obtain the corresponding repaired feature information.

[0084] It should be noted that, in the embodiments of this application, for the feature information of the feature group input to the global scale convolutional network, the feature information needs to be transformed from the spatial domain form to the frequency domain form (frequency domain feature map), then the frequency domain feature map is convolved, and then the feature map after the convolution operation is transformed to transform it from the frequency domain form back to the spatial domain form to obtain the corresponding repaired feature information.

[0085] Furthermore, in the embodiments of this application, based on this global-scale convolutional network, various transformation methods can be used to transform the feature information. For example, Discrete Fourier Transform, Cosine Fourier Transform, and Fast Fourier Transform can be used. Taking Discrete Fourier Transform as an example, the transformation formula for transforming feature information from spatial domain form to frequency domain form is as follows:

[0086]

[0087] Where W and H are the width and height of the feature map, respectively; x and y represent the horizontal and vertical coordinates of the feature map in the spatial domain form, respectively; and u and v represent the horizontal and vertical coordinates of the feature map in the frequency domain form, respectively.

[0088] As can be seen from formula (1), the value of each coordinate point in the frequency domain form of the feature map is calculated by the values ​​of all coordinate points in its spatial domain. Therefore, each coordinate point in the frequency domain form has a global view.

[0089] Then, convolution operations can be performed on the frequency domain feature map. The convolution operation module in this application includes a 3×3 convolution module, a batch normalization module, and a Rectified Linear Unit (ReLU) activation module.

[0090] Finally, the feature map after convolution can be transformed from its frequency domain form back to its spatial domain form (repaired feature information). Taking the inverse Discrete Fourier transform as an example, the transformation formula for converting feature information from its frequency domain form to its spatial domain form is as follows:

[0091]

[0092] Where W and H are the width and height of the feature map, respectively; x and y represent the horizontal and vertical coordinates of the feature map in the spatial domain form, respectively; and u and v represent the horizontal and vertical coordinates of the feature map in the frequency domain form, respectively.

[0093] Furthermore, in the embodiments of this application, after the feature information of the second feature map is input into the first-scale convolutional network, the second-scale convolutional network and the global-scale convolutional network according to multiple feature groups to generate multiple sets of repaired feature information, the terminal can then perform fusion processing on the multiple sets of repaired feature information based on the repair network, and finally generate the corresponding second repaired feature map.

[0094] Furthermore, in the embodiments of this application, the decoding network can be used to perform upsampling processing, that is, the decoding network can be used to process the input image. Therefore, the repaired image corresponding to the image to be repaired obtained based on the decoding network has the same size as the image to be repaired. For example, assuming the size of the image to be repaired is (H, W), then the size of the corresponding repaired image is also (H, W).

[0095] For example, in an embodiment of this application, the decoding network may include a three-layer deconvolution module (three-layer deconvolution network). Assuming the size of the image to be repaired is (H, W), the second repair feature map output by the repair network can be used as the input of the decoding network for upsampling operation, and finally the repaired image with a size of (H, W) is output.

[0096] Furthermore, in the embodiments of this application, after performing image inpainting processing based on the first image and the second image using a full-scale convolutional neural network to obtain the repaired image corresponding to the image to be repaired, the terminal can also display the repaired image. That is, the terminal can restrict the repaired image to the display screen to further process the repaired image according to the received processing instructions.

[0097] Furthermore, in the embodiments of this application, Figure 3 A schematic diagram of the implementation process of the image restoration method. Figure 2 ,like Figure 3 As shown in the embodiments of this application, the method for image restoration processing by the terminal may further include the following steps:

[0098] Step 103: Determine the training data; wherein, the training data includes the ground truth image, as well as the hole image and mask image corresponding to the ground truth image.

[0099] In embodiments of this application, the terminal can pre-train a preset repair network and a full-scale convolutional neural network. Specifically, the terminal can first determine the training data used for model training.

[0100] It should be noted that, in the embodiments of this application, the training data may include ground truth images, as well as hole images (as training data for the first image) and mask images (as training data for the second image) corresponding to the ground truth images.

[0101] It should be noted that, in the embodiments of this application, natural scene data with a resolution of 256×256 can be selected as the ground truth image. The mask image in the training data can be generated randomly, and a mask image with the same resolution as the input image can be generated randomly. At the same time, the input image can be directly processed using the mask image to obtain the corresponding hole image. The input image can be used as the ground truth image to train the preset repair network and the full-scale convolutional neural network.

[0102] Step 104: Train the preset repair image using the ground truth image and the hole image; train the full-scale convolutional neural network using the ground truth image, the hole image, and the mask image.

[0103] In the embodiments of this application, after determining the training data including the ground truth image, the hole image and the mask image corresponding to the ground truth image, the terminal can further use the ground truth image and the hole image to train the preset repair image; at the same time, the ground truth image, the hole image and the mask image can be used to train a full-scale convolutional neural network.

[0104] Furthermore, in the embodiments of this application, model training can be divided into two stages, namely two branches. The first branch can train a preset repair network, and the second branch can train a full-scale convolutional neural network.

[0105] It should be noted that when training the preset repair network, the holed image can be reduced in size and smoothed to obtain a reduced smooth image. This reduced smooth image is then input into the preset repair network to obtain its repair features. These features can then be input into the decoding network (decoder) to obtain the repaired reduced smooth image. Finally, the network parameters of the preset repair network can be updated based on the repaired reduced smooth image to complete the training. Specifically, when updating the network parameters based on the repaired reduced smooth image, the L1 distance and adversarial loss function can be calculated between the repaired reduced smooth image and its corresponding ground truth image, allowing gradient backpropagation for updating the network parameters.

[0106] It should be noted that when training a full-scale convolutional neural network, the image with holes and the mask image can be concatenated first. The concatenated image is then input into the encoder network to obtain a feature map. Next, the feature map is concatenated with the repaired features of the scaled-down smoothed image. The concatenated image is then input into the fusion network and the repair network to obtain a repaired feature map. This repaired feature map can then be input into the decoder network. The decoder scales up the input repaired feature map according to the corresponding scale, resulting in a repaired image of the same size as the model's input image. Finally, the network parameters of the full-scale convolutional neural network can be updated based on the repaired image to complete the training of the full-scale convolutional neural network. Specifically, when updating the network parameters of the full-scale convolutional neural network based on the repaired image, the L1 distance, perceptual distance, and adversarial loss function can be calculated between the repaired image and its corresponding ground truth image, allowing gradient backpropagation for updating the network parameters.

[0107] Therefore, in the embodiments of this application, the image restoration method proposed in steps 101 to 104 above is used to perform image restoration processing by combining the first image (hole image) and the second image (mask image) of the image to be restored. Specifically, the terminal can use the restoration features (first restoration feature map) of the scaled-down smoothed map of the first image to guide the image restoration processing of the full-scale convolutional neural network. Compared to the common approach of using edge maps or segmentation maps of the same size as the original image as the guiding input, in the embodiments of this application, firstly, compared to edge maps, smoothed maps are richer in information, including not only structural information but also color information; secondly, compared to segmentation maps, smoothed maps are easier to obtain and can be directly calculated, while segmentation maps require deep segmentation networks, resulting in a large computational load; thirdly, using the restored features as input, rather than the unrestored image, can mitigate the error propagation caused by errors in smoothed map restoration; and fourthly, using a scaled-down image (e.g., scaled down by a factor of 8) as input allows for better structural restoration, which can be used to guide the restoration of high-definition images.

[0108] Furthermore, in the embodiments of this application, when the terminal uses a full-scale convolutional neural network for image inpainting, the core module is the full-scale convolutional module. In common schemes that employ multiple-scale convolutional modules, in ultra-high-resolution image inpainting tasks, the area of ​​the hole region is large, and the perception range of the multi-scale module is still insufficient to effectively model the relationship between the hole region and the known region. In the embodiments of this application, the full-scale convolutional module introduces multiple levels of convolutional operations at multiple perception scales. For example, a first-scale convolutional network including two small-scale convolutional modules with perception ranges of 3×3 and 5×5 respectively; a second-scale convolutional network including two medium-scale convolutional modules with perception ranges of 9×9 and 17×17 respectively; and a global-scale convolutional network with a global-scale convolutional module. The multi-level full-scale convolutional module can effectively model the local and global contextual relationships between the hole region and the known region, enabling better inpainting operations.

[0109] Currently, there are two main approaches to restoring high-resolution images. This application selects a representative work from each for comparison. The HiFill algorithm is a network algorithm with a large perceptual range module design, while the AOTGAN algorithm is a scheme that first restores the downsampled image and then performs upsampling. The test images are two real high-resolution images taken with a mobile phone's high-resolution camera. The images are 2K images with a resolution of 1500×2000. Larger foreground objects were removed, and the images were then restored.

[0110] Figures 4a-4d For comparison purposes Figure 1 , Figures 5a-5d For comparison purposes Figure 2 As shown in the figure, Figure 4a To create a hole image after removing the foreground object, Figure 4b The image is a restored image after processing using the AOTGAN algorithm. Figure 4c This is the restored image after processing using the HiFill algorithm. Figure 4d The image is a restored image after being restored using the image restoration method proposed in this application. Figure 5a To create a hole image after removing the foreground object, Figure 5b The image is a restored image after processing using the AOTGAN algorithm. Figure 5c This is the restored image after processing using the HiFill algorithm. Figure 5dThe image shown is the restored image after processing using the image inpainting method proposed in this application. The visualized restoration results show that the HiFill algorithm, due to its direct introduction of pixel residual information from known surrounding areas during upsampling, results in significant structural distortion. Although AOTGAN incorporates a convolutional module with a large receptive range, it still cannot meet the restoration requirements of ultra-high-resolution images, resulting in poor restoration performance. The image inpainting method of this application, by introducing a downscaling smoothing map for restoration and guidance, improves the model's ability to restore the global structure of ultra-high-resolution images. The use of a full-scale convolutional neural network further enhances the model's restoration capabilities, ultimately producing better restoration results.

[0111] This application proposes an image inpainting method. The terminal determines a target region in the image to be inpainted and generates a first image and a second image corresponding to the target region. Based on a full-scale convolutional neural network, image inpainting processing is performed on the first and second images to obtain the inpainted image. Therefore, in the embodiments of this application, the terminal can generate a first image (hole image) and a second image (mask image) of the image to be inpainted based on the target region, and then use a full-scale convolutional neural network to combine the first and second images for image inpainting processing. Specifically, by using a full-scale convolutional neural network, multiple levels and multiple perceptual scales of convolutional operations are introduced during image inpainting, which can effectively model the local and global contextual relationships between the hole region and the known region, resulting in better inpainting operations and thus improving the image inpainting effect.

[0112] Based on the above embodiments, in another embodiment of this application, Figure 6 This is a schematic diagram illustrating the implementation principle of the image restoration proposed in the embodiments of this application. Figure 1 ,like Figure 6 As shown, after reading, capturing, or receiving an image (the image to be repaired), the terminal can perform repair processing on the image based on the corresponding repair instructions. Specifically, the terminal can first determine the region (target region) on the image corresponding to the target object (target object) to be removed through touch commands received on the user interface. Then, it generates a hollow image (first image) of the target region after removing the target object and a mask image (second image) with the same resolution as the original image (the image to be repaired).

[0113] It should be noted that the mask image is a black and white image. The area that is painted over (the target area) has a pixel value of 255 and is displayed as white, while the pixel value of the rest of the area is 0 and is displayed as black.

[0114] Next, the terminal can perform image restoration processing based on the hole image and the mask image to obtain the restored image (the restored image), and then display the restored image on the interface for further operation.

[0115] Furthermore, in the embodiments of this application, Figure 7 This is a schematic diagram illustrating the implementation principle of the image restoration proposed in the embodiments of this application. Figure 2 ,like Figure 7 As shown, the image inpainting method proposed in this application requires two branches for input hole images and mask images of size (H, W). The first branch reduces the hole image to (1 / 8×H, 1 / 8×W) and performs smoothing processing, so that the image retains good structural information while smoothing the texture details. The reduced smooth image is then input into a convolutional neural network (preset inpainting network) for inpainting, resulting in the inpainting features of the reduced smooth image (first inpainting feature map). The size of the first inpainting feature map is (1 / 8×H, 1 / 8×W), and the number of channels is 256. The second branch uses a full-scale convolutional neural network to repair the holed image. First, the holed image and the mask image are concatenated and input into the encoder (encoding network) to obtain a feature map (first feature map) with a size of (1 / 8×H, 1 / 8×W) and 512 channels. Then, a fusion network is used to concatenate and fuse the repaired features (first repaired feature map) of the reduced smoothed image from the first branch with the output features (first feature map) of the encoder, outputting a feature map (second feature map) with a size of (1 / 8×H, 1 / 8×W) and 512 channels. The second feature map is then input into multiple full-scale convolutional neural network modules (repair network) to obtain the repaired feature map (second repaired feature map). Finally, the repaired feature map is input into a decoder (decoding network) for amplification to obtain a repaired image with a size of (H, W).

[0116] It should be noted that, in the embodiments of this application, the basic principle of the reduction and smoothing process is to smooth the texture details of the image while maintaining its structural information, i.e., keeping the edges of the image clear. Specifically, the reduction processing of the first image in the first branch can employ local mean shifting, or methods such as nearest neighbor, bilinear interpolation, or bicubic interpolation; the smoothing processing of the first image in the first branch can employ L0 norm smoothing algorithms, or other smoothing filtering methods.

[0117] Further, in the embodiments of this application, assuming the size of the image to be repaired is (H, W) and the number of channels is 256, the preset repair network of the first branch may include a three-layer downsampling convolutional module, a six-layer residual convolutional module, and a three-layer upsampling deconvolutional module. The feature map output by the three-layer downsampling convolutional module is 1 / 4 × 1 / 4 the size of the input image (reduced smoothed image) and has 256 channels. The processing of the six-layer residual convolutional module maintains the size of the feature map. The feature map output by the three-layer upsampling deconvolutional module (the first repaired feature map) has the same size as the input image and 256 channels, and is used for fusion processing with the intermediate layer feature map (the first feature map) of the second branch.

[0118] It should be noted that in the embodiments of this application, the preset repair network of the first branch can also adopt other network structures. For example, the residual convolution module can be replaced by a dilated convolution module (dilated convolution network) or the full-scale convolution module (full-scale convolution network) proposed in the embodiments of this application.

[0119] Furthermore, in the embodiments of this application, the full-scale convolutional neural network of the second branch may include four parts: an encoder (encoding network), a fusion network, a repairer (repair network), and a decoder (decoding network).

[0120] It should be noted that, in the embodiments of this application, the encoder may include a four-layer convolutional module (convolutional network), the output feature map (first feature map) is 1 / 8 × 1 / 8 the size of the input image (first image and second image), and the number of channels is 512.

[0121] It should be noted that in the embodiments of this application, the fusion unit is a single-layer convolutional module (convolutional network). The output feature map of the first branch (first repaired feature map) and the output feature map of the encoder of the second branch (first feature map) are concatenated as input. The feature map size of the input to the fusion unit is (1 / 8×H, 1 / 8×W), and the number of channels is 768. The size of the output feature map (second feature map) is (1 / 8×H, 1 / 8×W), and the number of channels is 512.

[0122] For the repair network, the terminal can pre-configure at least one first-scale convolutional network, at least one second-scale convolutional network, and at least one global-scale convolutional network. That is, the number of first-scale convolutional networks, second-scale convolutional networks, and global-scale convolutional networks can be set to any size, and this application does not impose specific limitations.

[0123] Furthermore, in the embodiments of this application, the first-scale convolutional network and the second-scale convolutional network have the same convolutional kernel; the different networks in the first-scale convolutional network have different sensing ranges and dilation coefficients; the different networks in the second-scale convolutional network have different sensing ranges and dilation coefficients.

[0124] In other words, in the embodiments of this application, different first-scale convolutional networks have different sensing ranges and dilation coefficients, and different second-scale convolutional networks also have different sensing ranges and dilation coefficients.

[0125] It should be noted that, in the embodiments of this application, the repairer may include an 8-layer full-scale convolutional module (full-scale convolutional network). Figure 8 This is a schematic diagram of the structure of a full-scale convolutional module, as shown below. Figure 8 As shown, given the input features (second feature map), the feature information is first divided into 16 equal parts according to the number of channels and then into 5 groups (feature groups). The first four groups each have 3 channels, and the fifth group has 4 channels. The network consists of five convolutional networks: a first-scale convolutional network (e.g., Network 1) with a kernel size of 3×3 and a receptive range of 3×3; a second-scale convolutional network (e.g., Network 2) with a kernel size of 3×3, a dilation factor of 2, and a receptive range of 5×5; a third-scale convolutional network (e.g., Network 3) with a kernel size of 3×3, a dilation factor of 4, and a receptive range of 9×9; a fourth-scale convolutional network (e.g., Network 4) with a kernel size of 3×3, a dilation factor of 8, and a receptive range of 17×17; and a fifth-scale global convolutional network (e.g., Network 5) that transforms the spatial domain to the frequency domain and performs convolutional operations in the frequency domain.

[0126] It should be noted that, in the embodiments of this application, the number of full-scale convolutional modules in the repairer of the second branch can be adjusted as needed. Within the allowable range of complexity, increasing the number of modules helps to improve the repair capability of the model.

[0127] Furthermore, in the embodiments of this application, after obtaining multiple sets of repaired feature information corresponding to multiple feature groups of the second feature map through the first-scale convolutional network, the second-scale convolutional network, and the global-scale convolutional network, the multiple sets of repaired feature information can be concatenated and fused to finally output the second repaired feature map.

[0128] It should be noted that, in the embodiments of this application, Figure 9 This is a schematic diagram of the structure of a global-scale convolutional network, such as... Figure 9As shown, for the feature information of the feature group input to the global scale convolutional network (global perception module), the feature information needs to be transformed from the spatial domain form to the frequency domain form (frequency domain feature map), then the frequency domain feature map is convolved, and then the feature map after the convolution operation is transformed to transform it from the frequency domain form back to the spatial domain form to obtain the corresponding repaired feature information.

[0129] Furthermore, in the embodiments of this application, based on the global scale convolutional network, various transformation processing methods can be used to transform the feature information. For example, Discrete Fourier Transform, Cosine Fourier Transform, Fast Fourier Transform, and other transformation processing methods can be used. Taking Discrete Fourier Transform as an example, the transformation formula for transforming feature information from spatial domain form to frequency domain form is shown in formula (1).

[0130] As can be seen from formula (1), the value of each coordinate point in the frequency domain form of the feature map is calculated by the values ​​of all coordinate points in its spatial domain. Therefore, each coordinate point in the frequency domain form has a global view.

[0131] Then, convolution operations can be performed on the frequency domain feature map. The convolution operation module in this application includes a 3×3 convolution module, a batch normalization module, and a ReLU activation module.

[0132] Finally, the feature map after convolution can be transformed from the frequency domain form back to the spatial domain form (repaired feature information). Among them, taking the inverse discrete Fourier transform as an example, the transformation formula for transforming feature information from the frequency domain form to the spatial domain form is (2).

[0133] Furthermore, in the embodiments of this application, the decoder may include a three-layer deconvolution module. The decoder may use the output features of the repairer (the second repair feature map) as input for upsampling operation, and finally output a repaired image of the same size as the image to be repaired.

[0134] Furthermore, in the embodiments of this application, the terminal can pre-train an image inpainting model that includes a preset inpainting network and a full-scale convolutional neural network. The image inpainting method proposed in this application is implemented using an image inpainting model that includes a preset inpainting network and a full-scale convolutional neural network.

[0135] It should be noted that, in the embodiments of this application, during the training of the image restoration model, natural scene data with a resolution of 256×256 can be selected. The mask image in the training data can be generated randomly, with the same resolution as the input image. At the same time, the input image can be directly processed using the mask image to obtain the corresponding hole image. The input image can be used as the ground truth image to train the image restoration model, which includes a preset restoration network and a full-scale convolutional neural network.

[0136] It is understood that, in the embodiments of this application, the training of the image restoration model can be divided into two stages: first, training the restoration network for the reduced smooth image (i.e., the preset restoration network of the first branch), and then training the training network for the original image (i.e., the full-scale convolutional neural network of the second branch). The forward inference operations and specific network structures of the preset restoration network and the full-scale convolutional neural network have been described in the above embodiments and will not be repeated here.

[0137] Furthermore, in the embodiments of this application, during the training of the image restoration model, for the preset restoration network of the first branch, the loss function for training the restoration network (preset restoration network) of the reduced smooth image may include a reconstruction loss function. and adversarial discriminant loss function The formulas are as follows:

[0138]

[0139] Among them, X s For the ground truth image to be scaled down and smoothed, M s To be with X s A scaled-down mask image with the same resolution, X s_hole For the input image with reduced smooth holes, F s This is a preset network repair function.

[0140]

[0141] Among them, P data (X s F represents the sample distribution of the true smoothed scaled-down plot. s (X s_hole M s This refers to the process of inputting the hole image and mask image into the repair network for repair. This is a discriminator model used to determine whether the input data is a generated sample or a real sample.

[0142] Furthermore, in the embodiments of this application, during the training of the image restoration model, for the second branch of the full-scale convolutional neural network, the loss function for training the restoration network (full-scale convolutional neural network) of the original image may include a reconstruction loss function. Perceptual loss function and adversarial discriminant loss function The formulas are as follows:

[0143]

[0144] Where X is the ground truth image, M is the mask image, and X hole Let F be the input holed image, and F be the inpainting network (full-scale convolutional neural network).

[0145]

[0146] Where P is a perceptual network used to obtain the perceptual features of the input image.

[0147]

[0148] Among them, P data (X) represents the sample distribution of real natural images, F(X) hole M) represents the operation of inputting the hole image and mask image into the repair network for repair. This is a discriminator model used to determine whether the input data is a generated sample or a real sample.

[0149] For example, Figure 10 This is a schematic diagram illustrating the training method for an image restoration model, as shown below. Figure 10 As shown, when training an image inpainting model that includes a preset inpainting network and a full-scale convolutional neural network, natural scene data can be selected as the input image of the model, i.e., the ground truth image. The mask image in the training data can be generated randomly, with the same resolution as the input image (ground truth image). At the same time, the input image can be directly processed using the mask image to obtain the corresponding hole image. The input image can be used as the ground truth image to train the image inpainting model that includes the preset inpainting network and the full-scale convolutional neural network.

[0150] Furthermore, in the embodiments of this application, the training of the image restoration model can be divided into two stages, namely two branches. The first branch can be trained on a preset restoration network, and the second branch can be trained on a full-scale convolutional neural network.

[0151] It should be noted that when training the preset repair network, the holed image can be reduced in size and smoothed to obtain a reduced smooth image. This reduced smooth image is then input into the preset repair network to obtain its repair features. These features can then be input into the decoding network (decoder) to obtain the repaired reduced smooth image. Finally, the network parameters of the preset repair network can be updated based on the repaired reduced smooth image to complete the training. Specifically, when updating the network parameters based on the repaired reduced smooth image, the L1 distance and adversarial loss function can be calculated between the repaired reduced smooth image and its corresponding ground truth image, allowing gradient backpropagation for updating the network parameters.

[0152] It should be noted that when training a full-scale convolutional neural network, the image with holes and the mask image can be concatenated first. The concatenated image is then input into the encoder network to obtain a feature map. Next, the feature map is concatenated with the repaired features of the scaled-down smoothed image. The concatenated image is then input into the fusion network and the repair network to obtain a repaired feature map. This repaired feature map can then be input into the decoder network. The decoder scales up the input repaired feature map according to the corresponding scale, resulting in a repaired image of the same size as the model's input image. Finally, the network parameters of the full-scale convolutional neural network can be updated based on the repaired image to complete the training of the full-scale convolutional neural network. Specifically, when updating the network parameters of the full-scale convolutional neural network based on the repaired image, the L1 distance, perceptual distance, and adversarial loss function can be calculated between the repaired image and its corresponding ground truth image, allowing gradient backpropagation for updating the network parameters.

[0153] In summary, the image restoration method proposed in the embodiments of this application combines a first image (hole image) and a second image (mask image) of the image to be restored for image restoration processing. Specifically, the terminal can use the restoration features (first restoration feature map) of the scaled-down smoothed map of the first image to guide the image restoration processing of a full-scale convolutional neural network. Compared to common schemes that use edge maps or segmentation maps of the same size as the original image as guiding input, in the embodiments of this application, firstly, compared to edge maps, smoothed maps are richer in information, including both structural and color information; secondly, compared to segmentation maps, smoothed maps are more readily available and can be obtained directly through computation, while segmentation maps require deep segmentation networks, resulting in high computational costs; thirdly, using the restored features as input, rather than the unrestored image, can mitigate the error propagation caused by errors in smoothed map restoration; and fourthly, using a scaled-down image (e.g., scaled down by a factor of 8) as input allows for better structural restoration, which is used to guide the restoration of high-definition images.

[0154] Furthermore, in the embodiments of this application, when the terminal uses a full-scale convolutional neural network for image inpainting, the core module is the full-scale convolutional module. In common schemes that employ multiple-scale convolutional modules, in ultra-high-resolution image inpainting tasks, the area of ​​the hole region is large, and the perception range of the multi-scale module is still insufficient to effectively model the relationship between the hole region and the known region. In the embodiments of this application, the full-scale convolutional module introduces multiple levels of convolutional operations at multiple perception scales. For example, a first-scale convolutional network including two small-scale convolutional modules with perception ranges of 3×3 and 5×5 respectively; a second-scale convolutional network including two medium-scale convolutional modules with perception ranges of 9×9 and 17×17 respectively; and a global-scale convolutional network with a global-scale convolutional module. The multi-level full-scale convolutional module can effectively model the local and global contextual relationships between the hole region and the known region, enabling better inpainting operations.

[0155] This application proposes an image inpainting method. The terminal determines a target region in the image to be inpainted and generates a first image and a second image corresponding to the target region. Based on a full-scale convolutional neural network, image inpainting processing is performed on the first and second images to obtain the inpainted image. Therefore, in the embodiments of this application, the terminal can generate a first image (hole image) and a second image (mask image) of the image to be inpainted based on the target region, and then use a full-scale convolutional neural network to combine the first and second images for image inpainting processing. Specifically, by using a full-scale convolutional neural network, multiple levels and multiple perceptual scales of convolutional operations are introduced during image inpainting, which can effectively model the local and global contextual relationships between the hole region and the known region, resulting in better inpainting operations and thus improving the image inpainting effect.

[0156] Based on the above embodiments, in another embodiment of this application... Figure 11 Diagram of the terminal's structural composition Figure 1 ,like Figure 11 As shown, the terminal 10 proposed in this application embodiment may include a determining unit 11, a generating unit 12, and a repairing unit 13.

[0157] The determining unit 11 is used to determine the target region in the image to be repaired;

[0158] The generation unit 12 is used to generate a first image and a second image corresponding to the image to be repaired based on the target region.

[0159] The repair unit 13 is used to perform image repair processing based on the first image and the second image using a full-scale convolutional neural network to obtain the repaired image corresponding to the image to be repaired.

[0160] Figure 12 Diagram of the terminal's structural composition Figure 2 ,like Figure 12 As shown, the terminal 10 proposed in this application embodiment may further include a processor 14, a memory 15 storing instructions executable by the processor 14, and further, the terminal 10 may also include a communication interface 16 and a bus 17 for connecting the processor 14, the memory 15 and the communication interface 16.

[0161] In the embodiments of this application, the processor 14 can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field-Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above-mentioned processor function can also be other, and this application embodiment does not specifically limit it. The terminal 10 may also include a memory 15, which can be connected to the processor 14. The memory 15 is used to store executable program code, which includes computer operation instructions. The memory 15 may include high-speed RAM memory and may also include non-volatile memory, such as at least two disk drives.

[0162] In embodiments of this application, bus 17 is used to connect communication interface 16, processor 14, and memory 15, as well as the mutual communication between these devices.

[0163] In embodiments of this application, memory 15 is used to store instructions and data.

[0164] Furthermore, in an embodiment of this application, the processor 14 is configured to determine a target region in the image to be repaired, and generate a first image and a second image corresponding to the image to be repaired based on the target region; and perform image repair processing based on the first image and the second image using a full-scale convolutional neural network to obtain a repaired image corresponding to the image to be repaired.

[0165] In practical applications, the aforementioned memory 15 can be volatile memory, such as random-access memory (RAM); or non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); or a combination of the above types of memory, and provide instructions and data to the processor 14.

[0166] Furthermore, in this embodiment, the functional modules can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0167] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] This application proposes a terminal that determines a target region in an image to be repaired and generates a first image and a second image corresponding to the image to be repaired based on the target region. Using a full-scale convolutional neural network, image repair processing is performed on the first and second images to obtain a repaired image. Thus, in this embodiment, the terminal can generate a first image (hole image) and a second image (mask image) of the image to be repaired based on the target region, and then use a full-scale convolutional neural network to combine the first and second images for image repair processing. Specifically, by using a full-scale convolutional neural network, multiple levels and multiple perceptual scales of convolutional operations are introduced during image repair, which can effectively model the local and global contextual relationships between the hole region and the known region, resulting in better repair operations and improved image repair performance.

[0169] This application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the image restoration method described above.

[0170] Specifically, the program instructions corresponding to an image restoration method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to an image restoration method in the storage media are read or executed by an electronic device, the following steps are included:

[0171] A target region is determined in the image to be repaired, and a first image and a second image corresponding to the image to be repaired are generated based on the target region;

[0172] Based on a full-scale convolutional neural network, image inpainting processing is performed on the first image and the second image to obtain the inpainted image corresponding to the image to be inpainted.

[0173] Those skilled in the art will understand that embodiments of this application can be provided as methods, terminals, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage) containing computer-usable program code.

[0174] This application is described with reference to schematic and / or block diagrams of implementations of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the schematic and / or block diagrams can be implemented by computer program instructions, and combinations of blocks in the schematic and / or block diagrams can be implemented. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the schematic and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0175] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the implementation flow diagram. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0176] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0177] The above description is merely a preferred embodiment of this application and is not intended to limit the scope of protection of this application.

Claims

1. An image restoration method, characterized in that, The method includes: A target region is determined in the image to be repaired, and a first image and a second image corresponding to the image to be repaired are generated based on the target region; The first image is reduced in size and smoothed to obtain a reduced smooth image corresponding to the first image. The reduced smooth map is input into a preset repair network, and the first repair feature map is output. The first image, the second image, and the first repaired feature map are input into a full-scale convolutional neural network to generate the repaired image.

2. The method according to claim 1, characterized in that, The full-scale convolutional neural network includes an encoding network, a fusion network, a repair network, and a decoding network.

3. The method according to claim 2, characterized in that, The step of inputting the first image, the second image, and the first repaired feature map into a full-scale convolutional neural network to generate the repaired image includes: The first image and the second image are input into the encoding network to generate a first feature map; Based on the fusion network, the first repair feature map and the first feature map are fused to generate a second feature map; The second feature map is repaired based on the repair network to generate a second repaired feature map. The second repaired feature map is input into the decoding network to generate the repaired image.

4. The method according to claim 3, characterized in that, The repair network includes a first-scale convolutional network, a second-scale convolutional network, and a global-scale convolutional network. The step of repairing the second feature map based on the repair network to generate a second repaired feature map includes: The feature information of the second feature map is grouped according to the number of channels in the second feature map to obtain multiple feature groups; According to the multiple feature groups, the feature information of the second feature map is respectively input into the first-scale convolutional network, the second-scale convolutional network, and the global-scale convolutional network to generate multiple sets of repaired feature information; The multiple sets of repaired feature information are fused to generate the second repaired feature map.

5. The method according to claim 4, characterized in that, The first-scale convolutional network and the second-scale convolutional network have the same convolutional kernel; The different networks in the first-scale convolutional network have different sensing ranges and dilation coefficients; The different networks in the second-scale convolutional network have different sensing ranges and dilation coefficients.

6. The method according to claim 4, characterized in that, The global-scale convolutional network is used to transform the input feature information to obtain the corresponding frequency domain feature map; to perform convolution processing on the frequency domain feature map to obtain the convolution result; and to transform the convolution result to obtain the corresponding repaired feature information.

7. The method according to claim 1, characterized in that, The preset repair network includes a downsampling convolutional network, and / or a residual convolutional network, and / or an upsampling deconvolutional network, and / or a dilated convolutional network, and / or a full-scale convolutional network.

8. The method according to claim 1, characterized in that, The method further includes: Determine the training data; wherein the training data includes ground truth images, and hole images and mask images corresponding to the ground truth images; The preset repair image is trained using the ground truth image and the hole image; the full-scale convolutional neural network is trained using the ground truth image, the hole image, and the mask image.

9. The method according to claim 1, characterized in that, The step of determining the first image corresponding to the image to be repaired based on the target region includes: Remove pixels from the target region from the image to be repaired to generate the first image; and / or, The pixel value corresponding to the target region in the image to be repaired is set to a first value to generate the first image.

10. The method according to claim 1, characterized in that, The step of determining the second image corresponding to the image to be repaired based on the target region includes: The pixel value corresponding to the target region in the image to be repaired is set to a second value, and the pixel values ​​corresponding to other regions in the image to be repaired other than the target region are set to a third value, thereby generating the second image.

11. A terminal, characterized in that, The terminal includes: a determining unit, a generating unit, and a repair unit. The determining unit is used to determine the target region in the image to be repaired; The generation unit is used to generate a first image and a second image corresponding to the image to be repaired based on the target region. The repair unit is used to perform reduction and smoothing processing on the first image to obtain a reduced smooth map corresponding to the first image; input the reduced smooth map into a preset repair network to output a first repair feature map; and input the first image, the second image, and the first repair feature map into a full-scale convolutional neural network to generate a repaired image.

12. A terminal, characterized in that, The terminal includes: a processor and a memory storing processor-executable instructions, wherein when the instructions are executed by the processor, the method described in any one of claims 1-10 is implemented.

13. A computer-readable storage medium having a program stored thereon for use in a terminal, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-10.

Images