Image inpainting method and device, electronic equipment and storage medium

Abstract

The application provides an image repairing method and device, electronic equipment and storage medium. The method comprises: acquiring a to-be-repaired image and a reference image, aligning and matching the to-be-repaired image and the reference image, and determining key point information corresponding to the to-be-repaired image and the reference image; inputting the to-be-repaired image and the reference image into a preset neural network model to determine gradient features of the to-be-repaired image and the reference image; extracting matching image features of the matching image, fusing the matching image features and the gradient features, and obtaining a repaired image of the to-be-repaired image. The information of the reference image can be effectively used to recover the mask area of the to-be-repaired image, and the repairing efficiency and the repairing quality of the to-be-repaired image are improved.

Description

Technical Field

[0001] The present application relates to the field of image restoration technology, and in particular to an image restoration method, device, electronic device, and storage medium. Background Art

[0002] In recent years, deep learning technology has achieved rapid development and its application in the field of image restoration has become increasingly widespread. In order to meet users' demand for image restoration quality, image restoration technology has gradually shifted from traditional non-reference image guidance systems to reference image guidance systems. Traditional image restoration systems are mainly based on encoder-decoder systems of generative adversarial networks (GANs) or variational auto-encoder models. They extract features of the image to be restored and use the local information around the mask area of ​​the image to be restored and the overall global information to obtain the final restoration result. However, when the mask area of ​​the image to be restored is large, the final restoration result obtained by using the local information around the mask area and the overall global information cannot ensure the restoration quality of the image to be restored in the case of complex texture and structural information.

[0003] With the rapid development of the Internet and social media, users have increasingly higher requirements for image editing. How to ensure the restoration quality of the image to be restored in the presence of complex texture and structural information has become the key to image restoration. If the complex texture and structural information of the image to be restored can be accurately and high-quality by using the prior information of the reference image, the defects of related technologies in image restoration of large mask areas can be overcome. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose an image restoration method, device, electronic device and storage medium that overcome the above-mentioned problems or at least partially solve the above-mentioned problems.

[0005] Based on the above objectives, the first aspect of the present application provides an image restoration method, comprising:

[0006] Acquire an image to be repaired and a reference image, and perform alignment and matching on the image to be repaired and the reference image to determine a matching image of the reference image relative to the image to be repaired;

[0007] Using a preset neural network model, the image to be repaired and the reference image are input, and gradient features of the image to be repaired and the reference image are determined;

[0008] Extracting matching image features of the matching image, fusing the matching image features with the gradient features, and obtaining a restored image of the image to be restored.

[0009] Optionally, obtaining an image to be repaired and a reference image, and performing alignment and matching on the image to be repaired and the reference image to determine a matching image of the reference image relative to the image to be repaired includes:

[0010] Constructing an image pyramid using Gaussian convolution to respectively determine a first multi-scale image and a second multi-scale image at corresponding scales of the image to be restored and the reference image;

[0011] comparing any pixel in the first multiscale image with pixels in a neighborhood, determining an extreme point of the first multiscale image and using the extreme point of the first multiscale image as a first key point of the first multiscale image;

[0012] comparing any pixel in the second multiscale image with pixels in a neighborhood, determining an extreme point of the second multiscale image and using the extreme point of the second multiscale image as a second key point of the second multiscale image;

[0013] Calculating the gradient direction and gradient magnitude of the first key point according to the position of the first key point to determine the first key point vector;

[0014] Calculating the gradient direction and gradient magnitude of the second key point according to the position of the second key point to determine the second key point vector;

[0015] The second key point vector is mapped to a position corresponding to the first key point vector, and a matching image of the reference image relative to the image to be repaired is determined.

[0016] Optionally, before mapping the second key point vector to a position corresponding to the first key point vector and determining a matching image of the image to be repaired, the method further includes:

[0017] Calculating the similarity between any of the first key points and all of the second key points to obtain a plurality of second key points similar to any of the first key points;

[0018] The multiple second key points obtained by random sampling are randomly selected two points each time to fit into the first straight line, the interior points of the first straight line are determined and the exterior points are eliminated until the first straight line with the most interior points is obtained, and the second straight line is refitted using the obtained most interior points, and the interior point closest to the second straight line is determined as the second key point matching the first key point.

[0019] Optionally, the gradient feature includes a first gradient feature and a second gradient feature;

[0020] Using the preset neural network model, inputting the image to be repaired and the reference image, and determining the gradient features of the image to be repaired and the reference image, including:

[0021] Calculating the gradient amplitudes of the image to be restored and the reference image in the x-direction and the y-direction;

[0022] Extracting and fusing the gradient information of the image to be repaired and the reference image respectively through gated convolution;

[0023] Using large kernel attention to enlarge the receptive field and represent the long-range relationship of the gradient information;

[0024] According to the gradient magnitude and the long-range relationship, the encoder and the decoder of the preset neural network are used to respectively output a first gradient feature and a second gradient feature between the image to be restored and the reference image.

[0025] Optionally, fusing the matching image feature and the gradient feature to obtain a restored image of the image to be restored includes:

[0026] fusing the matching image feature, the first gradient feature, and the second gradient feature to form a first mixed feature;

[0027] Converting the first mixed feature from spatial domain information to frequency domain information using fast Fourier convolution;

[0028] fusing the spatial domain information and the frequency domain information of the first hybrid feature to obtain a second hybrid feature;

[0029] Performing spectral transformation on the second mixed feature using a global branch of fast Fourier convolution to obtain a frequency domain feature;

[0030] Obtaining a restored image of the image to be restored according to the frequency domain features;

[0031] The frequency domain feature is expressed as:

[0032] feat=Relu(BN(Conv(x))),

[0033] freq=FFT(feat),

[0034] freq=Relu(BN(Conv(freq))),

[0035] out=Conv 1×1 (feat+IFFT(freq)),

[0036] Among them, Relu(·) represents the activation function, x represents the input of the spectral transformation, Conv 1×1represents point convolution, feat represents mixed features, FFT represents real fast Fourier transform, IFFT represents real inverse fast Fourier transform, freq represents frequency domain information, BN(·) represents batch normalization processing, and Conv represents convolution.

[0037] Optionally, the direction and amplitude of the first key point are represented as:

[0038]

[0039] θ(x1, y1) = tan -1 ((L(x1, y1+1)-L(x1, y1-1)) / (L(x1+1, y1)-L(x1-1, y1))),

[0040] wherein m(x1, y1) represents the amplitude of the first key point, θ(x1, y1) represents the direction of the first key point, and L(x1, y1) represents the pixel value of the first multi-scale image in the scale space of the Gaussian pyramid;

[0041] The direction and amplitude of the second key point are represented as:

[0042]

[0043] θ(x2, y2) = tan -1 ((L(x2, y2+1)-L(x2, y2-1)) / (L(x2+1, y2)-L(x2-1, y2))),

[0044] wherein m(x2, y2) represents the amplitude of the second key point, θ(x2, y2) represents the amplitude of the second key point, and L(x2, y2) represents the pixel value of the second multi-scale image in the scale space of the Gaussian pyramid.

[0045] Optionally, the first gradient feature is represented as:

[0046] f e = GateConv i (inp) i∈1, 2, 3,

[0047] wherein GateConv i represents a gated convolution of an encoder of a neural network,

[0048] The second gradient feature is represented as:

[0049] f d = Res∪GateTans i (inp) i∈1, 2, 3, 4,

[0050] Among them, Res represents the residual output result of the neural network, GateTans i Represents the gated convolution of the decoder of a neural network.

[0051] A second aspect of the present application provides an image restoration device, comprising:

[0052] an alignment module, configured to obtain an image to be restored and a reference image, and perform alignment and matching between the image to be restored and the reference image to determine a matching image of the reference image relative to the image to be restored;

[0053] a gradient feature module, configured to use a preset neural network model to input the image to be repaired and the reference image, and determine the gradient features of the image to be repaired and the reference image;

[0054] The image restoration module is used to extract the matching image features of the matching image, fuse the matching image features with the gradient features, and obtain the restoration image of the image to be restored.

[0055] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method described in the first aspect when executing the program.

[0056] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method described in the first aspect.

[0057] From the above, it can be seen that the image restoration method, device, electronic device and storage medium provided by the present application use a reference image to align with the image to be restored, realize the matching image of the reference image relative to the image to be restored, and realize the matching of the position, angle, etc. of the reference image relative to the image to be restored. The gradient features of the reference image and the image to be restored are further extracted to describe the degree of change of the grayscale values ​​therein, and the light difference of the image to be restored relative to the reference image is supplemented. Finally, before fusion, the large core attention block will extract the matching image features of the matching image, fuse the matching image features and gradient information, and more completely restore the complex structure and texture of the image to be restored.

[0058] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in this application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are merely embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0060] Figure 1 This is a flow chart of the image restoration method according to an embodiment of the present application;

[0061] Figure 2 This is a schematic diagram of comparing pixels in the neighborhood of any pixel in an embodiment of the present application;

[0062] Figure 3 This is a schematic diagram of an image restoration device according to an embodiment of the present application;

[0063] Figure 4 This is a schematic diagram of fast Fourier convolution in an embodiment of the present application;

[0064] Figure 5a-5c This is a schematic diagram of the image restoration result of an embodiment of the present application;

[0065] Figure 6 This is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the objectives, technical solutions and advantages of this application more clear, this application is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings.

[0067] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this application belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0068] Figure 1 This is a flowchart of the image restoration method according to an embodiment of the present application.

[0069] like Figure 1As shown, the image repairing method provided in the present application comprises:

[0070] S101, acquire a to-be-repaired image and a reference image, and align and match the to-be-repaired image and the reference image to determine a matching image of the reference image relative to the to-be-repaired image.

[0071] For the mask region, that is, the to-be-repaired image with a larger damaged region, only the gradient information, edge information, segmentation information, etc. of the to-be-repaired image itself are used to recover the structural information of the to-be-repaired image, or the attention mechanism is used to obtain the long-range relationship between the features of the to-be-repaired image itself to perform image repairing, which can achieve corresponding repairing effect, but cannot effectively repair complex texture and structural information. Therefore, in order to effectively repair the complex texture and structural information, the complex texture and structural information need to be effectively extracted, and the extracted texture information and structural information are used to repair the mask region of the to-be-repaired image. However, the to-be-repaired image and the reference image may not be completely consistent in some aspects, such as angle, color, etc. Therefore, the to-be-repaired image and the reference image need to be matched and then the corresponding feature extraction is performed to repair the to-be-repaired image.

[0072] The present application aims to repair the to-be-repaired image more comprehensively, so the feature extraction based on the reference image is used to match the features of the to-be-repaired image, so that the texture, color, etc. of the mask region and the normal region of the to-be-repaired image tend to be consistent.

[0073] However, due to the size of the viewing angle, the angle, the natural light, the size, etc., there are differences between the to-be-repaired image and the reference image. Therefore, the corresponding features in the reference image cannot be directly applied to the to-be-repaired image. Therefore, the reference image and the to-be-repaired image need to be aligned first, so that the feature size and angle in the reference image and the to-be-repaired image are more similar, thereby completing the matching of the reference image and the to-be-repaired image.

[0074] In some embodiments, first, a Gaussian convolution is used to construct an image pyramid and a Gaussian difference pyramid. The Gaussian function is used to perform convolution operation on the to-be-repaired image and the reference image respectively to construct a Gaussian pyramid. The process is as follows:

[0075]

[0076] K(x, y, σ) = G(x, y, σ) * I(x, y),

[0077] where G(x i ,yi ,σ) represents the Gaussian function, I(x,y) represents the original image (the image to be repaired is used when calculating the image to be repaired, and the reference image is used when calculating the reference image), σ represents the scale standard deviation of the Gaussian function, * represents the convolution operation, and L(x, y, σ) represents the multi-scale image at the corresponding scale of the original image (the first multi-scale image at the corresponding scale of the image to be repaired is calculated, and the second multi-scale image at the corresponding scale of the reference image is calculated).

[0078] In some specific embodiments, L(x1, y1, σ) = G(x1, y1, σ) * I(x1, y1) is used to represent the first multi-scale image, and L(x2, y2, σ) = G(x2, y2, σ) * I(x2, y2) is used to represent the second multi-scale image.

[0079] By adjusting the value of the standard deviation, images of different scale spaces can be constructed.

[0080] After obtaining the Gaussian difference pyramid of the image to be inpainted and the reference image, the differences between adjacent layers of the Gaussian difference pyramid are calculated to obtain the Gaussian difference pyramid. In the intermediate layer of the obtained Gaussian difference pyramid, any pixel in the first-scale image is compared with pixels in its neighborhood to determine the extreme point of the first-scale image and use it as the first keypoint of the first-scale image.

[0081] At the same time, any pixel point in the second scale image is compared with the pixel points in the neighborhood, the extreme point of the second scale image is determined, and the extreme point of the second scale image is used as the second key point of the second scale image.

[0082] In this embodiment, the Gaussian difference pyramid has four layers. Therefore, the extreme value of any pixel in the first scaled image is determined by comparing it with pixels in its neighborhood in the middle two layers of the Gaussian difference pyramid. Similarly, the extreme value of any pixel in the second scaled image is determined by comparing it with pixels in its neighborhood in the middle two layers of the Gaussian difference pyramid.

[0083] Exemplary, reference Figure 2 As shown in the figure, in the scale space of the Gaussian difference pyramid, except for edge points, each point has 26 neighborhoods. The maximum and minimum points are found in these 26 neighborhoods as extreme points, and the extreme points are used as key points, that is, the extreme point of the first scale image is used as the first key point, and the extreme point of the second scale image is used as the second key point. The "x" shape represents any pixel point in the first multi-scale image or the second multi-scale image, and the "o" shape represents a pixel point in the neighborhood.

[0084] Then, according to the position of the first key point, the direction and amplitude of the first key point are calculated to determine the first key point vector;

[0085] According to the position of the second key point, a direction and a magnitude of the second key point are calculated to determine a second key point vector.

[0086] The direction and the magnitude of the first key point are calculated according to the following formula:

[0087]

[0088] θ(x1, y1) = tan -1 ((L(x1, y1+1)-L(x1, y1-1)) / (L(x1+1, y1)-L(x1-1, y1))),

[0089] wherein m(x1, y1) represents the magnitude of the first key point, θ(x1, y1) represents the direction of the first key point, and L(x1, y1) represents a pixel value of the first multi-scale image in a scale space of a Gaussian pyramid;

[0090] The direction and the magnitude of the second key point are calculated according to the following formula:

[0091]

[0092] θ(x2, y2) = tan -1 ((L(x2, y2+1)-L(x2, y2-1)) / (L(x2+1, y2)-L(x2-1, y2))),

[0093] wherein m(x2, y2) represents the magnitude of the second key point, θ(x2, y2) represents the direction of the second key point, and L(x2, y2) represents a pixel value of the second multi-scale image in a scale space of a Gaussian pyramid.

[0094] In some possible implementation scenarios, after the gradient direction and the gradient magnitude of the first key point and the second key point are completed, the gradient direction and the gradient magnitude of the pixels in the neighborhood are counted using a histogram, the peak value of the histogram represents the direction of the extreme point, and the auxiliary direction is greater than 80% of the peak value.

[0095] Each extreme point (key point) has three information of position, magnitude and direction, a descriptor is established for each key point, and the key point is described through a set of vectors, i.e., a first key point vector and a second key point vector.

[0096] In some optional embodiments, the first key point and the second key point are described by a set of 4x4x8-dimensional vectors.

[0097] According to the first key point vector and the second key point vector, the second key point vector is mapped to a corresponding position of the first key point vector to determine a matching image of the reference image relative to the image to be repaired. That is, by corresponding the first key point vector to the second key point vector, the alignment of the reference image and the corresponding position of the image to be repaired is achieved, and the similar features in the reference image and the reference image are more close in size, angle, etc. It is equivalent to finding the corresponding positions of the normal region and the mask region of the image to be repaired in the reference image.

[0098] Further, the second key point vector is mapped to the corresponding position of the first key point vector to determine the matching image of the image to be repaired, and the method further comprises:

[0099] The similarity of any of the first key points and all of the second key points is calculated to obtain a plurality of second key points similar to any of the first key points.

[0100] In some optional embodiments, the above calculation is implemented by using k-nearest neighbor matching. First, the first key point and the second key point are determined in the above embodiment, and here the first descriptor of the first key point can be calculated, and the second descriptor of the second key point can be calculated, and the similarity of the first descriptor and the second descriptor is calculated, so as to find a plurality of second descriptors similar to the first descriptor in the second scale image, that is, a plurality of second key points similar to the first key point are obtained in the second scale image.

[0101] After determining the plurality of second key points, in order to further accurately determine which of the specific similar second key points corresponds to the first key point, it is necessary to filter out the inaccurate key points in the plurality of second multi-scale images.

[0102] A plurality of second key points are randomly sampled, two points are randomly selected each time to fit a first straight line, the inliers of the first straight line are determined and the outliers are removed, until the first straight line with the most inliers is obtained, and the second straight line is refitted using the most inliers obtained, and the inlier closest to the second straight line is determined as the second key point matched with the first key point.

[0103] Through similarity judgment, multiple second key points similar to the first key point are obtained. However, compared with the first multiscale image and the second multiscale image, the first key point of the first multiscale image and the corresponding second key point of the second multiscale image are unique. For example, the only corresponding to the first key point A of the first multiscale image is the second key point A1 of the second multiscale image. However, through similarity judgment, multiple second key points may be found, such as A1, A2, A3, A4..., then it is necessary to find the second key point corresponding to the first key point from the multiple second key points, and select two second key points each time. Use two second key points to fit the first straight line, set a threshold here, and use second key points smaller than the threshold as inliers, and second key points larger than the threshold as outliers. Remove the outliers, filter the second key points, and reselect two second key points. Fit the first straight line again, and use the above method to determine the inliers until the first straight line with the most inliers is found. Then use the obtained most inliers to refit the second straight line, and determine the inlier closest to the second straight line as the second key point that matches the first key point. That is to say, the inlier closest to the second straight line as the second key point has the highest similarity with the first key point and is also a match.

[0104] It can be understood that when the first straight line with the most inliers is obtained, and the first straight line is fitted using two random second key points, it has the largest number of inliers and also has a higher accuracy rate, but it cannot accurately reflect the distribution of all inliers. The second straight line is refitted using all the inliers to reflect the distribution of all second key points.

[0105] In some possible implementations, after identifying multiple first keypoints in the first multi-scale image and multiple second keypoints corresponding to the second multi-scale image, a homography matrix is ​​estimated using the first keypoints and the matching second keypoints. The homography matrix can map and transform points in the two images from different perspectives. The homography matrix is ​​used to map the second-scale image to the first-scale image, i.e., to map the reference image to the image to be restored.

[0106] refer to Figure 3 As shown, the matching image is obtained in Figure 3 This is done in the alignment module in .

[0107] S102: Using a preset neural network model, inputting the image to be restored and the reference image, and determining gradient features of the image to be restored and the reference image.

[0108] After completing the alignment of the image to be repaired and the reference image, due to the differences in perspective and light between the reference image and the image to be repaired, directly aligning and copying the reference image will also have defects. Therefore, it is necessary to supplement the image grayscale value caused by the light difference. This application uses gradient features to describe the degree of change in the grayscale values ​​of the reference image and the image to be repaired, and uses the degree of change in the grayscale value to supplement the light difference.

[0109] Specifically, step S102 includes:

[0110] Calculate the gradient magnitudes of the image to be restored and the reference image in the x-direction and the y-direction.

[0111] Optionally, the gradients of the image to be restored and the reference image in the x-direction and the y-direction may be calculated using a Sobel operator, and the gradient amplitudes of the image to be restored and the reference image may be obtained using a square root operation.

[0112] refer to Figure 3 As shown in the gradient extraction module, the gradient information of the image to be repaired and the reference image are respectively extracted and fused through gated convolution.

[0113] Specifically, gated convolution is used to calculate the gradient information Feature and gating of the x-direction gradient and y-direction gradient of the fused image to be repaired and the reference image through two convolutions. The process is as follows:

[0114] Gating y，x =∑∑W g I,

[0115] Feature y，x =∑∑W f I,

[0116] O y，x =φ(Feature y，x )⊙σ(Gating y，x ),

[0117] Among them, W g and W f Represent the convolution kernel weight parameters respectively, I represents the fusion image of the image to be repaired and the reference image, O y，x Represents the output of gated convolution, that is, gradient information, φ(.) represents the GELU activation function, and σ(.) represents the Sigmiod activation function. In the process of extracting and fusing the gradient information of the image to be repaired and the reference image by means of gated convolution, the valid pixel area and invalid pixel area of ​​the image to be repaired, that is, the normal area and the mask area, are judged.

[0118] refer to Figure 3As shown in the gradient extraction module, the large core attention is used to increase the receptive field and characterize the long-range relationship of the gradient information, that is, the large core attention block is used to increase the receptive field and characterize the long-range relationship of the gradient information, wherein the large core attention is implemented by a large core attention block in the form of software or hardware.

[0119] Specifically,

[0120] F=GELU(BN(Conv(feature))),

[0121] Attention=Conv 1×1 (Conv DW - D (Conv DW (F))),

[0122]

[0123] Output=FFN(Attention_out),

[0124] Among them, GELU represents the activation function, BN(·) represents batch normalization processing, feature represents the gradient information Feature output by gated convolution, Conv DW Represents depth convolution, Conv DW-D Denotes deep dilated convolution, Conv 1×1 Represents point convolution.

[0125] It is worth noting that the reference Figure 3 As shown in the lower right corner, the key to the large kernel attention is to decompose the large kernel volume into a deep convolution Conv after the gradient information Feature is processed by convolution (Conv), batch normalization (BN) and activation function (GELU). DW , deep dilated convolution Conv DW-D And channel convolution, where channel convolution can be point convolution Conv 1×1 ,The large kernel attention combines the local receptive field of convolution operation and the long-distance information of self-attention mechanism,overcoming the shortcomings of small receptive field of convolution operation and large computational complexity of self-attention mechanism.

[0126] The long-range relationship of the gradient information is then encoded in the feedforward network (FFN) of the neural network model. Residual calculation is used to increase the depth of the neural network through convolution and activation functions, avoiding the disappearance of key gradient information. The residual calculation is implemented through residual blocks in software or hardware, specifically through the following formula:

[0127] Output=Input+GELU(BN(Conv(Input))),

[0128] wherein, Input is the output of the large kernel attention (i.e., the large kernel attention block), and the output of the residual (i.e., the residual block) is obtained through a skip connection.

[0129] According to the gradient amplitude and the long-range relationship, the first gradient feature and the second gradient feature between the to-be-repaired image and the reference image are respectively output by using the encoder and the decoder of the preset neural network.

[0130] Specifically, the first gradient feature is represented as:

[0131] f e = GateConv i (inp) i∈1, 2, 3,

[0132] wherein, GateConv i represents the gate convolution of the encoder of the neural network,

[0133] The second gradient feature is represented as:

[0134] f d = Res∪GateTans i (inp) i∈1, 2, 3, 4,

[0135] wherein, Res represents the residual output result of the neural network, and GateTans i represents the gate convolution of the decoder of the neural network.

[0136] In this embodiment, the core of the gate convolution is to distinguish the mask region and the normal region of the to-be-repaired image by learning the dynamic features of each channel, that is, even in the deep network, the gate convolution can highlight the information of the mask region in different channels, so that the gate convolution can more effectively extract the gradient information of the normal pixels, thereby avoiding confusion, that is, the gate convolution aims to effectively separate the effective pixel region and the invalid pixel region, that is, the normal region and the mask region, and ensure that the convolution result is only affected by the effective pixels. In order to capture the long-distance relationship of the gradient information, the large kernel attention combines the local receptive field of the convolution operation and the long-distance information of the self-attention mechanism, while overcoming the shortcomings of small receptive field of the convolution operation and large calculation amount of the sub-attention mechanism.

[0137] Finally, the first gradient feature and the second gradient feature are respectively output by using the encoder and the decoder of the neural network.

[0138] It can be understood that the neural network includes an encoding stage and a decoding stage, wherein the encoder herein represents the encoding stage, and the decoder herein represents the decoding stage, wherein the encoder and the decoder do not represent physically existing physical devices.

[0139] In some possible embodiments, the first gradient feature and the second gradient feature are fused through cross-connection. The first gradient feature and the second gradient feature represent the encoder feature and the decoder feature extracted by gated convolution and residual calculation, respectively (the neural network model includes an encoder and a decoder). In order to more effectively fuse the first gradient feature and the second gradient feature, the present application adopts a cross-connection fusion method. Specifically, during the fusion process, the decoder part and the encoder part are fused separately. The cross-fusion method enables the neural network to perceive the decoder features of the gradient information in the encoding stage during the feedforward repair process. Similarly, the encoder features of the gradient information can be perceived in the decoding stage, which is conducive to the perception and utilization of the gradient information and improves the image repair effect.

[0140] S103 : extracting matching image features of the matching image, fusing the matching image features with the gradient features, and obtaining a restored image of the image to be restored.

[0141] Specifically, the matching image feature, the first gradient feature and the second gradient feature are fused to form a first mixed feature. Figure 3 As shown in the figure, before the image restoration module performs fusion, the large core attention block will first extract the matching image features of the matching image obtained by the alignment module, and then fuse the matching image features with the feature group obtained by the gradient feature module (i.e., the first gradient feature and the second gradient feature).

[0142] Optionally, in the process of extracting the first gradient feature and the second gradient feature, the image restoration module also uses the large core attention to extract and characterize the long-range relationship between the image to be restored and the matching image. The specific implementation process of the large core attention is the same as that of obtaining the first gradient feature and the second gradient feature as described above. Through the decomposition of a large core convolution operation, the long-range dependency relationship between the image to be restored and the matching image is obtained. Then, the first gradient feature and the second gradient feature are fused using the efficient receptive field of the large core convolution to form a first mixed feature. Here, the large core attention is also implemented by a large core attention block in the form of software or hardware in the image restoration module.

[0143] Then, the first mixed feature is converted from spatial domain information to frequency domain information using fast Fourier convolution. Fast Fourier convolution also has a wide receptive field and high perceptual loss, which can effectively extract global and local information of the image. Specifically, refer to Figure 4 As shown, the fast Fourier convolution consists of two branches: local branch and global branch. In the local branch, two-dimensional convolution is used (such as Figure 4The global branch uses a fast Fourier transform (FFT) in the spectral transform to convert the first mixed feature from spatial domain information to frequency domain information and perform convolution on the frequency domain information. Similarly, fast Fourier convolution is implemented using a fast Fourier block in software or hardware.

[0144] The spatial and frequency domain information of the first hybrid feature are fused to generate a second hybrid feature. After acquiring the spatial and frequency domain information, the information from these two branches is combined to form the second hybrid feature. This fully utilizes the frequency domain information to improve the ability to capture global and local features of the image.

[0145] The second mixed feature is spectrally transformed using a global branch of fast Fourier convolution to obtain a frequency domain feature.

[0146] refer to Figure 4 As shown in the spectrum transformation process, the input of the spectrum transformation is first processed by convolution + batch normalization + Relu activation function to obtain frequency domain information, and then the frequency domain information is processed by real part fast Fourier transform to obtain the first mixed feature, and then through convolution, batch normalization, Relu activation function processing, and finally the frequency domain information is obtained by inverse real part fast Fourier transform. The inverse real part fast Fourier transform result is point convolved by 1×1 point convolution to obtain frequency domain features.

[0147] The specific spectrum transformation process is as follows:

[0148] feat=Relu(BN(Conv(x))),

[0149] freq=FFT(feat),

[0150] freq=Relu(BN(Conv(freq))),

[0151] Finally, the frequency domain feature is expressed as:

[0152] out=Conv 1×1 (feat+IFFT(freq)),

[0153] Among them, Relu(·) represents the activation function, x represents the input of the spectral transformation, that is, the second mixed feature, Conv 1×1 Represents point convolution, feat represents the first mixed feature, FFT represents real part fast Fourier transform, IFFT represents real part inverse fast Fourier transform, fieq represents frequency domain information, GELU represents activation function, BN(-) represents batch normalization processing, and Conv represents convolution.

[0154] Finally, according to the frequency domain features, the repaired image of the image to be repaired is obtained, and the reference image is Figure 4As shown, after obtaining the frequency domain features, the global branch performs batch normalization processing and Relu activation function processing on the frequency domain features and the output of the local branch, thereby improving the capturing ability of the global and local features of the matching image and the image to be repaired, and obtaining the repaired image of the image to be repaired.

[0155] Repair result reference Figure 5a 、 5b , 5c, wherein Figure 5a is the image to be repaired, Figure 5b is the reference image, Figure 5c is the repaired image, thereby achieving effective repair of the image.

[0156] In general, in step S103, large kernel attention and fast Fourier convolution are used to effectively represent the encoder features, the decoder features and the matching image. Not only the degree of change of the first gradient features and the second gradient features with respect to the image gray value is emphasized, but also the processing combined with the matching image is emphasized, thereby improving the restoration ability of the complex structure and texture of the image.

[0157] In some possible scenarios, the present application is a pre-set image repair model, and the image repair model is trained in the form of an adversarial network, wherein the discriminator adopts a PatchGAN structure. The core of the PatchGAN discriminator is to refine the discrimination task of the entire image into the discrimination of multiple local regions in the image, so as to enhance the adversarial ability of the image repair model. The PatchGAN discriminator helps to more stably and quickly train the generator and the discriminator, so that the training process of the adversarial network is easier to reach convergence. The PatchGAN discriminator can also more sharply capture the local features of the image, further improving the effect of adversarial learning.

[0158] It should be noted that the method of the embodiment of the present application can be executed by a single device, such as a computer or a server. The method of the embodiment can also be applied to a distributed scenario, and completed by multiple devices cooperating with each other. In this distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiment of the present application, and the multiple devices can interact with each other to complete the method.

[0159] It should be noted that some embodiments of the present application have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order described above and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0160] Corresponding to the method of any of the above embodiments, the application also provides an image inpainting device based on the same technical concept.

[0161] Reference Figure 3 The image inpainting device comprises:

[0162] An alignment module is configured to acquire a to-be-inpainted image and a reference image, and align and match the to-be-inpainted image and the reference image to determine a matching image of the reference image relative to the to-be-inpainted image.

[0163] A gradient feature module is configured to input the to-be-inpainted image and the reference image by using a preset neural network model to determine gradient features of the to-be-inpainted image and the reference image.

[0164] An image inpainting module is configured to extract matching image features of the matching image, fuse the matching image features and the gradient features, and obtain an inpainted image of the to-be-inpainted image.

[0165] In some embodiments, the alignment module is further configured to,

[0166] construct first multi-scale images and second multi-scale images of the to-be-inpainted image and the reference image at corresponding scales respectively by using Gaussian convolution to construct an image pyramid;

[0167] determine extreme points of the first multi-scale images by comparing any pixel point in the first multi-scale images with pixel points in a neighborhood of the pixel point, and take the extreme points of the first multi-scale images as first key points of the first multi-scale images;

[0168] determine extreme points of the second multi-scale images by comparing any pixel point in the second multi-scale images with pixel points in a neighborhood of the pixel point, and take the extreme points of the second multi-scale images as second key points of the second multi-scale images;

[0169] calculate gradient directions and gradient amplitudes of the first key points according to positions of the first key points to determine first key point vectors;

[0170] calculate gradient directions and gradient amplitudes of the second key points according to positions of the second key points to determine second key point vectors;

[0171] map the second key point vectors to corresponding positions of the first key point vectors to determine the matching image of the reference image relative to the to-be-inpainted image.

[0172] In some embodiments, the alignment module is further configured to,

[0173] Calculating the similarity between any of the first key points and all of the second key points to obtain a plurality of second key points similar to any of the first key points;

[0174] The multiple second key points obtained by random sampling are randomly selected two points each time to fit into the first straight line, the interior points of the first straight line are determined and the exterior points are eliminated until the first straight line with the most interior points is obtained, and the second straight line is refitted using the obtained most interior points, and the interior point closest to the second straight line is determined as the second key point matching the first key point.

[0175] In some embodiments, the gradient feature module is further configured to:

[0176] Calculating the gradient amplitudes of the image to be restored and the reference image in the x-direction and the y-direction;

[0177] Extracting and fusing the gradient information of the image to be repaired and the reference image respectively through gated convolution;

[0178] Using large kernel attention to enlarge the receptive field and represent the long-range relationship of the gradient information;

[0179] According to the gradient magnitude and the long-range relationship, the encoder and the decoder of the preset neural network are used to respectively output a first gradient feature and a second gradient feature between the image to be restored and the reference image.

[0180] In some embodiments, the image restoration module is further configured to:

[0181] fusing the matching image feature, the first gradient feature, and the second gradient feature to form a first mixed feature;

[0182] Converting the first mixed feature from spatial domain information to frequency domain information using fast Fourier convolution;

[0183] Fusion

[0184] The spatial domain information and the frequency domain information of the first mixed feature are used to obtain a second mixed feature;

[0185] Performing spectral transformation on the second mixed feature using a global branch of fast Fourier convolution to obtain a frequency domain feature;

[0186] Obtaining a restored image of the image to be restored according to the frequency domain features;

[0187] The frequency domain feature is expressed as:

[0188] feat=Relu(BN(Conv(x))),

[0189] freq=FFT(feat),

[0190] freq=Relu(BN(Conv(freq))),

[0191] out=Conv 1×1 (feat+IFFT(freq)),

[0192] Among them, Conv 1×1 represents point convolution, feat represents mixed features, IFFT represents real part fast Fourier transform, and freq represents frequency domain information.

[0193] For the convenience of description, the above devices are described as being divided into various modules according to their functions. Of course, when implementing this application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0194] The apparatus of the above embodiment is used to implement the corresponding image restoration method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0195] Based on the same technical concept, corresponding to any of the above-mentioned embodiments and methods, the present application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and runnable on the processor, wherein when the processor executes the program, the image restoration method described in any of the above embodiments is implemented.

[0196] Figure 6 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0197] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0198] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0199] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0200] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0201] The bus 1050 comprises a path for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0202] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0203] The electronic device of the above embodiment is used to implement the corresponding image restoration method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0204] Based on the same technical concept, corresponding to any of the above-mentioned embodiment methods, the present application also provides a non-transitory computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable the computer to execute the image restoration method described in any of the above embodiments.

[0205] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0206] The computer instructions stored in the storage medium of the above embodiment are used to enable the computer to execute the image restoration method described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0207] Based on the same inventive concept, corresponding to the image restoration method described in any of the above embodiments, the present disclosure further provides a computer program product comprising computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processors to perform the image restoration method. For the execution entities corresponding to the steps in each embodiment of the image restoration method, the processors executing the corresponding steps can belong to the corresponding execution entities.

[0208] The computer program product of the above embodiment is used to enable the computer and / or the processor to execute the image restoration method described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0209] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present application (including the claims) is limited to these examples. Within the scope of the present application, the technical features in the above embodiments or different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0210] In addition, for simplicity of description and discussion, and in order not to make the embodiment of the application difficult to understand, the known power supply / ground connection with integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the application difficult to understand, and this also takes into account the following fact, that is, the details of the embodiment of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuit) are set forth to describe exemplary embodiments of the application, it will be apparent to those skilled in the art that the embodiment of the application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0211] Although the present invention has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may utilize the embodiments discussed.

[0212] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. An image restoration method, characterized in that: include: Acquire an image to be repaired and a reference image, and perform alignment and matching on the image to be repaired and the reference image to determine a matching image of the reference image relative to the image to be repaired; Using a preset neural network model, the image to be repaired and the reference image are input, and gradient features of the image to be repaired and the reference image are determined; Extracting matching image features of the matching image, fusing the matching image features with the gradient features, and obtaining a repaired image of the image to be repaired; The gradient feature includes a first gradient feature and a second gradient feature; Using the preset neural network model, inputting the image to be repaired and the reference image, and determining the gradient features of the image to be repaired and the reference image, including: Calculating the gradient amplitudes of the image to be restored and the reference image in the x-direction and the y-direction; Extracting and fusing the gradient information of the image to be repaired and the reference image respectively through gated convolution; Using large kernel attention to enlarge the receptive field and represent the long-range relationship of the gradient information; outputting a first gradient feature and a second gradient feature between the image to be restored and the reference image respectively using an encoder and a decoder of the preset neural network according to the gradient magnitude and the long-range relationship; The matching image feature and the gradient feature are integrated to obtain a repaired image of the image to be repaired, comprising: fusing the matching image feature, the first gradient feature, and the second gradient feature to form a first mixed feature; Converting the first mixed feature from spatial domain information to frequency domain information using fast Fourier convolution; fusing the spatial domain information and the frequency domain information of the first hybrid feature to obtain a second hybrid feature; Performing spectral transformation on the second mixed feature using a global branch of fast Fourier convolution to obtain a frequency domain feature; A restored image of the image to be restored is obtained according to the frequency domain features.

2. The method according to claim 1, characterized in that Acquiring an image to be repaired and a reference image, and performing alignment and matching on the image to be repaired and the reference image to determine a matching image of the reference image relative to the image to be repaired, including: Constructing an image pyramid using Gaussian convolution to respectively determine a first multi-scale image and a second multi-scale image at corresponding scales of the image to be restored and the reference image; comparing any pixel in the first multiscale image with pixels in a neighborhood, determining an extreme point of the first multiscale image and using the extreme point of the first multiscale image as a first key point of the first multiscale image; comparing any pixel in the second multiscale image with pixels in a neighborhood, determining an extreme point of the second multiscale image and using the extreme point of the second multiscale image as a second key point of the second multiscale image; Calculating the gradient direction and gradient magnitude of the first key point according to the position of the first key point to determine the first key point vector; Calculating the gradient direction and gradient magnitude of the second key point according to the position of the second key point to determine the second key point vector; The second key point vector is mapped to a position corresponding to the first key point vector, and a matching image of the reference image relative to the image to be repaired is determined.

3. The method according to claim 2, characterized in that Mapping the second key point vector to a position corresponding to the first key point vector to determine a matching image of the image to be repaired, the method further includes: Calculating the similarity between any of the first key points and all of the second key points to obtain a plurality of second key points similar to any of the first key points; The multiple second key points obtained by random sampling are randomly selected two points each time to fit into the first straight line, the interior points of the first straight line are determined and the exterior points are eliminated until the first straight line with the most interior points is obtained, and the second straight line is refitted using the obtained most interior points, and the interior point closest to the second straight line is determined as the second key point matching the first key point.

4. The method according to claim 1, wherein The frequency domain feature is expressed as: feat=Relu(BN(Conv(x))), freq=FFT(feat), freq=Relu(BN(Conv(freq))), out=Conv 1×1 (feat+IFFT(freq)), Among them, Relu(·) represents the activation function, x represents the input of the spectral transformation, Conv 1×1 denotes point convolution, feat denotes the first mixed feature, FFT denotes real fast Fourier transform, IFFT denotes real inverse fast Fourier transform, freq denotes frequency domain information, BN(·) denotes batch normalization, and Conv denotes convolution.

5. The method according to claim 2, characterized in that The direction and magnitude of the first key point are expressed as: θ(x1,y1)=tan -1 ((L(x1,y1+1)-L(x1,y1-1)) / (L(x1+1,y1)-L(x1-1,y1))), Where m(x1,y1) represents the amplitude of the first key point, θ(x1,y1) represents the direction of the first key point, and L(x1,y1) represents the pixel value of the first multi-scale image in the scale space of the Gaussian pyramid; The direction and magnitude of the second key point are expressed as: θ(x2,y2)=tan -1 ((L(x2,y2+1)-L(x2,y2-1)) / (L(x2+1,y2)-L(x2-1,y2))), Among them, m(x2,y2) represents the amplitude of the second key point, θ(x2,y2) represents the amplitude of the second key point, and L(x2,y2) represents the pixel value of the second multi-scale image in the scale space of the Gaussian pyramid.

6. The method according to claim 1, characterized in that The first gradient feature is expressed as: f e =GateConv i (inp) i∈1,2,3, Among them, GateConv i represents the gated convolution of the encoder of the neural network, The second gradient feature is expressed as: f d =Res∪GateTans i (inp) i∈1,2,3,4, Among them, Res represents the residual output result of the neural network, GateTans i Represents the gated convolution of the decoder of a neural network.

7. An image restoration device, characterized in that: include: an alignment module, configured to obtain an image to be restored and a reference image, and perform alignment and matching between the image to be restored and the reference image to determine a matching image of the reference image relative to the image to be restored; a gradient feature module, configured to use a preset neural network model to input the image to be repaired and the reference image, and determine the gradient features of the image to be repaired and the reference image; An image restoration module is configured to extract matching image features of the matching image, fuse the matching image features with the gradient features, and obtain a restoration image of the image to be restored; The gradient feature includes a first gradient feature and a second gradient feature; Using the preset neural network model, inputting the image to be repaired and the reference image, and determining the gradient features of the image to be repaired and the reference image, including: Calculating the gradient amplitudes of the image to be restored and the reference image in the x-direction and the y-direction; Extracting and fusing the gradient information of the image to be repaired and the reference image respectively through gated convolution; Using large kernel attention to enlarge the receptive field and represent the long-range relationship of the gradient information; outputting a first gradient feature and a second gradient feature between the image to be restored and the reference image respectively using an encoder and a decoder of the preset neural network according to the gradient magnitude and the long-range relationship; The matching image feature and the gradient feature are integrated to obtain a repaired image of the image to be repaired, comprising: fusing the matching image feature, the first gradient feature, and the second gradient feature to form a first mixed feature; Converting the first mixed feature from spatial domain information to frequency domain information using fast Fourier convolution; fusing the spatial domain information and the frequency domain information of the first hybrid feature to obtain a second hybrid feature; Performing spectral transformation on the second mixed feature using a global branch of fast Fourier convolution to obtain a frequency domain feature; A restored image of the image to be restored is obtained according to the frequency domain features.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable a computer to execute the method according to any one of claims 1 to 6.

Images