An image inpainting method, device, apparatus and medium

Abstract

The application discloses an image repairing method, device and equipment and a medium, and belongs to the field of image processing. The method comprises the following steps: obtaining an image to be repaired and information of a region to be repaired of the image to be repaired; inputting the image to be repaired and the information of the region to be repaired into a pre-established image repairing model to obtain a repaired image; the image repairing model comprises an encoder and a decoder, the number of encoder layers of the encoder is equal to the number of decoder layers of the decoder, and a gate connection is arranged between corresponding layers; the encoder layer encodes and processes the image to be repaired through a cross-layer feature fusion mode, the corresponding decoder layer decodes and processes according to a feature vector and the information of the region to be repaired, and image content of the region to be repaired is generated. The cross-layer feature fusion technology fully fuses the shallow features and the deep features of the image; the gate mechanism avoids the problems of gradient disappearance and gradient explosion, so that the model has strong robustness and effectively repairs the damaged image.

Description

Technical Field

[0001] This invention belongs to the field of image processing and relates to an image restoration method, apparatus, device, and medium. Background Technology

[0002] With the rapid development of technology, digital images play a crucial role in information transmission. However, images are easily damaged during acquisition, transmission, and storage, such as pixel loss, blurring, and noise interference. These damages severely affect image quality and the accuracy of information. Especially in fields like social media, commercial advertising, and medical imaging, the demands for image clarity and integrity are increasing, making image inpainting technology particularly important. While traditional image inpainting methods are effective for simple damage, their effectiveness is limited when faced with complex damage. In recent years, the rise of deep learning technology has brought revolutionary progress to the field of image inpainting. In particular, convolutional neural networks can learn the structural and textural information of images from large amounts of data, providing powerful support for image inpainting. However, existing deep learning image inpainting methods still face challenges when dealing with large-area missing areas or the need to fuse multi-scale features. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the prior art and provide an image restoration method, apparatus, device and medium to achieve efficient and accurate restoration of damaged images.

[0004] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0005] In a first aspect, the present invention provides an image restoration method, comprising:

[0006] Obtain the image to be repaired and the information of the region to be repaired in the image;

[0007] The image to be repaired and the information of the region to be repaired are input into a pre-established image repair model to obtain the repaired image;

[0008] The image restoration model includes an encoder and a decoder, wherein the encoder has an equal number of encoder layers and the decoder has an equal number of decoder layers and gated connections are provided between corresponding layers.

[0009] The encoder layer encodes the image to be repaired through cross-layer feature fusion to obtain the feature vector of the encoder layer. The decoder layer connected to it decodes the image based on the feature vector and the information of the region to be repaired to generate the image content of the region to be repaired.

[0010] Furthermore, the image to be repaired is subjected to data augmentation and interpolation scaling.

[0011] Furthermore, the encoder layer encodes the image to be repaired through cross-layer feature fusion to obtain the feature vector of the encoder layer, including:

[0012] ,

[0013] in, Indicates the first Feature vectors of each encoder layer Indicates the number of encoder layers. , Indicates after the first The output feature vector after convolution processing by each encoder layer; The cross-layer feature filling fusion function is calculated using the following formula:

[0014] f ( Φ i − 1 , Φ i ) = [ Φ i − 1 , Loc ( Φ i )] ,

[0015] in, Indicates to Perform interpolation upsampling.

[0016] Furthermore, the decoder layer performs decoding processing, including:

[0017] φ L = C L ( ϕ L ), φ L − 1 = C L − 1 ([ Loc ( φ L ), ϕ L − 1 ]), ...... φ i = C i ([ Loc ( φ i + 1 ), ϕ i ]), ...... φ 1 = C 1 ([ Loc ( φ 2 ), ϕ 1 ]) ,

[0018] in, Indicates the first The decoding results of each decoder layer Indicates the first The input feature vector of each decoder layer; Indicates the first Convolutional operations in each decoder layer; Indicates the number of decoder layers. .

[0019] Furthermore, the encoding process for the image to be repaired also includes: the encoder layer introducing a global attention mechanism, including:

[0020] No. Feature vectors of each encoder layer The formula for assigning adaptive attention weights is:

[0021] ,

[0022] in, express Adaptive attention weights; Represents the query matrix. This represents the attention matrix. Represents a numerical matrix. express The dimension; This represents the softmax function;

[0023] Will Adaptive attention weights and Perform residual join:

[0024] ;

[0025] The first Feature vectors of each encoder layer Divided into A portion, represented as:

[0026] ffn i = [ head 1 , head 2 ,..., head j ,..., head m ] ∗ W O ,

[0027] in, Indicates the first share Adaptive attention weights, ; Indicates the output transformation matrix;

[0028] Will and Perform residual connections to obtain the first result under global attention. Feature vectors of each encoder layer , Represented as:

[0029] .

[0030] Furthermore, a gating connection is established between the encoder layer and the decoder layer, including:

[0031] No. The gated output of each encoder layer is represented as follows:

[0032] ,

[0033] in, The function represents the gating weight parameter, and the calculation formula is:

[0034] ,

[0035] in, Indicates to Perform a convolution operation. The function is represented as: .

[0036] Furthermore, the loss function of the image restoration model for:

[0037] ,

[0038] in, Represents the adversarial loss function. The weights represent the adversarial loss function; Describes the L1 loss function. This represents the weights of the L1 loss function; Represents the perceptual loss function. The weights represent the perceptual loss function; Represents the style loss function, The weights represent the weights of the style loss function.

[0039] Secondly, the present invention also provides an image restoration apparatus, the apparatus comprising:

[0040] The image and defect information acquisition module is used to acquire the image to be repaired and the information of the area to be repaired in the image to be repaired;

[0041] The image restoration module is used to input the image to be restored and the information of the region to be restored into a pre-established image restoration model to obtain the restored image;

[0042] The image restoration model includes an encoder and a decoder, wherein the encoder has an equal number of encoder layers and the decoder has an equal number of decoder layers and gated connections are provided between corresponding layers.

[0043] The encoder layer encodes the image to be repaired through cross-layer feature fusion to obtain the feature vector of the encoder layer. The decoder layer connected to it decodes the image based on the feature vector and the information of the region to be repaired to generate the image content of the region to be repaired.

[0044] Thirdly, the present invention also provides a computer device, comprising:

[0045] Memory, used to store computer programs;

[0046] A processor for executing the computer program to implement the steps of the image restoration method described above.

[0047] Fourthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the steps of the above-described image restoration method.

[0048] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0049] The image restoration method provided by this invention fully utilizes the positional and edge information of shallow feature maps and the high-level semantic information of deep feature maps through cross-layer feature fusion. It introduces a global attention feature extraction method, which can capture the dependencies of features globally, efficiently extracting the contextual associations of input information and solving the long-distance dependency problem in feature extraction. Compared with the module connection method of traditional deep learning algorithms, this invention designs a gating mechanism that allows input signals to bypass some modules and directly affect the result, while also flexibly adjusting the weight parameters of different modules of the algorithm to avoid gradient vanishing and gradient exploding problems. This makes the model highly robust, achieving efficient and accurate restoration of damaged images. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating an image restoration method provided in an embodiment of the present invention;

[0051] Figure 2 This is a schematic diagram of the structure of an image restoration model in an image restoration method provided in an embodiment of the present invention;

[0052] Figure 3 This is a schematic diagram of the attention mechanism of the image restoration model in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram comparing the gating mechanism in this embodiment of the invention with the connection method of a classic deep learning module;

[0054] Figure 5 This is a schematic diagram of the generator-discriminator game training architecture in an embodiment of the present invention;

[0055] Figure 6 This is a schematic diagram of the structure of an image restoration device provided in an embodiment of the present invention;

[0056] Figure 7 An internal structural diagram of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0057] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. The same reference numerals in the drawings indicate the same or similar components or parts. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. The embodiments and specific features within the embodiments of this application are detailed descriptions of the technical solution of this application, and not limitations thereof. Where there is no conflict, the embodiments and technical features within the embodiments of this application can be combined with each other.

[0058] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0059] Example 1:

[0060] like Figures 1 to 5 As shown, this embodiment of the invention provides an image restoration method. Figure 1 This is a flowchart illustrating the image restoration method. This flowchart only shows the logical sequence of the method described in this embodiment. In other possible embodiments of the invention, different methods may be used, provided there are no conflicts. Figure 1 Complete the steps shown or described in the order indicated.

[0061] The circuit board defect detection method provided in this embodiment can be applied to a terminal and can be executed by a circuit board defect detection device. This device can be implemented by software and / or hardware and can be integrated into the terminal.

[0062] See Figure 1 The method of this embodiment specifically includes steps 1 to 2. Wherein:

[0063] Step 1: Obtain the image to be repaired and the information of the region to be repaired in the image to be repaired.

[0064] Among them, the information of the area to be repaired in the image to be repaired is the mask of the image to be repaired (the mask marks the missing pixel blocks in the image to be repaired, 0 indicates missing, and 1 indicates no missing).

[0065] The image to be repaired is preprocessed, including data augmentation and interpolation scaling.

[0066] The original input (image to be repaired) of this invention is a color image with RGB three channels, and the dimension is... To enable the image restoration model to adapt to different data distributions and have better robustness, this invention performs data augmentation operations:

[0067] ,

[0068] The F operation randomly selects a region from the image to be restored and crops it to a specified size, which helps the image restoration model learn features at different scales. The G operation horizontally flips the image to be restored with a certain probability, increasing the image restoration model's ability to perceive symmetry. The H operation slightly and randomly changes the brightness, contrast, saturation, and hue of the image to be restored, increasing the image restoration model's adaptability to color changes.

[0069] Data augmentation not only adds noise to the input data to enhance the model's resistance to interference, but also improves the model's robustness.

[0070] Next, the image to be repaired is interpolated and scaled to make its size fit the input requirements of the image restoration model.

[0071] In this embodiment of the invention, bilinear interpolation is used to interpolate and scale the image to be repaired. The formula for bilinear interpolation can be expressed as:

[0072] ,

[0073] in, express coordinate, express coordinate, express coordinate, express coordinate, It is based on , , and The result after interpolation of the four points.

[0074] The processed image is formatted and the pixel values ​​are normalized from [0,255] to the [-1,1] range. Then, an additional dimension is added to the image and mask to accommodate the input requirements of the image inpainting model.

[0075] Step 2: Input the image to be repaired and the information of the region to be repaired into a pre-established image repair model to obtain the repaired image;

[0076] like Figure 2 As shown, the image inpainting model includes an encoder and a decoder. The encoder has an equal number of encoder layers, and the decoder has an equal number of decoder layers, with corresponding layers interconnected. The encoder layer encodes the image to be inpainted, obtaining its feature vector. The decoder layer, connected to the encoder, decodes the image based on the feature vector and the information of the region to be inpainted, generating the image content of the region to be inpainted.

[0077] Specifically, the encoder layer encodes the image to be repaired using a cross-layer feature fusion method to obtain the feature vector of the encoder layer, including:

[0078]

[0079] in, Indicates the first Feature vectors of each encoder layer Indicates the number of encoder layers. , Indicates after the first Feature vectors after convolution processing by each encoder layer; The cross-layer feature filling fusion function is calculated using the following formula:

[0080] f ( Φ i − 1 , Φ i ) = [ Φ i − 1 , Loc ( Φ i )] ,

[0081] in, Indicates to Perform interpolation upsampling.

[0082] The decoder layer performs decoding processing, including:

[0083] φ L = C L ( ϕ L ), φ L − 1 = C L − 1 ([ Loc ( φ L ), ϕ L − 1 ]), ...... φ i = C i ([ Loc ( φ i + 1 ), ϕ i ]), ...... φ 1 = C 1 ([ Loc ( φ 2 ), ϕ 1 ])

[0084] in, Indicates the first Decoding results of each decoder layer; Indicates the first In this embodiment of the invention, the input feature vectors of each decoder layer are... express In other embodiments, the feature vectors after global attention and gating control are... It can be equal to , can also be equal to Feature vectors after global attention mechanisms or gating control; Indicates the first Convolutional operations in each decoder layer; Indicates the number of decoder layers. .

[0085] The encoding and decoding methods designed in this invention differ from classic deep learning methods. Classic deep learning algorithms typically use stacked convolutions to encode layer by layer from shallow to deep during the encoding process, and stacked deconvolutions to decode layer by layer from deep to shallow during the decoding process. This invention uses a multi-scale feature fusion encoding method, and the decoding process also fully utilizes the feature information extracted from multi-scale encoding, employing a cross-layer feature fusion method between deep decoding features and shallow encoding features to decode layer by layer from deep to shallow. On one hand, the generated reconstructed features encode more low-level information for missing regions. This design enables the decoder to produce visually realistic results with fine-grained details. On the other hand, the compressed latent features obtained through convolution can synthesize new targets in missing regions, even if no target can be found outside the missing regions. Combining these two aspects, the encoding and decoding methods proposed in this invention can synthesize new objects with highly coherent semantics and texture within the image context.

[0086] like Figure 3 As shown, this invention also innovatively uses a global attention feature extraction method to extract contextual information of image features to overcome the long-distance dependency problem, including:

[0087] No. Feature vectors of each encoder layer The formula for assigning adaptive attention weights is:

[0088] ,

[0089] in, express Adaptive attention weights; Represents the query matrix. This represents the attention matrix. Represents a numerical matrix. express The dimension; This represents the softmax function, which serves to smooth data.

[0090] Will Adaptive attention weights and Perform residual join:

[0091] ;

[0092] The first Feature vectors of each encoder layer Divided into A portion, represented as:

[0093] ffn i = [ head 1 , head 2 ,..., head j ,..., head m ] ∗ W O ,

[0094] in, It can be any constant. In the embodiments of the present invention, The value is 8; Indicates the first share Adaptive attention weights, ; Indicates the output transformation matrix;

[0095] Will and Perform residual connections to obtain the first result under global attention. Feature vectors of each encoder layer , Represented as:

[0096] .

[0097] like Figure 4 As shown, classic deep learning algorithms typically handle connections between modules in two ways. One is as follows: Figure 4 As shown in (a), the result of one module is directly used as the input of the next layer; another method is as follows: Figure 4 As shown in (b), the algorithm uses residual connections between modules. Residual connections allow the input signal to bypass some modules directly and be added to the outputs of those modules. The network no longer needs to learn a complete function mapping the input to the output, but instead learns a residual function, i.e., the difference between the input and the desired output. In this invention, gating control is designed between the corresponding encoder and decoder layers to control their connection method, such as... Figure 4 As shown in (c), the gating control of the present invention includes:

[0098] No. The output of each encoder layer is represented as follows:

[0099] ,

[0100] in, The function represents the gating weight parameter, and the calculation formula is:

[0101] ,

[0102] in, Indicates to Perform a convolution operation. The function is used to prevent gradient vanishing, and its calculation formula is:

[0103] ,

[0104] when hour, The derivative of the function is 1, so there is no gradient decay issue; when When the function value is 0, the intermediate output of the neural network is sparse, preventing overfitting. Furthermore, because the computation is simple, the function converges quickly.

[0105] In this embodiment of the invention, an adaptive learning rate can be calculated for each parameter by training the optimizer through game theory. Initially, two vectors m and v are initialized, where m is used to store the moving average of the gradient, and v tracks the moving average of the squared gradient.

[0106] For each iteration step, calculate the gradient. :

[0107] ,

[0108] Therefore, update the first moment vector m of the moving average of the stored gradient:

[0109] ,

[0110] Then, update the second-order moment vector v of the squared cumulative gradient:

[0111] ,

[0112] To ensure the representativeness of the moving average, a bias correction is performed:

[0113] ,

[0114] ,

[0115] Shift the parameters in the direction that minimizes the loss function:

[0116] .

[0117] like Figure 5 As shown, the loss function of the image restoration model of the present invention for:

[0118] ,

[0119] in, Represents the adversarial loss function. The weights represent the adversarial loss function; Describes the L1 loss function. This represents the weights of the L1 loss function; Represents the perceptual loss function. The weights represent the perceptual loss function; Represents the style loss function, The weights represent the weights of the style loss function.

[0120] Specifically, assuming the image inpainting model performs m-masking on the image x to be inpainted, the resulting image z is:

[0121] ,

[0122] A soft mask is obtained around the boundary of the missing region using Gaussian filtering. The adversarial loss of the discriminator is:

[0123] L adv D = E z ~ pz [( D ( z ) − τ ( 1 − m )) 2 ] + E z ~ pdata [( D ( x ) − 1 ) 2 ] ,

[0124] in, Let be the composite function of downsampling and Gaussian filtering. Correspondingly, the adversarial loss of the generator is denoted as:

[0125] L adv G = E z ~ pz [( D ( x ) − 1 ) 2 ⊙ m ] .

[0126] This invention also employs L1 loss to ensure pixel-level reconstruction accuracy:

[0127] ,

[0128] Perceptual loss can minimize the L1 distance between the activation maps of hand-drawn images and real images:

[0129] ,

[0130] Style loss is the L1 distance between the Gram matrix of depth features in the hand-drawn image and the real image:

[0131] L sty = E i [|| ϕ i ( x ) T ϕ i ( x ) − ϕ i ( z ) T ϕ i ( z )|| 1 ] .

[0132] Example 2:

[0133] Based on the same inventive concept as Embodiment 1, this embodiment of the invention also provides an image restoration apparatus for implementing the above-described image restoration method. The solution provided by this apparatus is similar to the implementation scheme described in the above-described method; therefore, the specific limitations in the image restoration apparatus embodiments provided below can be found in the limitations of the image restoration method described above, and will not be repeated here.

[0134] like Figure 6 As shown, an embodiment of the present invention provides an image restoration device, comprising:

[0135] The image and defect information acquisition module is used to acquire the image to be repaired and the information of the area to be repaired in the image to be repaired;

[0136] The image restoration module is used to input the image to be restored and the information of the region to be restored into a pre-established image restoration model to obtain the restored image;

[0137] The image restoration model includes an encoder and a decoder, wherein the encoder has an equal number of encoder layers and the decoder has an equal number of decoder layers and gated connections are provided between corresponding layers.

[0138] The encoder layer encodes the image to be repaired through cross-layer feature fusion to obtain the feature vector of the encoder layer. The decoder layer connected to it decodes the image based on the feature vector and the information of the region to be repaired to generate the image content of the region to be repaired.

[0139] Example 3:

[0140] This invention also provides a computer device, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements the image restoration method described in the preceding embodiments.

[0141] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] Example 4:

[0143] This invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the following method:

[0144] Obtain the image to be repaired and the information of the region to be repaired in the image;

[0145] The image to be repaired and the information of the region to be repaired are input into a pre-established image repair model to obtain the repaired image;

[0146] The image restoration model includes an encoder and a decoder, wherein the encoder has an equal number of encoder layers and the decoder has an equal number of decoder layers and gated connections are provided between corresponding layers.

[0147] The encoder layer encodes the image to be repaired through cross-layer feature fusion to obtain the feature vector of the encoder layer. The decoder layer connected to it decodes the image based on the feature vector and the information of the region to be repaired to generate the image content of the region to be repaired.

[0148] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0150] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0152] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. An image restoration method, characterized in that, include: Obtain the image to be repaired and the information of the region to be repaired in the image; The image to be repaired and the information of the region to be repaired are input into a pre-established image repair model to obtain the repaired image; The image restoration model includes an encoder and a decoder, wherein the encoder has an equal number of encoder layers and the decoder has an equal number of decoder layers and gated connections are provided between corresponding layers. The encoder layer encodes the image to be repaired through cross-layer feature fusion to obtain the feature vector of the encoder layer. The decoder layer connected to it performs decoding based on the feature vector and the information of the region to be repaired to generate the image content of the region to be repaired. The encoding process for the image to be repaired includes: the encoder layer introducing a global attention mechanism, including: No. Feature vectors of each encoder layer The formula for assigning adaptive attention weights is: ， in, express Adaptive attention weights; Represents the query matrix. This represents the attention matrix. Represents a numerical matrix. express The dimension; This represents the softmax function; Will Adaptive attention weights and Perform residual join: ； The first Feature vectors of each encoder layer Divided into A portion, represented as: ， in, Indicates the first share Adaptive attention weights ; Indicates the output transformation matrix; Will and Perform residual connections to obtain the first result under global attention. Feature vectors of each encoder layer , Represented as: ； The encoder layer and decoder layer are connected via a gated connection, including: No. The gated output of each encoder layer is represented as follows: ， in, The function represents the gating weight parameter, and the calculation formula is: ， in, Indicates to Perform a convolution operation. The function is represented as: .

2. The image restoration method according to claim 1, characterized in that, Also includes: Data augmentation and interpolation scaling are performed on the image to be repaired.

3. The image restoration method according to claim 1, characterized in that, The encoder layer encodes the image to be repaired through cross-layer feature fusion to obtain the feature vector of the encoder layer, including: ， in, Indicates the first Feature vectors of each encoder layer Indicates the number of encoder layers. ; Indicates after the first Feature vectors after convolution processing by each encoder layer; The cross-layer feature filling fusion function is calculated using the following formula: ， in, Indicates to Perform interpolation upsampling.

4. The image restoration method according to claim 1, characterized in that, The decoder layer performs decoding processing, including: ， in, Indicates the first The decoding results of each decoder layer Indicates the first The input feature vector of each decoder layer; Indicates the first Convolution operations in each decoder layer; Indicates the number of decoder layers. .

5. The image restoration method according to claim 1, characterized in that, The loss function of the image restoration model for: ， in, Represents the adversarial loss function. The weights represent the adversarial loss function; Describes the L1 loss function. This represents the weights of the L1 loss function; Represents the perceptual loss function. The weights represent the perceptual loss function; Represents the style loss function, The weights represent the weights of the style loss function.

6. An image restoration device, characterized in that, include: The image and defect information acquisition module is used to acquire the image to be repaired and the information of the area to be repaired in the image to be repaired; The image restoration module is used to input the image to be restored and the information of the region to be restored into a pre-established image restoration model to obtain the restored image; The image restoration model includes an encoder and a decoder, wherein the encoder has an equal number of encoder layers and the decoder has an equal number of decoder layers and gated connections are provided between corresponding layers. The encoder layer encodes the image to be repaired through cross-layer feature fusion to obtain the feature vector of the encoder layer. The decoder layer connected to it performs decoding based on the feature vector and the information of the region to be repaired to generate the image content of the region to be repaired. The encoding process for the image to be repaired includes: the encoder layer introducing a global attention mechanism, including: No. Feature vectors of each encoder layer The formula for assigning adaptive attention weights is: ， in, express Adaptive attention weights; Represents the query matrix. This represents the attention matrix. Represents a numerical matrix. express The dimension; This represents the softmax function; Will Adaptive attention weights and Perform residual join: ； The first Feature vectors of each encoder layer Divided into A portion, represented as: ， in, Indicates the first share Adaptive attention weights ; Indicates the output transformation matrix; Will and Perform residual connections to obtain the first result under global attention. Feature vectors of each encoder layer , Represented as: ； The encoder layer and decoder layer are connected via a gated connection, including: No. The gated output of each encoder layer is represented as follows: ， in, The function represents the gating weight parameter, and the calculation formula is: ， in, Indicates to Perform a convolution operation. The function is represented as: .

7. A computer device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the image restoration method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the image restoration method according to any one of claims 1 to 5.

Images