Image restoration method, system, device and medium integrating cross attention

Abstract

The present application discloses an image restoration method, system, device and medium integrating cross-attention, and relates to the field of image restoration. The method includes inputting a damaged image into an image restoration model to obtain a restored image. The training method of the image restoration model is as follows: construct a training data set, whose sample data includes damaged images and corresponding true images. Construct a target network including a generator network, a discriminator network and a multi-receptive field discriminator network; wherein the generator network includes an encoder and a decoder; the decoder includes a fused decoding layer formed by fusing multiple layers of cross-attention layers with convolutional layers, which is used to learn the explicit dependency between potential features and true images. The target network parameters are trained and optimized to obtain an image restoration model. The present application uses a cross-attention mechanism to enable the image restoration model to better understand the relationship between the damaged image and the true image, thereby improving the quality of the restored image.

Description

Technical Field

[0001] The present application relates to the field of image restoration, and in particular to an image restoration method, system, device and medium integrating cross-attention. Background Art

[0002] Image inpainting aims to use the intact areas of the image to generate missing areas with semantically reasonable and visually realistic content. It is one of the key research topics in low-level vision tasks and has a wide range of applications in image dehazing, image recognition, face editing, and restoration of precious historical data.

[0003] Early image restoration methods fill in missing areas by propagating local information from adjacent visible areas or searching for matching patches; however, such methods often ignore image semantic information, resulting in unreasonable restoration results. In recent years, the development of deep learning technology, especially the use of deep neural networks and adversarial learning ideas, has promoted the progress of image restoration research. For example, a context encoder with an encoder-decoder structure is used to fill regular invisible areas; however, even with the use of a context encoder, the generated output image usually contains obvious visual artifacts. To solve this problem, researchers proposed a contextual attention method that uses an embeddable attention module to transfer features from non-missing areas to missing areas; however, traditional attention mechanisms require roughly complete feature information, and it is difficult to obtain sufficient information in the case of large area missing, resulting in such improvements being insufficient to generate reasonable semantics. To this end, the emergence of complex models with intermediate predictions, such as two-stage or multi-stage progressive networks, solves the above problems. Such methods achieve matching and propagation between long-distance features during deep feature extraction through convolution; however, in terms of spatial feature extraction and propagation, learning adaptive matching weights to achieve long-distance dependencies remains a challenge.

[0004] With the development of the self-attention mechanism, image restoration methods based on the Transformer architecture can effectively solve the matching and propagation problems between long-distance features; however, existing methods often apply Transformer to the "bottleneck" area of ​​the encoder or generator, and rarely to the decoder stage. During the forward propagation of the decoder network parameters, only potential features are considered as input, and the explicit dependency between the potential features and the real image is not fully considered, which leads to the limited learning ability of the decoder, thus affecting the quality of the restored image. Summary of the invention

[0005] The purpose of this application is to provide an image restoration method, device, equipment, and medium that integrates cross-attention, which can solve the problem of poor image restoration quality in the prior art. By using the cross-attention mechanism, the restoration model can better understand the relationship between the damaged image and the true image, thereby improving the quality of the restored image.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides an image restoration method integrating cross attention, comprising:

[0008] Input the damaged image into the image restoration model to obtain the restored image;

[0009] The training method of the image restoration model is as follows:

[0010] Constructing a training data set, wherein each sample of the training data set includes a damaged image and a corresponding true value image;

[0011] Constructing a target network, the target network includes a generator network, a discriminator network and a multi-receptive field discriminator network; wherein the output end of the generator network is connected to the discriminator network and the multi-receptive field discriminator network respectively; the generator network includes an encoder and a decoder; the decoder includes multiple layers of fused decoding layers, each of which is a decoding layer fused by a cross attention layer and a convolution layer, and the multiple layers of fused decoding layers are connected in series; the decoder is used to learn the explicit dependency relationship between the potential feature and the true value image; the potential feature is the feature output by the encoder;

[0012] Each sample of the training data set is used as the input of the target network, and the parameters of the target network are trained and optimized to obtain an image restoration model.

[0013] In a second aspect, the present application provides an image restoration system integrating cross-attention, comprising:

[0014] A data acquisition module, used for acquiring damaged images;

[0015] The image restoration module inputs the damaged image into the image restoration model to obtain the restored image;

[0016] The training method of the image restoration model is as follows:

[0017] Constructing a training data set, wherein each sample of the training data set includes a damaged image and a corresponding true value image;

[0018] Constructing a target network, the target network includes a generator network, a discriminator network and a multi-receptive field discriminator network; wherein the output end of the generator network is connected to the discriminator network and the multi-receptive field discriminator network respectively; the generator network includes an encoder and a decoder; the decoder includes multiple layers of fused decoding layers, each of which is a decoding layer fused by a cross attention layer and a convolution layer, and the multiple layers of fused decoding layers are connected in series; the decoder is used to learn the explicit dependency relationship between the potential feature and the true value image; the potential feature is the feature output by the encoder;

[0019] Each sample of the training data set is used as the input of the target network, and the parameters of the target network are trained and optimized to obtain an image restoration model.

[0020] In a third aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image restoration method integrating cross-attention.

[0021] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image restoration method integrating cross-attention.

[0022] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0023] The present application proposes an image restoration method, system, device and medium that integrates cross-attention. The method includes inputting a damaged image into an image restoration model to obtain a restored image; the training method of the image restoration model is as follows: constructing a training data set, each sample of the training data set includes a damaged image and a corresponding true value image; constructing a target network, the target network includes a generator network, a discriminator network and a multi-receptive field discriminator network; wherein the output data of the generator network is used to input the discriminator network and the multi-receptive field discriminator network; the generator network includes an encoder and a decoder; the decoder includes multiple layers of fused decoding layers connected in sequence, which are used to learn the explicit dependency relationship between potential features and true value images, the potential features are the features output by the encoder; the fused decoding layer is formed by the fusion of a cross-attention layer and a convolutional layer; each sample of the training data set is used as the input of the target network, and the parameters of the target network are trained and optimized to obtain an image restoration model.

[0024] The decoder in this application can learn the explicit dependency between the latent features of the damaged image and the true image by fusing the cross-attention layer and the convolutional layer, so that the repair model can better understand the relationship between the damaged image and the true image, and thus repair the damaged or missing parts in a targeted manner. The repaired image is more visually coherent and realistic, thereby improving the quality of the repaired image.

[0025] Moreover, the use of a multi-receptive field discriminator network can evaluate the quality of the generated images at different scales, thereby guiding the generator network to learn more structured and realistic restorations, allowing the generator to more accurately simulate the statistical characteristics of real images, avoiding distortion or unnaturalness in the restored images, and further improving the quality of the restored images. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 This is an application environment diagram of an image restoration method integrating cross-attention in one embodiment of the present application;

[0028] Figure 2 A schematic diagram of an image restoration model provided in one embodiment of the present application;

[0029] Figure 3 A flowchart of a training method for an image restoration model provided in one embodiment of the present application;

[0030] Figure 4 A schematic diagram of subjective comparison results obtained by testing different image restoration models on the Places2 dataset provided in an embodiment of the present application;

[0031] Figure 5 A schematic diagram of subjective comparison results obtained by testing different image restoration models on the CelebA-HQ dataset provided in an embodiment of the present application;

[0032] Figure 6 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0034] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0035] The image restoration method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, or it can be integrated on the server 104, or it can be placed on the cloud or other servers. The terminal 102 can send the damaged image to the server 104. After the server 104 receives the damaged image, for the damaged image, the server 104 inputs the damaged image into the image restoration model to obtain the restored image. The server 104 can feed back the restored image to the terminal 102. In addition, in some embodiments, the image restoration method can also be implemented by the server 104 or the terminal 102 alone, such as the terminal 102 can directly perform image restoration processing on the damaged image, or the server 104 can obtain the damaged image from the data storage system and perform image restoration processing on the damaged image.

[0036] Among them, the terminal 102 can be but is not limited to various desktop computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the server 104 can be implemented by an independent server or a server cluster composed of multiple servers, and can also be a cloud server.

[0037] In an exemplary embodiment, Figure 2 As shown, a method for image restoration integrating cross attention is provided, which is executed by a computer device, and can be executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 Taking the server 104 in the example as an example, the main steps include:

[0038] Step 1: Input the damaged image into the image restoration model to obtain the restored image.

[0039] The image restoration model training method includes the following steps 201 to 205: Figure 2and Figure 3 As shown, where:

[0040] Step 201: construct a training data set, each sample of which includes a damaged image I in , and the corresponding true value image I gt .

[0041] Step 202, constructing a target network, the target network includes a generator network G, a discriminator network D1 and a multi-receptive field discriminator network D2; wherein the output end of the generator network G is connected to the discriminator network D1 and the multi-receptive field discriminator network D2 respectively.

[0042] Furthermore, the generator network G includes an encoder and a decoder, the input end of the generator network is the input of the encoder, and the input data of the generator network G is I in =I gt e(1-M); where M is a mask, which is a binary rectangular area (the value of the pixel in the missing area is 0, and the value of the pixel in the known area is 1). The encoder includes a convolutional layer; the encoder is used to in Convolution and downsampling operations are performed to obtain potential features I. The decoder includes multiple layers of fused decoding layers, each of which is a decoding layer fused by a cross attention layer and a convolution layer, and the multiple layers of fused decoding layers are connected in series; the decoder is used to learn the potential features I and the true value image I gt The explicit dependency between them is used to upsample the latent feature I to the full RGB image.

[0043] Furthermore, the first fusion decoding layer of the decoder includes a first cross attention layer and a first convolution layer; the input of the first cross attention layer includes three parts, one part comes from the true value image I gt， The other two parts come from latent feature I.

[0044] The first cross attention layer is used to perform 1×1 convolution and linear operation on the potential features and the true value image to obtain three feature spaces, namely, the first feature space a=W a Conv(I gt ), the second feature space b = W b Conv(I), the third feature space c = W c Conv(I); where W a , W b and W c is a learnable weight matrix.

[0045] The cross attention weights are calculated for the first feature space a and the second feature space b to obtain the cross attention weights; the cross attention weights represent the weights of the model for synthesizing each pixel of the potential feature I for the true value image I gt The specific calculation process of the cross attention weight is:

[0046] s=softmax(ab T )(1);

[0047] And according to the cross attention weight s, the learnable weight matrix W d And the third feature space c, calculate the output data d of the first cross attention layer, the calculation formula is as follows:

[0048] d=W d ·s·c(2);

[0049] The first convolutional layer is used to:

[0050] The output data of the first cross attention layer is convolved to obtain the convolution feature value Dec(d), and the convolution feature value is fused with the potential feature I to obtain the output data of the first fusion decoding layer; wherein Dec(·) represents the convolution operation of the decoder; the specific process of the fusion calculation is:

[0051] I d =Concat(I,Dec(d))(3);

[0052] In the decoder, the structure of each fusion decoding layer is the same. Except for the first fusion decoding layer, the input data of each other fusion decoding layer is the output data and the true value image of the previous fusion decoding layer, until the output data of the decoder is obtained through the last fusion decoding layer;

[0053] Finally, the output data of the decoder is subjected to convolution and activation operations to obtain the output data I of the generator. out .

[0054] Furthermore, the discriminator in the discriminator network D1 is a spectral normalized Markov discriminator, which focuses on pixel-level constraints. The discriminator includes 5 convolutional layers, and the last layer is a fully connected layer.

[0055] Furthermore, the multi-receptive field discriminator network D2 is a network obtained by introducing multiple layers of sequentially connected dilated convolutional layers on the basis of the discriminator network, wherein the multiple layers of dilated convolutional layers are embedded between the convolutional layers, and the dilation rate of each dilated convolutional layer is different. The discriminator in the multi-receptive field discriminator network D2 in the present application includes 5 convolutional layers. After the 3rd convolutional layer, 3 dilated convolutional layers are embedded to increase the receptive field, focusing on the understanding of the global structure, and the dilation rates of the 3 dilated convolutional layers are 2, 4, and 8, respectively.

[0056] In addition, the discriminator network D1 and the multi-receptive field discriminator network D2 are trained using a joint loss to obtain a joint loss value; the joint loss includes a multi-receptive field feature-matching (MRF) loss, an adversarial loss, a reconstruction loss, and a style loss; the calculation process of the joint loss is:

[0057]

[0058] in, is the joint loss value, To combat the loss value, To reconstruct the loss value, Style loss value, is the MRF loss value, is the feature matching loss of the discriminator network; r , s and λ f is the balancing factor.

[0059] Specifically, the MRF loss is used to constrain the structure; the MRF loss is obtained by applying the feature matching loss to the activation map of the unfolded convolutional layer in the MRF subnet; the MRF loss is defined as:

[0060]

[0061] Among them, I out is the output of the generator, L represents the number of expanded convolutional layers in the MRF subnet, and N i is the number of elements in the i-th activation layer, HRF (i) is the activation feature of the i-th layer of the MRF subnet.

[0062] The training purpose of the discriminator network D1 and the multi-receptive field discriminator network D2 is to drive the generator to achieve the goal by distinguishing the real situation in the generated image, and the adversarial loss plays a key role in the game process. The adversarial loss can be expressed as:

[0063]

[0064] in, I gt Obey P data expectations, I out Obey P G The expected D1(I out ) indicates that after being judged by the discriminator network D1, I out From P G The probability of D1(I gt ) indicates that after being judged by the discriminator network D1, I gt From P data The probability of D2(I out ) indicates that after being discriminated by the multi-receptive field discriminator network D2, I out From P G The probability of D2(I gt ) After being judged by the multi-receptive field discriminator network D2, I gt From P data The probability of data and P G represent the real data distribution and the generated data distribution respectively.

[0065] The reconstruction loss includes pixel-level L1 loss and VGG loss; L1 loss ensures reconstruction accuracy by calculating the similarity of pixels at each scale, and VGG loss constrains information by extracting deep features. out (i.e., the image generated by the generator) and the true value image I gt Perform feature extraction and make a difference comparison between the extracted features. The difference is the VGG loss. The reconstruction loss can be defined as:

[0066]

[0067] Among them, VGG h (·) is the activation feature map of the selected h-th VGG network layer.

[0068] The style loss constrains the consistency of the generated image style with the true value image style; before applying the L1 loss, an autocorrelation operation (Gram matrix) is performed on each specific VGG feature map. The style loss can be defined as:

[0069]

[0070] Among them, ψ n (·) is the activation feature map of the nth selected layer.

[0071] Step 203: Use each sample of the training data set as the input of the target network, train and optimize the parameters of the target network, and obtain an image restoration model.

[0072] The image restoration model described in this application was tested on the Places2 and CelebA-HQ datasets using irregular masks, and the image restoration model described in this application was evaluated and compared with the mainstream image restoration models in the prior art from the perspectives of objective quantification and subjective perception. The mainstream image restoration models in the prior art include the GMCNN model, the PEN-Net model, the EC model, the HiFill model, and the GDS model.

[0073] From the objective comparison experiments, it can be concluded that the image restoration model described in the present application has achieved superior average values ​​in terms of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) and Fréchet Inception Distance (FID) evaluation indicators compared with the mainstream image restoration models in the prior art. Tables 1 and 2 give the quantitative evaluation results of the Places2 and CelebA-HQ datasets, respectively; the comparative experiments were all conducted using irregular masks. As shown in Table 1, for the Places2 dataset, the image restoration model described in the present application produced the best results in all evaluation indicators; compared with the PEN-Net image restoration model, the image restoration model described in the present application has an average PSNR improvement of 10%, and compared with the HiFill image restoration model, the image restoration model described in the present application has an average SSIM value improvement of 5%. In addition, the image restoration model described in the present application makes the average FID value 22.69, which is 42% lower than the GDS image restoration model. As shown in Table 2, for the CelebA-HQ dataset, the image restoration model described in this application achieves the highest average PSNR value and SSIM value.

[0074] For subjective comparison experiments, the image restoration model described in this application is superior to the mainstream image restoration model in the prior art in terms of structure and detail restoration. Subjective comparison results were obtained by testing different image restoration models on the Places2 and CelebA-HQ datasets. Figure 4 It can be seen that the results generated by the image restoration model described in this application are more semantically reasonable than those of the PEN-Net image restoration model. In contrast, the results generated by the image restoration model described in this application are more natural and smoother than those generated by the HiFill image restoration model, while the images generated by the HiFill image restoration model appear blurred and distorted. The EC and GDS image restoration models perform well when the content of the missing area is simple, but show semantic loss or visual blur when the structure of the missing area is complex. In addition, Figure 4The results in the fourth row (floor diamond area restoration) also show that the image restoration model described in this application has the advantage of better consistent structure reconstruction in terms of smooth transition from background area to missing area. The results obtained using different image restoration models on the CelebA-HQ dataset are shown in Figure 2. Figure 5 As shown. It can be seen that although the GMCNN image restoration model produces structurally reasonable results, there are artifacts such as water ripples in the hair area. The results generated by the PEN-Net and GDS image restoration models are often blurry. The structure generated by the EC image restoration model is inconsistent with the background, especially the eyes. Compared with the mainstream image restoration model in the prior art, the image restoration model described in this application generates results with clear facial features and better color consistency.

[0075] Table 1 Quantitative evaluation results on the Places2 dataset

[0076] Evaluation indicators <![CDATA[PSNR ↑ ]]> <![CDATA[SSIM ↑ ]]> FID↓ PEN-Net Model 23.69 0.868 50.99 EC Model 26.22 0.880 22.82 HiFill Model 23.95 0.855 48.32 GDS Model 24.54 0.876 39.08 SGGAN model 26.44 0.900 22.69

[0077] Table 2 Quantitative evaluation results on the CelebA-HQ dataset

[0078] Evaluation indicators <![CDATA[PSNR ↑ ]]> <![CDATA[SSIM ↑ ]]> FID↓ GMCNN Model 27.23 0.922 31.58 PEN-Net Model 25.04 0.907 57.73 EC Model 28.89 0.934 18.78 GDS Model 26.39 0.920 30.42 SGGAN model 28.97 0.935 21.60

[0079] Among them, the peak signal-to-noise ratio (PSNR) indicator can evaluate the L2 distance between the true value and the generated image pixels, the structural similarity (SSIM) indicator evaluates the structural similarity between the true value and the generated image, and the Fréchet inception distance (FID) simulates human perception of similarity. Moreover, '↑' means the larger the value, the better, and '↓' means the smaller the value, the better.

[0080] The present application also provides an application scenario, which applies the above-mentioned image restoration method. Specifically: The image restoration method provided in this embodiment can be applied in an image restoration display scenario. The image restoration display scenario includes an image restoration link and a display link. The image restoration method provided in this embodiment is used in the image restoration link.

[0081] Based on the same inventive concept, the embodiment of the present application also provides an image restoration system for implementing the above-mentioned image restoration method. The implementation scheme for solving the problem provided by the system is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more image restoration system embodiments provided below can refer to the limitations on the image restoration method above, and will not be repeated here.

[0082] In an exemplary embodiment, a cross-attention-fused image restoration system is provided, comprising:

[0083] The data acquisition module is used to acquire the damaged image.

[0084] The image restoration module inputs the damaged image into the image restoration model to obtain the restored image.

[0085] The training method of the image restoration model is as follows:

[0086] Constructing a training data set, wherein each sample of the training data set includes a damaged image and a corresponding true value image;

[0087] Constructing a target network, the target network includes a generator network, a discriminator network and a multi-receptive field discriminator network; wherein the output end of the generator network is connected to the discriminator network and the multi-receptive field discriminator network respectively; the generator network includes an encoder and a decoder; the decoder includes multiple layers of fused decoding layers, each of which is a decoding layer fused by a cross attention layer and a convolution layer, and the multiple layers of the fused decoding layers are connected in series; the decoder is used to learn the explicit dependency relationship between the potential features and the true value image;

[0088] Each sample of the training data set is used as the input of the target network, and the parameters of the target network are trained and optimized to obtain an image restoration model.

[0089] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image restoration data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image restoration method integrating cross-attention is implemented.

[0090] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0091] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0092] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0093] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0094] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0095] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0096] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An image restoration method integrating cross attention, characterized in that: The image restoration method integrating cross attention includes: Input the damaged image into the image restoration model to obtain the restored image; The training method of the image restoration model is as follows: Constructing a training data set, wherein each sample of the training data set includes a damaged image and a corresponding true value image; Constructing a target network, the target network includes a generator network, a discriminator network and a multi-receptive field discriminator network; wherein the output end of the generator network is connected to the discriminator network and the multi-receptive field discriminator network respectively; the generator network includes an encoder and a decoder; the decoder includes multiple layers of fused decoding layers, each of which is a decoding layer fused by a cross-attention layer and a convolution layer, and the multiple layers of fused decoding layers are connected in series; the decoder is used to learn the explicit dependency relationship between the potential feature and the true value image; the potential feature is the feature output by the encoder; Taking each sample of the training data set as the input of the target network, training and optimizing the parameters of the target network to obtain an image restoration model; The first fusion decoding layer of the decoder includes a first cross attention layer and a first convolutional layer; The first cross attention layer is used to: Performing convolution and linear operations on the potential features and the true value image to obtain a first feature space, a second feature space, and a third feature space; Calculating cross-attention weights on the first feature space and the second feature space to obtain cross-attention weights; Calculate output data of the first cross attention layer according to the cross attention weight and the third feature space; The first convolutional layer is used to: Perform a convolution operation on the output data of the first cross attention layer to obtain a convolution feature value; The convolution feature value is fused and calculated with the potential feature to obtain output data of the first fusion decoding layer; In the decoder, except for the first fusion decoding layer, the input data of each other fusion decoding layer are the output data of the previous fusion decoding layer and the true value image; The discriminator in the discriminator network is a spectral normalized Markov discriminator, and the discriminator includes a convolutional layer and a fully connected layer; The multi-receptive field discriminator network is a network obtained by embedding multiple layers of sequentially connected dilated convolutional layers between the convolutional layers of the discriminator network; the dilation rate of each dilated convolutional layer is different; The discriminator network and the multi-receptive field discriminator network are trained using a joint loss to obtain a joint loss value; the joint loss includes a multi-receptive field feature matching loss, an adversarial loss, a reconstruction loss, and a style loss; the calculation process of the joint loss is: ； in, is the joint loss value, To combat the loss value, To reconstruct the loss value, Style loss value, is the multi-receptive field feature matching loss value, is the feature matching loss of the discriminator network; , and is the balancing factor.

2. The image restoration method integrating cross-attention according to claim 1, characterized in that: The encoder comprises a convolution layer; the encoder is used to perform convolution and downsampling operations on the damaged image to obtain the potential features.

3. The image restoration method integrating cross-attention according to claim 1, characterized in that: The generator network is also used to perform convolution operations and activation operations on the output data of the decoder to obtain the output data of the generator.

4. An image restoration system integrating cross attention, characterized in that: The image restoration method integrating cross-attention according to any one of claims 1 to 3 is applied, and the image restoration system integrating cross-attention comprises: A data acquisition module, used for acquiring damaged images; The image restoration module inputs the damaged image into the image restoration model to obtain the restored image; The training method of the image restoration model is as follows: Constructing a training data set, wherein each sample of the training data set includes a damaged image and a corresponding true value image; Constructing a target network, the target network includes a generator network, a discriminator network and a multi-receptive field discriminator network; wherein the output end of the generator network is connected to the discriminator network and the multi-receptive field discriminator network respectively; the generator network includes an encoder and a decoder; the decoder includes multiple layers of fused decoding layers, each of which is a decoding layer fused by a cross-attention layer and a convolution layer, and the multiple layers of fused decoding layers are connected in series; the decoder is used to learn the explicit dependency relationship between the potential feature and the true value image; the potential feature is the feature output by the encoder; Each sample of the training data set is used as the input of the target network, and the parameters of the target network are trained and optimized to obtain an image restoration model.

5. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image restoration method of fused cross-attention described in any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the image restoration method integrating cross-attention described in any one of claims 1 to 3.

Images