An image inpainting method, device, electronic equipment and storage medium

Abstract

The application provides an image repairing method and device, electronic equipment and storage medium. The feature extraction layer of a pre-trained repairing model is used to extract features of an image to be repaired to obtain image features. Then, the encoder and decoder of the repairing model are used to perform self-attention encoding operation based on the image features to obtain deep encoding features further encoded from the features of the image to be repaired, and the encoding features are decoded to obtain decoding features. Finally, the target image is obtained based on the decoding features. The method can perform self-attention encoding operation based on the image features to obtain deep encoding features further encoded from the features of the image to be repaired, and perform self-attention decoding operation on the encoding features to obtain decoding features, and then obtain the target image based on the decoding features. Without constructing a mathematical model, the ideal target image can be obtained based on the features of the image itself.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to an image restoration method, apparatus, electronic device, and storage medium. Background Technology

[0002] Image restoration refers to the process of reconstructing lost or damaged parts of images and videos. It is commonly used in fields such as image enhancement, image generation, video editing, and video production, and is a common image processing technique.

[0003] Existing image restoration methods are typically based on the type of image degradation. Specifically, the process begins with acquiring the image to be restored. Then, assumptions are made about the image degradation process, and a corresponding mathematical model is established. For example, if the image to be restored is assumed to be an image obtained after the original image has been contaminated with noise, a Gaussian denoising model can be manually constructed. Next, an optimization algorithm is used to solve for the model parameters, thereby obtaining a refined mathematical model. Finally, the refined mathematical model is used to restore the image.

[0004] However, when using manually constructed mathematical models to restore images, it is necessary to manually estimate the causes of degradation. This makes it difficult to restore the image using information from the image itself, thus making it impossible to obtain a satisfactory target image. Summary of the Invention

[0005] In view of this, the present invention aims to provide an image restoration method, apparatus, electronic device and storage medium, which restores the image to be restored by using a pre-trained restoration model, so as to solve the problem that the prior art requires manual assumptions about the image degradation process to construct a mathematical model, resulting in poor restoration effect.

[0006] This invention provides an image restoration method, comprising:

[0007] Identify the image to be repaired;

[0008] The image to be repaired is input into the feature extraction layer of the pre-trained repair model to obtain the image features output by the feature extraction layer.

[0009] By using the encoder of the repair model, a self-attention encoding operation is performed based on image features to obtain the deep encoded features of the image to be repaired. The self-attention encoding operation is as follows: for each element in the first feature to be encoded, a second feature to be encoded is obtained based on other elements in the same row as the element; and for each element in the second feature to be encoded, an encoded feature is obtained based on other elements in the same column as the element in the second feature to be encoded.

[0010] By using the decoder of the repair model, a self-attention decoding operation is performed on the deep encoded features to obtain the decoded features of the image to be repaired, and the target image is obtained based on the decoded features. The target image is the repaired image of the image to be repaired.

[0011] Alternatively, the repair model is trained in the following manner:

[0012] Obtain sample images corresponding to various degradation types, and determine the original images corresponding to each sample image as annotations for each sample image;

[0013] Each sample image is used as input and fed into the feature extraction layer of the repair model to be trained, so as to obtain the features of each sample output by the feature extraction layer.

[0014] Each sample feature is input into the encoder of the repair model to obtain the deep encoded features corresponding to each sample feature output by the encoder.

[0015] By using the decoder of the repair model, the deep encoded features corresponding to each sample feature are decoded, and the repair results of each sample image are obtained based on the decoded features.

[0016] The restoration model is trained based on the difference between the restoration results of each sample image and the corresponding annotations of each sample image.

[0017] Optionally, by performing a self-attention encoding operation based on image features through the encoder of the restoration model, the deep encoded features of the image to be restored are obtained, including:

[0018] Perform self-attention encoding on the image features to obtain the first feature, and then downsample the first feature to obtain the downsampled first feature;

[0019] The first feature after downsampling is subjected to self-attention encoding to obtain the second feature;

[0020] By repairing the decoder of the model and performing self-attention decoding on the deep encoded features, the decoded features of the target image are obtained, including:

[0021] A self-attention decoding operation is performed based on the second feature to obtain the third feature, and then upsampling is performed based on the third feature to obtain the upsampled third feature. Finally, a fusion operation is performed based on the upsampled third feature and the first feature to obtain the fused third feature.

[0022] A self-attention decoding operation is performed on the fused third feature to obtain the fourth feature, which is used as the decoding feature of the image.

[0023] Optionally, the encoder and decoder include a high-dimensional feature map structure, which includes a first convolutional branch and a second convolutional branch;

[0024] By using the encoder of the restoration model to perform a self-attention encoding operation based on image features, the deep encoded features of the image to be restored are obtained, including:

[0025] The first convolutional feature of the image is obtained through the first convolutional branch of the high-dimensional feature mapping structure;

[0026] The second convolutional feature of the image is obtained through the second convolutional branch of the high-dimensional feature mapping structure, wherein the convolutional depths of the first and second convolutional branches are different.

[0027] Element-wise multiplication is performed on the first and second convolutional features to obtain high-dimensional features;

[0028] By using an attention network, self-attention encoding is performed on the high-dimensional features to obtain the deep encoded features of the image to be repaired.

[0029] Optionally, by performing a self-attention encoding operation based on image features through the encoder of the restoration model, the deep encoded features of the image to be restored are obtained, including:

[0030] Based on the image features, a first feature to be encoded is determined. For each element in the first feature to be encoded, a first row vector is determined with the element as the center and the number of elements contained therein matching the window length, based on the preset window length.

[0031] Based on the elements corresponding to each channel in the first feature to be encoded, determine the attention weights corresponding to each element in the first row vector.

[0032] Based on the first row vector and the attention weights corresponding to each element in the first row vector, the first encoding of the element is determined, and based on the first encodings corresponding to each element in the first feature to be encoded, the intermediate features of the first feature to be encoded are obtained.

[0033] For each element in the intermediate features, the sampling interval is determined according to the window length. Equal interval sampling is performed with the element as the center to determine the second row vector with the number of elements contained in the element as the center and the window length is matched.

[0034] Based on the elements corresponding to each channel in the intermediate features, determine the attention weights corresponding to each element in the second row vector;

[0035] Based on the second row vector and the attention weights corresponding to each element in the second row vector, the second encoding of the element is determined, and based on the second encodings corresponding to each element in the intermediate features, the second feature to be encoded is obtained.

[0036] A longitudinal attention encoding operation is performed on the second feature to be encoded to obtain the depth encoded features of the image to be repaired.

[0037] Optionally, the sampling interval is determined based on the window length, including:

[0038] Based on the window length, determine the distance from the element at the center of the first row vector to the element at the edge of the first row vector, and use this distance as the sampling interval.

[0039] Optionally, based on the difference between the restoration results of each sample image and the corresponding annotations of each sample image, a restoration model is trained, including:

[0040] The pixel value difference between the restoration result of each sample image and the corresponding annotation of each sample image is determined as the first difference, and the frequency domain difference between the restoration result of each sample image and the corresponding annotation of each sample image is determined as the second difference.

[0041] Based on the first and second gaps, the loss is determined, and the repair model is trained with the goal of minimizing the loss.

[0042] This invention provides an image restoration apparatus, comprising:

[0043] The determination module is used to determine the image to be repaired;

[0044] The feature extraction module is used to input the image to be repaired into the feature extraction layer of the pre-trained repair model to obtain the image features output by the feature extraction layer;

[0045] The encoding module is used to perform self-attention encoding based on image features through the encoder of the repair model to obtain the encoded features of the image to be repaired. The self-attention encoding operation is as follows: for each element in the first feature to be encoded, a second feature to be encoded is obtained based on other elements in the same row as the element; and for each element in the second feature to be encoded, a deep encoded feature is obtained based on other elements in the same column as the element in the second feature to be encoded.

[0046] The decoding module is used to perform self-attention decoding on the deep encoded features through the decoder of the repair model to obtain the decoded features of the target image, and obtain the target image based on the decoded features. The target image is the image to be repaired and then repaired.

[0047] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image restoration method.

[0048] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image restoration method described above.

[0049] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0050] In the image restoration method provided in this specification, the feature extraction layer of the restoration model, which has been pre-trained, is used to extract features from the image to be restored to obtain image features. Then, the encoder and decoder of the restoration model are used to perform self-attention encoding based on the image features to obtain deep encoded features that further encode the features of the image to be restored. The deep encoded features are then decoded to obtain decoded features. Finally, the restored image, i.e., the target image, is obtained based on the decoded features.

[0051] This image inpainting method performs self-attention encoding based on the image's own features in the encoder to obtain deep encoded features that further encode the features of the image to be inpainted. Then, in the decoder, it performs self-attention decoding on these deep encoded features to obtain decoded features, which are then used to obtain the target image. In other words, this image inpainting method can restore an ideal target image based on the features of the image to be inpainted. Attached Figure Description

[0052] Figure 1 The diagram shown is a flowchart of the image restoration method provided in this manual;

[0053] Figure 2 This is a schematic diagram of the network architecture of the repair model provided in this specification;

[0054] Figure 3 This is a schematic diagram of the high-dimensional feature mapping structure provided in this specification;

[0055] Figure 4 This is a flowchart illustrating the process of determining the second feature to be encoded, as provided in this specification.

[0056] Figure 5 The diagram shown is a structural schematic of the image restoration device provided in this specification.

[0057] Figure 6 The image shown corresponds to Figure 1 Electronic devices. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] It should be noted that all actions involving the acquisition of signals, information, or data in this manual are performed in accordance with the relevant data protection laws and regulations of the locality and with authorization from the owner of the relevant device.

[0060] An embodiment of the present invention provides an image restoration method. The method extracts features from the image to be restored by using a feature extraction layer of a pre-trained restoration model. Then, the encoder and decoder of the restoration model perform self-attention encoding based on the image features to obtain deep encoded features that further encode the features of the image to be restored. The deep encoded features are then decoded to obtain decoded features. Finally, the target image is obtained based on the decoded features.

[0061] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0062] Figure 1 This is a flowchart illustrating one image restoration method described in this specification, which specifically includes the following steps:

[0063] S100: Identify the image to be repaired.

[0064] In one or more embodiments provided in this specification, the image restoration method can be applied to a server with a restoration model deployed on it, or to electronic devices such as smart terminals that can interact with the server with the restoration model deployed on it. For ease of description, the following description will use the execution process of the image restoration method performed by the server as an example.

[0065] Typically, the server can first identify the image to be repaired.

[0066] In one example, the server can receive image repair requests sent by other devices, parse the image repair requests, and determine the image carried in the image repair requests as the image to be repaired.

[0067] In one example, the server may pre-store images to be repaired. When the repair conditions are triggered, the server may select any image to be repaired from its pre-stored images as the image to be repaired for which the image repair method described in this specification needs to be performed.

[0068] In one example, the image to be repaired could be an overexposed image due to limitations in photographic conditions, or an image whose quality is reduced due to limitations in transmission bandwidth.

[0069] In one example, the image to be repaired could be an image in which parts of the image are damaged (e.g., pixels are set to 0).

[0070] The specific degradation type corresponding to the image to be repaired can be set as needed, and this manual does not impose any restrictions on it.

[0071] S102: Input the image to be repaired into the feature extraction layer of the pre-trained repair model to obtain the image features output by the feature extraction layer. The repair model is trained by sample images corresponding to various degradation types and the annotations corresponding to this image.

[0072] In one or more embodiments provided in this specification, the image restoration method can project the information of the image to be restored into a high-dimensional space through a pre-trained restoration model to obtain image features, and then perform self-attention encoding on the image features to further amplify the features of the image to be restored, so as to restore the image based on its own features.

[0073] Therefore, the server can determine the image features corresponding to the image to be repaired by using the repair model.

[0074] In one example, the restoration model includes a feature extraction layer, an encoder, and a decoder. The server can then use the image to be restored as input to the feature extraction layer of the restoration model to obtain the image features output by the feature extraction layer.

[0075] In one example, the restoration model is trained by a server used to train the model based on sample images corresponding to various degradation types and the annotations corresponding to the sample images.

[0076] S104: By using the encoder of the repair model, a self-attention encoding operation is performed based on the image features to obtain the deep encoded features of the image to be repaired. The self-attention encoding operation is as follows: for each element in the first feature to be encoded, a second feature to be encoded is obtained based on other elements in the same row as the element, and for each element in the second feature to be encoded, an encoded feature is obtained based on other elements in the same column as the element in the second feature to be encoded.

[0077] In one or more embodiments provided in this specification, as described above, the model result of the restoration model is an encoder-decoder structure. Therefore, after obtaining the image features, the server can perform self-attention encoding based on the image features through the encoder of the restoration model, thereby further amplifying the features of the image to be restored and obtaining deep encoded features. Here, amplifying the features of the image to be restored does not refer to physically increasing the size or brightness of the image, but rather to enhancing or highlighting certain features in the image through a self-attention mechanism. In the image restoration scenario applied in this specification, further amplifying the features of the image to be restored specifically refers to enhancing or highlighting features in the image that facilitate image restoration through a self-attention mechanism.

[0078] Therefore, the server can use the image features as input to the encoder of the pre-trained repair model, in which the image features are subjected to self-attention encoding to obtain deep encoded features.

[0079] Specifically, firstly, the server can use the image features as input to the encoder of the repair model.

[0080] Then, in the encoder, the server can use the image feature as the first feature to be encoded, and for each element in the first feature to be encoded, perform self-attention encoding on that element based on other elements in the same row as that element in the first feature to be encoded, to obtain the second feature to be encoded. The second encoded feature is the feature obtained by the server performing lateral self-attention encoding on the first feature to be encoded.

[0081] Then, for each element in the second feature to be encoded, the server performs self-attention encoding on that element based on other elements in the same column as that element in the second feature to be encoded, to obtain a third encoded feature. This third encoded feature is the feature obtained by the server performing vertical attention encoding on the second feature to be encoded.

[0082] Finally, the server can use this third encoded feature as the depth encoded feature of the image to be repaired output by the encoder.

[0083] Optionally, in one example, the server may first perform vertical attention encoding on the image features to obtain a second feature to be encoded, then perform horizontal attention encoding on the second feature to be encoded to obtain a third encoded feature, and finally use the third encoded feature as the depth encoded feature of the image to be repaired. Specifically, the vertical attention encoding operation involves performing self-attention encoding on each element based on other elements in the same column. The horizontal attention encoding operation involves performing self-attention encoding on each element based on other elements in the same row.

[0084] In one example, the horizontal attention encoding operation is as follows: for each element, the element is given attention based on the element itself and at least one other element located in the same row. The vertical attention encoding operation is as follows: for each element, the element is given attention based on the element itself and at least one other element located in the same column.

[0085] In one example, the network structure inside the encoder can be a star network structure.

[0086] S106: By performing self-attention decoding on the deep coding features through the decoding layer of the repair model, the decoded features of the image to be repaired are obtained, and the target image is obtained based on the decoded features. The target image is the image after repair of the image to be repaired.

[0087] In one or more embodiments provided in this specification, as previously described, the model result of the repair model is an encoder-decoder structure. Therefore, after obtaining the deep encoded features, the server can decode the deep encoded features through the decoder of the repair model to obtain the decoded features, and then obtain the target image.

[0088] Specifically, the server can use the deep encoded features as input to the decoder of the pre-trained repair model. In the decoder, the server can perform a self-attention decoding operation based on the deep encoded features to obtain the decoded features.

[0089] Then, the server can obtain the target image based on the decoding features.

[0090] In one example, the server can input depth-coded features into the decoder, where a self-attention decoding operation is performed based on the depth-coded features to obtain decoded features, and the target image is obtained based on the decoded features as the output data of the decoder.

[0091] In one example, the self-attention decoding operation is as follows: for each element in the first feature to be decoded, a second feature to be decoded is obtained based on other elements in the same row as that element, and for each element in the second feature to be decoded, a decoded feature is obtained based on other elements in the same column as that element in the second feature to be decoded.

[0092] In one example, the server can directly use the deep-encoded feature as the first feature to be decoded, perform lateral attention encoding on the first feature to be decoded to obtain the second feature to be decoded, and then perform vertical attention encoding on the second feature to be decoded to obtain the third decoded feature, which is then output as the decoded feature.

[0093] It should be noted that the internal network structures of the encoder and decoder can be the same or similar. During the training of the repair model, the internal parameters of the encoder and decoder can be adjusted to different parameters to achieve different functions.

[0094] based on Figure 1 The image restoration method described herein extracts features from the image to be restored using a pre-trained feature extraction layer of the restoration model. Then, the encoder and decoder of the restoration model perform self-attention encoding based on these image features to obtain deep encoded features that further encode the features of the image to be restored. Finally, these deep encoded features are decoded to obtain decoded features, and the target image is obtained based on these decoded features. This method can perform self-attention encoding based on image features to obtain deep encoded features that further encode the features of the image to be restored, and perform self-attention decoding based on these deep encoded features to obtain decoded features. The target image is then obtained based on these decoded features. Without needing to construct a mathematical model, an ideal target image can be restored solely based on the image's own features.

[0095] Meanwhile, compared to using a pre-defined square convolution kernel and determining a matrix of the same size as the kernel centered on each element in the image features, and then performing self-attention encoding on the element through the matrix dot product of this matrix and the kernel, this method requires less computational resources because it only needs to perform self-attention encoding on the element based on the elements in the same row and column of the element in the same window, even with the same window size.

[0096] Furthermore, the repair model in steps S100 to S106 can be trained in the following manner:

[0097] First, the server used to train the model can obtain sample images corresponding to each degradation type and determine the annotations for each sample image. Specifically, for each sample image, the annotation is the original image of that sample image, which is the degradation result obtained by performing degradation processing on the original image.

[0098] In one example, the degradation process could be compressing the image quality, adding noise to the original image, or deleting local pixels from the original image.

[0099] In one example, for each sample image, the corresponding degradation type is at least one.

[0100] Secondly, the server used to train the model can take each sample image as input and input it into the feature extraction layer of the repair model to be trained, so as to obtain the sample features corresponding to the sample image output by the feature extraction layer.

[0101] Then, the server used to train the model can input the sample features into the encoder of the repair model to obtain the deep encoded features of the sample features output by the encoder.

[0102] Then, the server used to train the model can input the deep encoded features into the decoder of the restoration model, and the decoder will decode the deep encoded features of the sample to obtain the restoration result of the sample image.

[0103] Finally, based on the difference between the restoration results and annotations of each sample image, the server used to train the model can determine the loss and train the restoration model accordingly. The trained restoration model can then be used to restore the image to be restored.

[0104] It should be noted that the server used to train the model and the server that executes the image restoration method can be the same electronic device or different electronic devices.

[0105] Furthermore, based on the difference between the restoration results and annotations of each sample image, the server used to train the model can determine the loss. When training the restoration model based on this loss, the server can determine the pixel value difference between the restoration results and annotations of each sample image as the first difference, and the difference between the spectrogram of the restoration result and the spectrogram of its annotation as the second difference. The server then determines the loss based on the first and second differences, and trains the restoration model with the loss minimization as the optimization objective. In this way, not only the texture features of the image can be learned, but also the frequency domain features, and restoration of the image to be restored is performed based on these frequency domain features, resulting in better restoration performance.

[0106] Furthermore, in the embodiments of this specification, the repair model can be constructed using the U-Net architecture. Taking the need to process two features of different sizes in the repair model as an example, step S104 can be as follows: perform self-attention encoding on the image features to obtain a first feature, and downsample the first feature to obtain a downsampled first feature; perform self-attention encoding on the downsampled first feature to obtain a second feature; perform self-attention decoding on the second feature to obtain a third feature, and upsample the third feature to obtain an upsampled third feature; perform a fusion operation based on the upsampled third feature and the first feature to obtain a fused third feature; perform self-attention decoding on the fused third feature to obtain a fourth feature. The fourth feature serves as the decoded feature of the image features.

[0107] In one example, the server can use image features as input to the encoder of the restoration model. In this encoder, self-attention encoding is performed on the image features to obtain a first feature, and this first feature is downsampled to obtain a downsampled first feature. Then, self-attention encoding is performed on the downsampled first feature to obtain a second feature.

[0108] In one example, the server may use at least one of the first feature and the second feature as a deep-encoded feature.

[0109] In one example, the server can use the second feature as a deep-encoded feature and input it into the decoder of the repair model. In the decoder, the second feature is subjected to a self-attention decoding operation to obtain a third feature, and the third feature is upsampled to obtain an upsampled third feature. The upsampled third feature and the first feature are then fused to obtain a fused third feature. The fused third feature is then subjected to a self-attention encoding operation to obtain a fourth feature, which is used as a deep-encoded feature of the image features.

[0110] In one example, the first feature, the fourth feature, and the upsampled third feature have the same dimension, and the second feature, the third feature, and the downsampled first feature have the same dimension.

[0111] In one example, smaller feature sizes allow for the learning of richer global features, such as texture features, during attention operations. Larger feature sizes, on the other hand, allow for the learning of richer detail features during attention operations. Therefore, the deep-encoded features obtained based on these different feature sizes contain richer global and detail features, thus ensuring the accuracy of the target image determined based on these deep-encoded features.

[0112] It should be noted that the first feature mentioned above is a feature obtained by performing self-attention encoding on the image features, and is not the same feature as the first feature to be encoded. Similarly, the second feature is a feature obtained by performing self-attention encoding on the downsampled first feature, and is not the same feature as the second feature to be encoded.

[0113] In one example, the above description only illustrates the case where the encoder and decoder of the repair model contain branches that process features of two different sizes separately. However, in practice, the encoder and decoder of this repair model may also contain branches that process features of multiple sizes separately. Taking three such branches as an example, the corresponding network structure of the repair model is as follows: Figure 2 As shown.

[0114] Figure 2 This is a schematic diagram of the network architecture of the repair model provided in this specification. As can be seen, the repair model includes a feature extraction layer, an encoder, and a decoder. The encoder and decoder contain the same number of downsampling sub-layers and upsampling sub-layers. The downsampling sub-layers reduce the resolution of the features, and the upsampling sub-layers increase the resolution of the features. The multiple downsampling sub-layers are arranged in order of decreasing output resolution, and the multiple upsampling sub-layers are arranged in order of increasing output resolution.

[0115] In addition to the upsampling and downsampling sub-layers, the codec may also contain several attention sub-layers. The figure illustrates this by showing an example of a codec containing a total of six attention sub-layers.

[0116] Therefore, step S104 above can also be implemented through the following embodiments:

[0117] First, the server can input image features into the first attention sublayer to obtain the first feature output by the first attention sublayer, and then downsample the first feature to obtain the downsampled first feature. The dimensions of the image feature and the first feature can both be C×H×W.

[0118] Secondly, the server can use the downsampled first feature as input to the second attention sublayer. Through this second attention sublayer, the second feature is determined, and the downsampled second feature is also determined. The dimensions of the second feature and the downsampled first feature can be...

[0119] Then, the server can continue to use the downsampled second feature as input to the fifth attention sublayer, and through this fifth attention sublayer, determine the fifth feature. The dimensions of the fifth feature and the downsampled second feature can be...

[0120] The server can then use this fifth feature as input to the sixth attention sublayer to determine the sixth feature. The dimension of this sixth feature can be...

[0121] Next, the server can upsample the sixth feature and input the upsampled sixth feature and the second feature into the fusion sublayer to obtain the fused sixth feature. This fused sixth feature is then used as the input to the third attention sublayer to obtain the third feature. The dimensions of the third feature, the upsampled sixth feature, and the fused sixth feature can be...

[0122] Finally, the server can continue to upsample the third feature, and obtain the fused third feature based on the upsampled third feature and the first feature. Then, through the fourth attention sublayer and the fused third feature, the fourth feature is obtained, which serves as the depth coding feature corresponding to the image to be repaired.

[0123] In this model, multiple downsampling sub-layers form a contraction path, and multiple upsampling sub-layers form an expansion path. Thus, by using the features obtained from downsampling, the encoder can capture higher-level features. Simultaneously, based on upsampling, the features are restored, and the upsampling result is fused with the features before downsampling. The fusion result also retains the original spatial information. Based on the above, this image inpainting model can learn higher-dimensional information, thereby ensuring the information content of the determined deep-encoded features. Based on these deep-encoded features, an accurate target image can be determined, further guaranteeing the success rate of image inpainting.

[0124] It should be noted that for each self-attention sublayer in the codec, the server can perform self-attention encoding on the features input to that self-attention sublayer. The diagram illustrates this using an example of a codec containing two downsampling sublayers; however, in practice, the codec may contain only one downsampling sublayer or multiple downsampling sublayers. The specific number of downsampling sublayers, upsampling sublayers, and attention sublayers that the codec can contain can be set as needed, and this specification does not impose any restrictions on this.

[0125] Furthermore, the numbers corresponding to the names of the attention sub-layers in the aforementioned figures do not indicate the execution order of the attention sub-layers. The numbers corresponding to the names of each attention sub-layer can be set as needed, and this specification does not impose any restrictions on this.

[0126] Furthermore, higher-dimensional features typically imply more knowledge, or richer information. Therefore, target images obtained based on higher-dimensional deep-encoded features usually have higher accuracy than those obtained based on lower-dimensional deep-encoded features. Currently, when mapping image features to higher dimensions, stacking linear convolutional layers is commonly used. This involves stacking multiple linear convolutional layers, with each layer extracting new features based on the previous layer, thus gradually mapping the features to a higher-dimensional space. However, given the limited receptive field of linear convolutional layers and the potential for feature redundancy from stacking them, mapping features to higher dimensions clearly requires significant computational resources, resulting in a larger model structure. Therefore, this specification provides a feature mapping sublayer based on an improved star network, reducing the stacking of linear network layers and instead obtaining higher-dimensional features through element-wise multiplication. Thus, the aforementioned step S104 can be specifically described as follows:

[0127] Specifically, the encoder may include a high-dimensional feature map structure and an attention network, wherein the high-dimensional feature map structure includes a first convolutional branch and a second convolutional branch.

[0128] Therefore, the server can use image features as input, feeding them into the first and second convolutional branches of the high-dimensional feature mapping structure to obtain the first convolutional feature output by the first convolutional branch and the second convolutional feature output by the second convolutional branch. The convolutional depths of the first and second convolutional branches are different; that is, the area of ​​the input feature map covered by the convolutional kernel is also different.

[0129] The server can then perform element-wise multiplication on the first and second convolutional features to obtain high-dimensional features. Element-wise multiplication refers to multiplying corresponding elements of the features. Taking image feature X as an example, assuming the first convolutional feature is W1X and the second convolutional feature is W2X, where W1 and W2 are both m×n in dimension, and the image feature X is n×p in dimension, then the high-dimensional feature can be... in, Let be the element in the i-th row and j-th column of W1X. Let be the element in the i-th row and j-th column of W2X. The dimension of this high-dimensional feature is the same as that of W1X and W2X, which is m×p. However, the implicit feature dimension in the high-dimensional feature is larger than that of the implicit feature dimension of the first and second convolutional features, and therefore can represent richer information.

[0130] Finally, using this high-dimensional feature, the attention network can perform a self-attention encoding operation on the high-dimensional feature to obtain a deep encoded feature, such as... Figure 3 As shown.

[0131] Figure 3 This is a schematic diagram of the high-dimensional feature mapping structure provided in this specification. In the diagram, the server can first normalize image features. Taking a Layer Normalization (LN) layer as an example, the normalized image features are X. LN =LN(X input ).

[0132] In one example, when the high-dimensional feature map structure includes an LN layer, X input The input features are those of this high-dimensional feature mapping structure.

[0133] In one example, where the high-dimensional feature map structure does not include an LN layer, X LN The input features of this high-dimensional feature mapping structure.

[0134] In one example, the server may not perform normalization on the image features, but instead process the image features directly through a high-dimensional feature mapping structure.

[0135] Then, the server can use the image features or the normalized image features as input, and input them into the first convolution branch and the second convolution branch respectively, to obtain the first convolution feature output by the first convolution branch and the second convolution feature output by the second convolution branch.

[0136] Finally, the server can perform element-wise multiplication on the first and second convolutional features to obtain high-dimensional features.

[0137] In one example, the server can perform convolution and residual connection operations on the element-wise multiplication operation to obtain high-dimensional features.

[0138] In this way, the server can obtain a high-dimensional feature that, although the feature size is the same as the input feature size of the high-dimensional feature mapping structure, contains elements that are more dimensional in the feature space. This high-dimensional feature has a larger implicit feature dimension and contains richer information. Therefore, the image is restored based on this high-dimensional feature, and the accuracy of the restored target image is higher.

[0139] Furthermore, to avoid the situation where high-dimensional features only contain higher-dimensional features and gradually forget the low-dimensional features, the server can also fuse the result of element-wise multiplication of the first convolutional feature and the second convolutional feature with the input feature of the high-dimensional feature mapping structure, and use the fused result as the high-dimensional feature.

[0140] In one example, the high-dimensional feature mapping structure can be an Efficient Residual Star Module (ERSM). That is, the high-dimensional feature mapping structure adds a context-aware star operation to the basic residual connection network structure. The star operation is the element-wise multiplication operation of the corresponding feature positions. Compared with the addition operation (the addition operation of the corresponding feature positions), the star operation can map the features to a higher-dimensional space. Furthermore, we designed a context-aware star operation in the second convolution branch to achieve the goal of mapping features to a higher dimension quickly and efficiently.

[0141] Furthermore, the attention layer can contain multiple high-dimensional feature mapping structures, with these structures connected sequentially. In other words, multiple high-dimensional feature mapping structures are stacked. Through this stacked high-dimensional feature mapping structure, features can be mapped to a higher dimension.

[0142] In this specification, typically, the server performs convolution on other elements within a fixed window corresponding to each element in the image features to determine the encoding corresponding to that element, thereby determining the depth-coded features of the image to be repaired. However, when the window size is limited, the depth-coded features of the image to be repaired still contain insufficient information. To improve this, in this specification, when determining the encoding corresponding to an element, in addition to the elements contained in the fixed window, the server can also sample other elements from outside the fixed window as one of the elements for self-attention encoding of that element.

[0143] Specifically, taking the direct lateral self-attention encoding operation on image features as an example, in step S104, the server can use the image features as the first feature to be encoded.

[0144] For each element in the first feature to be encoded, a first row vector is determined based on the preset window length, with the element as the center and the number of elements contained therein matching the window length.

[0145] Secondly, the server can determine the attention weights corresponding to each element in the first row vector based on the elements corresponding to each channel in the first feature to be encoded.

[0146] Then, based on the first row vector and the attention weights corresponding to each element in the first row vector, the server can perform a weighted summation of the first row vector and the attention weights corresponding to each element in the first row vector to obtain the first code corresponding to the element.

[0147] After obtaining the first code corresponding to each element, the server can determine the intermediate feature of the first feature to be encoded based on the first code corresponding to each element and the position of each element.

[0148] Then, for each element in the intermediate feature, the server can determine the sampling interval based on the window length, and perform equal-interval sampling centered on that element to obtain a second row vector. This second row vector is a vector containing the number of elements centered on that element that matches the window length.

[0149] Therefore, after obtaining the second row vector, the server can determine the attention weights corresponding to each element in the second row vector based on the elements corresponding to each channel in the intermediate features.

[0150] In one example, the server can perform global average pooling on each channel in the first feature to be encoded, and determine the channel code corresponding to each channel based on the pooling result. Then, by performing convolution and other operations on the vectors composed of the channel codes corresponding to each channel, attention weights that can be used to represent the element relationships between channels in the first feature to be encoded are obtained. The attention weights and the first row vector are then weighted and summed to obtain the first encoding, which simultaneously contains information about the element relationships between channels and the relationships between the element and its surrounding elements. The description of the second encoding is the same as the description of the first encoding described above, and will not be repeated here.

[0151] Finally, based on the second row vector and the attention weights corresponding to each element in the second row vector, the server can determine the second encoding of the element in the intermediate feature. Then, based on the second encodings corresponding to each element in the intermediate feature, the server can obtain the second encoding feature of the first encoding feature. Figure 4 As shown.

[0152] Figure 4 This is a flowchart illustrating the process of determining the second feature to be encoded, as provided in this specification. In the diagram, the black squares represent the elements to be encoded, and the gray squares in the first row vector represent elements located in the same row as the black squares and relatively close to them. The window length in the diagram is 5, and the number of elements contained in the first row vector matches this window length.

[0153] For each element in the first feature to be encoded, the server can determine a first row vector from the first feature to be encoded, and determine the attention weights corresponding to each element in the first row vector based on the first feature to be encoded, that is, Figure 4The attention weight in the first row vector is 1. Therefore, the server can perform a weighted summation of the attention weights corresponding to each element in the first row vector to obtain the first code. The first code corresponding to each element in the first feature to be encoded can constitute the intermediate feature of the image to be repaired.

[0154] After obtaining the intermediate features, the server can determine the sampling interval for each element in the intermediate features, based on the window length. Then, the server can sample from the element to its left and right, centered on the element, according to the preset sampling interval, to obtain a second row vector. The gray squares in the intermediate features and the second row vector represent the sampled elements. The number of elements in the second row vector matches the window length.

[0155] Then, the server can determine the attention weights corresponding to each element in the second row vector based on the intermediate features, that is, Figure 4 Attention weight 2 in the context.

[0156] Finally, the server can perform a weighted summation of the attention weights corresponding to each element in the second row vector to obtain the second code. The second code corresponding to each element in the intermediate features can constitute the second coded feature of the image to be repaired.

[0157] In one example, the above description uses lateral self-attention encoding of the first feature to be encoded as an example. The process of lateral self-attention encoding of the second feature to be encoded, vertical self-attention encoding of the first feature to be encoded, and vertical self-attention encoding of the second feature to be encoded can be referred to the above description, and will not be repeated here.

[0158] Thus, without increasing the size of the convolution kernel, the server can perform self-attention encoding based on more elements in the image features. In other words, without increasing the size of the convolution kernel, this application expands the receptive field, making the information contained in the depth-coded features of the extracted image to be repaired richer, thereby ensuring the accuracy of the image repair.

[0159] Furthermore, the server can also determine the distance from the element at the center of the first row vector to the element at the edge of the first row vector, based on the window length, as the sampling interval. Assuming the window length is K, the distance from the element at the center of the first row vector to the element at the edge of the first row vector can be: Therefore, the sampling interval can be... Figure 4 If K is 5, then the sampling interval is 2.

[0160] In one example, the server can also take a value greater than Any integer is used as the sampling interval.

[0161] In this way, it can be ensured that the receptive fields corresponding to the elements in the second row vector, except for the elements that need to be self-attention encoded (i.e., the elements located at the center of the vector), do not overlap with the receptive fields corresponding to the elements in the first row vector, thus further ensuring the richness of the information content of the learned features.

[0162] It should be noted that, in the case of the repair model being a U-net structure, for each attention sub-layer in the attention layer, the structure of the attention sub-layer can be a high-dimensional feature map structure + attention network. For details of this attention network, please refer to the aforementioned description of step S104. That is to say, apart from the content of the network structure, the structure of the attention layer described in step S104 can be the structure of this attention network. For details, please refer to the aforementioned description of step S104; this specification does not impose any limitations on it.

[0163] Furthermore, as mentioned earlier, the internal network structures of the encoder and decoder can be the same or similar. During the training process of the repair model, the internal parameters of the encoder and decoder can be adjusted to different parameters to achieve different functions. Similarly, self-attention encoding and self-attention decoding operations can be similar operations, differing only in the modules they belong to. Therefore, the specific steps of the self-attention decoding operation can be found in the description of the self-attention encoding operation above, and will not be repeated here.

[0164] The above are one or more embodiments of the image restoration method provided in this specification. Based on the same idea, this specification also provides an image restoration apparatus, such as... Figure 5 As shown.

[0165] Figure 5 The schematic diagram of the image restoration device provided in this specification shows that the image restoration device may include:

[0166] The determination module 200 is used to determine the image to be repaired.

[0167] The feature extraction module 202 is used to input the image to be repaired into the feature extraction layer of the pre-trained repair model to obtain the image features output by the feature extraction layer.

[0168] The encoding module 204 is used to perform a self-attention encoding operation based on image features through the encoder of the repair model to obtain the deep encoded features of the image to be repaired. The self-attention encoding operation is as follows: for each element in the first feature to be encoded, a second feature to be encoded is obtained based on other elements in the same row as the element, and for each element in the second feature to be encoded, an encoded feature is obtained based on other elements in the same column as the element in the second feature to be encoded.

[0169] The decoding module 206 is used to perform self-attention decoding on the deep coding features through the decoder of the repair model to obtain the decoded features of the image to be repaired, and to obtain the target image based on the decoded features, wherein the target image is the image after repair of the image to be repaired.

[0170] Optionally, the device further includes:

[0171] The training module 208 is used to train the restoration model in the following manner: acquiring sample images corresponding to various degradation types, and determining the original images corresponding to each sample image as annotations for each sample image; inputting each sample image as input to the feature extraction layer of the restoration model to be trained, and obtaining the sample features output by the feature extraction layer; inputting each sample feature into the encoder of the restoration model, and obtaining the deep encoded features corresponding to each sample feature output by the encoder; decoding the deep encoded features corresponding to each sample feature through the decoder of the restoration model, and obtaining the restoration result of each sample image based on the decoded features; and training the restoration model based on the difference between the restoration result of each sample image and the annotations corresponding to each sample image.

[0172] Optionally, the encoding module 204 is used to: perform self-attention encoding on the image features to obtain a first feature, and downsample the first feature to obtain a downsampled first feature; and perform self-attention encoding on the downsampled first feature to obtain a second feature.

[0173] Optionally, the decoding module 206 is used to: perform a self-attention decoding operation based on the second feature to obtain a third feature, and perform upsampling based on the third feature to obtain an upsampled third feature, and perform a fusion operation based on the upsampled third feature and the first feature to obtain a fused third feature; perform a self-attention decoding operation on the fused third feature to obtain a fourth feature, and the fourth feature is used as the decoding feature of the image.

[0174] Optionally, the encoder and decoder include a high-dimensional feature mapping structure, which includes a first convolutional branch and a second convolutional branch. The encoding module 204 is used to: obtain a first convolutional feature of the image features through the first convolutional branch of the high-dimensional feature mapping structure; obtain a second convolutional feature of the image features through the second convolutional branch of the high-dimensional feature mapping structure, wherein the convolutional depths of the first convolutional branch and the second convolutional branch are different; perform element-wise multiplication on the first convolutional feature and the second convolutional feature to obtain a high-dimensional feature; and perform self-attention encoding on the high-dimensional feature through an attention network to obtain a depth-coded feature of the image to be repaired.

[0175] Optionally, the encoding module 204 is configured to: determine a first feature to be encoded based on image features; for each element in the first feature to be encoded, determine a first row vector centered on that element, with the number of elements matching the window length, based on a preset window length; determine the attention weights corresponding to each element in the first row vector based on the elements corresponding to each channel in the first feature to be encoded; determine the first encoding of that element based on the first row vector and the attention weights corresponding to each element in the first row vector; and obtain the intermediate features of the first feature to be encoded based on the first encodings corresponding to each element in the first feature to be encoded; and for the intermediate features... For each element in the feature, the sampling interval is determined according to the window length. Equal-interval sampling is performed with the element as the center to determine a second row vector containing a number of elements that matches the window length. Based on the elements corresponding to each channel in the intermediate feature, the attention weights corresponding to each element in the second row vector are determined. Based on the second row vector and the attention weights corresponding to each element in the second row vector, the second encoding of the element is determined. Based on the second encodings corresponding to each element in the intermediate feature, the second feature to be encoded is obtained. Vertical attention encoding is performed on the second feature to be encoded to obtain the depth-coded features of the image to be repaired.

[0176] Optionally, the encoding module 204 is used to: determine the distance from the element located at the center of the first row vector to the element at the edge of the first row vector, based on the window length, as the sampling interval.

[0177] Optionally, the training module 208 is used to: determine the pixel value difference between the restoration result of each sample image and the corresponding annotation of each sample image as the first difference, and determine the frequency domain difference between the restoration result of each sample image and the corresponding annotation of each sample image as the second difference; determine the loss based on the first difference and the second difference, and train the restoration model with the loss minimization as the optimization objective.

[0178] The specification also provides a computer-readable storage medium storing a computer program that can be used to perform the above-described image restoration method.

[0179] This instruction manual also provides Figure 6 The diagram shows a schematic structural representation of the electronic device. Figure 6At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for the business operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the above-mentioned image restoration method. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0180] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0181] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0182] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0183] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0184] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0185] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0186] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0187] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0188] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0189] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0190] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0191] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0192] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0193] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing nodes connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage nodes.

[0194] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0195] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. An image restoration method, characterized in that, include: Identify the image to be repaired; The image to be repaired is input into the feature extraction layer of a pre-trained repair model to obtain the image features output by the feature extraction layer. The encoder of the restoration model performs a self-attention encoding operation based on the image features to obtain the depth-coded features of the image to be restored. The self-attention encoding operation involves: for each element in the first feature to be encoded, obtaining a second feature to be encoded based on other elements in the same row as the first element; and for each element in the second feature to be encoded, obtaining an encoded feature based on other elements in the same column as the first element in the second feature. The process of obtaining the depth-coded features of the image to be restored by performing a self-attention encoding operation based on the image features using the encoder of the restoration model includes: determining the first feature to be encoded based on the image features; for each element in the first feature to be encoded, determining a first row vector centered on that element, with the number of elements matching the window length, based on a preset window length; and determining the elements corresponding to each channel in the first feature to be encoded, and determining the elements corresponding to each channel in the first row vector. The attention weights are determined accordingly; based on the first row vector and the attention weights corresponding to each element in the first row vector, the first encoding of the element is determined, and based on the first encodings corresponding to each element in the first feature to be encoded, the intermediate features of the first feature to be encoded are obtained; for each element in the intermediate features, the sampling interval is determined according to the window length, and equal-interval sampling is performed with the element as the center to determine a second row vector with the number of elements centered on the element matching the window length; based on the elements corresponding to each channel in the intermediate features, the attention weights corresponding to each element in the second row vector are determined; based on the second row vector and the attention weights corresponding to each element in the second row vector, the second encoding of the element is determined, and based on the second encodings corresponding to each element in the intermediate features, the second feature to be encoded is obtained; vertical attention encoding is performed on the second feature to be encoded to obtain the depth encoding features of the image to be repaired; The decoder of the repair model performs a self-attention decoding operation on the deep coding features to obtain the decoded features of the image to be repaired, and obtains the target image based on the decoded features, wherein the target image is the image after repair of the image to be repaired.

2. The image restoration method according to claim 1, characterized in that, The repair model was trained using the following method: Obtain sample images corresponding to various degradation types, and determine the original images corresponding to each sample image, as annotations for each sample image; The sample images are used as input to the feature extraction layer of the repair model to be trained, and the features of each sample output by the feature extraction layer are obtained. The sample features are input into the encoder of the repair model to obtain the deep encoded features corresponding to each sample feature output by the encoder. The repair model decodes the deep coding features corresponding to each sample feature, and obtains the repair result of each sample image based on the decoded features. The restoration model is trained based on the difference between the restoration result of each sample image and the corresponding annotation of each sample image.

3. The image restoration method according to claim 1, characterized in that, The encoder of the repair model performs a self-attention encoding operation based on the image features to obtain the deep encoding features of the image to be repaired, including: The self-attention encoding operation is performed on the image features to obtain a first feature, and the first feature is downsampled to obtain a downsampled first feature; The self-attention encoding operation is performed on the downsampled first feature to obtain the second feature; The step of performing self-attention decoding on the deep encoded features through the decoder of the repair model to obtain the decoded features of the target image includes: The self-attention decoding operation is performed based on the second feature to obtain the third feature, and upsampling is performed based on the third feature to obtain the upsampled third feature. Then, a fusion operation is performed based on the upsampled third feature and the first feature to obtain the fused third feature. The self-attention decoding operation is performed on the fused third feature to obtain a fourth feature, which is used as the decoding feature of the image.

4. The image restoration method according to claim 1, characterized in that, The encoder and the decoder include a high-dimensional feature mapping structure, which includes a first convolutional branch and a second convolutional branch. The encoder of the repair model performs a self-attention encoding operation based on the image features to obtain the deep encoding features of the image to be repaired, including: The first convolutional feature of the image feature is obtained through the first convolutional branch of the high-dimensional feature mapping structure; The second convolutional feature of the image feature is obtained through the second convolutional branch of the high-dimensional feature mapping structure, wherein the convolutional depths of the first convolutional branch and the second convolutional branch are different; Element-wise multiplication is performed on the first convolutional feature and the second convolutional feature to obtain high-dimensional features; The self-attention encoding operation is performed on the high-dimensional features through the attention network in the encoder to obtain the deep encoded features of the image to be repaired.

5. The image restoration method according to claim 1, characterized in that, The step of determining the sampling interval based on the window length includes: Based on the window length, the distance from the element at the center of the first row vector to the element at the edge of the first row vector is determined as the sampling interval.

6. The image restoration method according to claim 2, characterized in that, The step of training the restoration model based on the difference between the restoration result of each sample image and the corresponding annotation of each sample image includes: The pixel value difference between the restoration result of each sample image and the corresponding annotation of each sample image is determined as the first difference, and the frequency domain difference between the restoration result of each sample image and the corresponding annotation of each sample image is determined as the second difference. Based on the first gap and the second gap, the loss is determined, and the repair model is trained with the goal of minimizing the loss.

7. An image restoration device, characterized in that, include: The determination module is used to determine the image to be repaired; The feature extraction module is used to input the image to be repaired into the feature extraction layer of a pre-trained repair model to obtain the image features output by the feature extraction layer; An encoding module is used to perform self-attention encoding based on the image features through the encoder of the repair model to obtain the encoded features of the image to be repaired. The self-attention encoding operation involves: for each element in the first to-be-encoded feature, obtaining a second to-be-encoded feature based on other elements in the same row as the first element; and for each element in the second to-be-encoded feature, obtaining an encoded depth-encoded feature based on other elements in the same column as the first element in the second to-be-encoded feature. The step of performing self-attention encoding based on the image features through the encoder of the repair model to obtain the depth-encoded features of the image to be repaired includes: determining the first to-be-encoded feature according to the image features; for each element in the first to-be-encoded feature, determining a first row vector centered on the element and containing a number of elements matching the window length, based on a preset window length; and determining each element in the first row vector based on the elements corresponding to each channel in the first to-be-encoded feature. The attention weights corresponding to each element are determined; based on the first row vector and the attention weights corresponding to each element in the first row vector, the first encoding of the element is determined, and based on the first encodings corresponding to each element in the first feature to be encoded, the intermediate features of the first feature to be encoded are obtained; for each element in the intermediate features, the sampling interval is determined according to the window length, and equal-interval sampling is performed with the element as the center to determine a second row vector with the number of elements centered on the element matching the window length; based on the elements corresponding to each channel in the intermediate features, the attention weights corresponding to each element in the second row vector are determined; based on the second row vector and the attention weights corresponding to each element in the second row vector, the second encoding of the element is determined, and based on the second encodings corresponding to each element in the intermediate features, the second feature to be encoded is obtained; vertical attention encoding is performed on the second feature to be encoded to obtain the depth encoding features of the image to be repaired; The decoding module is used to perform self-attention decoding on the deep coding features through the decoder of the repair model to obtain the decoded features of the target image, and to obtain the target image based on the decoded features, wherein the target image is the image after repair of the image to be repaired.

8. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of any one of claims 1 to 6.

Images