Image restoration method and device, electronic equipment, storage medium and program product

Abstract

The invention provides an image restoration method and device, electronic equipment, a storage medium and a program product, and belongs to the technical field of artificial intelligence, and the method comprises the steps: carrying out the global bidirectional attention interaction optimization of a local texture feature matrix set and a local structure feature matrix set of a to-be-restored image, obtaining a texture feature global optimization matrix set and a structural feature global optimization matrix set; based on the texture feature global optimization matrix set and the structural feature global optimization matrix set, determining a texture-structure joint perception feature map; and inputting the texture-structure joint perception feature map into an image restoration model to obtain a restored image. According to the method, the correlation between the local texture feature matrix set and the local structure feature matrix set of the to-be-restored image is captured and measured by using the bidirectional attention mechanism, so that the global bidirectional semantic association information between the texture features and the structure features is captured more comprehensively; according to the invention, the image restoration which retains the overall texture structure of the image, accurately restores image details and textures and has a more real and natural restoration effect is realized.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to an image restoration method, device, electronic equipment, storage medium and program product. Background Art

[0002] Image restoration refers to restoring damaged or missing parts of an image through algorithms to restore it to a complete and natural state. Image restoration technology has been widely used in the fields of old photo restoration, artwork restoration, medical image processing, video editing, etc.

[0003] Traditional image restoration methods are mainly based on local neighborhood information of images, including interpolation-based methods and texture synthesis methods. Among them, interpolation-based methods use the relationship between known pixels to estimate the value of missing pixels through neighbor interpolation, bilinear interpolation, and bicubic interpolation to achieve image restoration; texture synthesis methods select appropriate texture sample blocks from the source image and then copy them to the target area to fill the missing part to achieve image restoration.

[0004] However, although the restoration method based on local neighborhood information of the image can restore the missing part of the image to a certain extent when it relies on the simple copying or expansion of local neighborhood information, it has obvious limitations when dealing with complex scenes due to the lack of in-depth understanding of the overall structure of the image. It is often difficult to deal with large-scale missing areas and complex texture structures, and it is easy to produce obvious stitching marks. Summary of the invention

[0005] The present invention provides an image restoration method, device, electronic device, storage medium and program product, which are used to solve the defect that the restoration method based on local neighborhood information of the image in the prior art lacks a deep understanding of the overall structure of the image, and realizes image restoration that retains the overall texture structure of the image, accurately restores the image details and texture, and has a more realistic and natural restoration effect.

[0006] The present invention provides an image restoration method, comprising: Obtaining a local texture feature matrix set and a local structure feature matrix set of the image to be repaired; Performing global bidirectional attention interaction optimization on the local texture feature matrix set and the local structural feature matrix set to obtain a texture feature global optimization matrix set and a structural feature global optimization matrix set; Determining a texture-structure joint perception feature map based on the texture feature global optimization matrix set and the structure feature global optimization matrix set; The texture-structure joint perception feature map is input into an image restoration model to obtain a restoration image output by the image restoration model.

[0007] According to an image restoration method provided by the present invention, the local texture feature matrix set and the local structure feature matrix set are subjected to global bidirectional attention interaction optimization to obtain a texture feature global optimization matrix set and a structure feature global optimization matrix set, including: Determining a first query feature matrix based on the local texture feature matrices in the local texture feature matrix set; Based on the local structure feature matrix set, determining a first key feature matrix set; Based on the first key feature matrix set, the first query feature matrix is ​​optimized by global attention interaction to obtain a texture feature global optimization matrix, so as to determine a texture feature global optimization matrix set based on the texture feature global optimization matrix.

[0008] According to an image restoration method provided by the present invention, based on the first key feature matrix set, the first query feature matrix is ​​subjected to global attention interaction optimization to obtain a texture feature global optimization matrix, including: Determine first matrix product results based on the product of the first query feature matrix and the transposed matrix of each first key feature matrix in the first key feature matrix set; Determine each first attention weight matrix based on the quotient of each first matrix multiplication result divided by the scale square root of the first key feature matrix; The texture feature global optimization matrix is ​​determined based on the product of the first query feature matrix and the first attention weight matrices.

[0009] According to an image restoration method provided by the present invention, the local texture feature matrix set and the local structure feature matrix set are subjected to global bidirectional attention interaction optimization to obtain a texture feature global optimization matrix set and a structure feature global optimization matrix set, including: Determine a second query feature matrix based on the local structure feature matrices in the local structure feature matrix set; Based on the local texture feature matrix set, determining a second key feature matrix set; Based on the second key feature matrix set, the second query feature matrix is ​​optimized by global attention interaction to obtain a structural feature global optimization matrix, so as to determine the structural feature global optimization matrix set based on the structural feature global optimization matrix.

[0010] According to an image restoration method provided by the present invention, based on the second key feature matrix set, the second query feature matrix is ​​subjected to global attention interaction optimization to obtain a structural feature global optimization matrix, including: Determine each second matrix product result based on the product of the second query feature matrix and the transposed matrix of each second key feature matrix in the second key feature matrix set; Determine each second attention weight matrix based on the quotient of each second matrix multiplication result divided by the scale square root of the second key feature matrix; The structural feature global optimization matrix is ​​determined based on the product of the second query feature matrix and each of the second attention weight matrices.

[0011] According to an image restoration method provided by the present invention, the determining of a texture-structure joint perceptual feature map based on the texture feature global optimization matrix set and the structure feature global optimization matrix set includes: Performing feature aggregation on the texture feature global optimization matrix set along the channel dimension to obtain a texture global optimization feature map; Performing feature aggregation on the structural feature global optimization matrix set along the channel dimension to obtain a structural global optimization feature graph; Based on the texture global optimization feature map and the structure global optimization feature map, the texture-structure joint perception feature map is determined.

[0012] According to an image restoration method provided by the present invention, the step of obtaining a local texture feature matrix set and a local structure feature matrix set of an image to be restored includes: Extracting a first texture feature map and a structure feature map from the image to be repaired; Performing fine-grained deconstruction on the first texture feature map to obtain the local texture feature matrix set; The structural feature graph is fine-grainedly deconstructed to obtain the local structural feature matrix set.

[0013] According to an image restoration method provided by the present invention, extracting a texture feature map and a structure feature map from the image to be restored includes: Inputting the image to be repaired into a first texture feature extractor to obtain the first texture feature map output by the first texture feature extractor; Inputting the image to be repaired into a structural feature extractor to obtain the structural feature map output by the structural feature extractor; The loss function of the first texture feature extractor is determined based on texture consistency loss and perceptual loss; the loss function of the structural feature extractor is determined based on structural similarity index loss, shape preservation metric loss, boundary recall rate loss and precision loss.

[0014] According to an image restoration method provided by the present invention, before obtaining a local texture feature matrix set and a local structure feature matrix set of the image to be restored, the method comprises: Get the original image; Inputting the original image into a second texture feature extractor to obtain a second texture feature map output by the second texture feature extractor; The second texture feature map is input into a texture restoration model to obtain the image to be restored output by the texture restoration model.

[0015] The present invention also provides an image restoration device, comprising: A matrix set acquisition module is used to acquire a local texture feature matrix set and a local structure feature matrix set of the image to be repaired; A global attention optimization module, used for performing global bidirectional attention interaction optimization on the local texture feature matrix set and the local structure feature matrix set to obtain a texture feature global optimization matrix set and a structure feature global optimization matrix set; A joint perception feature map acquisition module, used to determine a texture-structure joint perception feature map based on the texture feature global optimization matrix set and the structure feature global optimization matrix set; The joint perception feature map restoration module is used to input the texture-structure joint perception feature map into the image restoration model to obtain the restored image output by the image restoration model.

[0016] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, any of the above-mentioned image restoration methods is implemented.

[0017] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-mentioned image restoration methods.

[0018] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the image restoration method described above is implemented.

[0019] The image restoration method, device, electronic device, storage medium and program product provided by the present invention utilize a bidirectional attention mechanism to capture and measure the correlation between a local texture feature matrix set and a local structural feature matrix set of an image to be restored, perform global bidirectional attention interaction optimization on the local texture feature matrix set and the local structural feature matrix set, perform semantic interaction between each local feature matrix in one set and all local feature matrices in another set, more comprehensively capture the global bidirectional semantic association information between the texture features and the structural features of the image to be restored, enhance and supplement the relationship between the texture features and the structural features of the image to be restored, and generate a texture-structure joint perception feature map by integrating the complementary information of the enhanced texture features and structural features of the image to be restored, thereby providing a richer feature basis for the fine restoration of the damaged image, and ultimately achieving image restoration that retains the overall texture structure of the image, accurately restores the image details and texture, and has a more realistic and natural restoration effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 This is one of the flow charts of the image restoration method provided by the present invention.

[0022] Figure 2 This is the second flow chart of the image restoration method provided by the present invention.

[0023] Figure 3 It is a structural schematic diagram of the image restoration device provided by the present invention.

[0024] Figure 4 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be noted that, in the description of the present invention, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0027] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more.

[0028] Combine the following Figure 1-Figure 4 The present invention describes an image restoration method, an apparatus, an electronic device, a storage medium and a program product.

[0029] Figure 1 It is one of the flow charts of the image restoration method provided by the present invention, such as Figure 1 As shown, the image restoration method includes but is not limited to steps 101 to 104.

[0030] It should be noted that the executor of the image restoration method provided by the present invention is the corresponding image restoration device, which can specifically be a server, a computer device, such as a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc.

[0031] Step 101: Obtain a local texture feature matrix set and a local structure feature matrix set of the image to be repaired.

[0032] The local texture feature matrix set is a set composed of multiple local texture feature matrices determined by extracting texture features of the image to be repaired and based on the extracted texture features; each local texture feature matrix represents the local texture features of the image to be repaired, and all local texture feature matrices in the local texture feature matrix set jointly represent the overall texture features of the image to be repaired.

[0033] The local structure feature matrix set is a set composed of multiple local structure feature matrices determined by extracting the structural features of the image to be repaired and based on the extracted structural features; each local structure feature matrix represents the local structure features of the image to be repaired, and all local structure feature matrices in the local structure feature matrix set jointly represent the overall structure features of the image to be repaired.

[0034] Specifically, during image restoration, feature extraction is performed on the texture features and structural features of the image to be restored, and multiple local texture feature matrices are determined based on the extracted texture features to obtain a local texture feature matrix set consisting of multiple local texture feature matrices. Multiple local structural feature matrices are determined based on the extracted structural features to obtain a local structural feature matrix set consisting of multiple local structural feature matrices.

[0035] Optionally, the image to be restored is an original image that has not been restored and has damaged or missing parts; or, the image to be restored is an original image that has undergone preliminary restoration.

[0036] For example, when the image to be repaired is an original image that has not been repaired and has damaged or missing parts, the image to be repaired can be obtained in the following ways: reading the image to be repaired from a local storage device (such as a hard disk, a USB flash drive); downloading the image to be repaired that needs to be repaired from a network server (such as a cloud storage service); using an image acquisition device such as a scanner to scan paper photos or other physical objects, and obtaining the image to be repaired that is converted into a digital image; obtaining the image to be repaired that is directly uploaded by the user in an application or system; intercepting the image to be repaired from the image stream captured in real time by the camera, etc.

[0037] Step 102: Perform global bidirectional attention interaction optimization on the local texture feature matrix set and the local structural feature matrix set to obtain a texture feature global optimization matrix set and a structural feature global optimization matrix set.

[0038] Specifically, a global bidirectional attention interaction optimization is performed on the local texture feature matrix set and the local structure feature matrix set of the image to be repaired, including: using the local structure feature matrix set to perform attention optimization on each local texture feature matrix in the local texture feature matrix set, that is, using the overall structural features of the image to be repaired to perform global attention optimization on the local texture features, and obtaining a texture feature global optimization matrix set composed of texture feature global optimization matrices after global attention optimization of multiple local texture feature matrices; using the local texture feature matrix set to perform attention optimization on each local structure feature matrix in the local structure feature matrix set, that is, using the overall texture of the image to be repaired The local structure features are optimized with global attention, and a set of structure feature global optimization matrices is obtained, which is composed of the structure feature global optimization matrices after global attention optimization of multiple local structure feature matrices; the local texture feature matrix set is used to perform attention optimization on each local structure feature matrix in the local structure feature matrix set, and the local structure feature matrix set is used to perform attention optimization on each local texture feature matrix in the local texture feature matrix set, thereby realizing the global two-way attention interactive optimization of the local texture feature matrix set and the local structure feature matrix set, that is, realizing the global two-way attention interactive optimization of the local texture features and the local structure features of the image to be repaired.

[0039] Optionally, the global bidirectional attention interaction optimization of the local texture feature matrix set and the local structure feature matrix set is implemented based on the multi-head attention mechanism in the Transformer model.

[0040] Step 103: Determine a texture-structure joint perceptual feature map based on the texture feature global optimization matrix set and the structure feature global optimization matrix set.

[0041] Specifically, the texture feature global optimization matrix set and the structural feature global optimization matrix set obtained after the global bidirectional attention interaction optimization contain all the information that can characterize the global texture features and structural features of the image to be repaired. Therefore, after feature aggregation and other processing on the texture feature global optimization matrix set and the structural feature global optimization matrix set, a texture-structure joint perception feature map that completely characterizes the global texture features and global structural features of the image to be repaired can be obtained.

[0042] As an optional embodiment, determining the texture-structure joint perceptual feature map based on the texture feature global optimization matrix set and the structure feature global optimization matrix set includes: The texture feature global optimization matrix set is feature aggregated along the channel dimension to obtain a texture global optimization feature map; the structure feature global optimization matrix set is feature aggregated along the channel dimension to obtain a structure global optimization feature map; based on the texture global optimization feature map and the structure global optimization feature map, the texture-structure joint perception feature map is determined.

[0043] Specifically, when determining the texture-structure joint perception feature map, the texture features, structural features, and feature global optimization matrices on the local dimension obtained after the global bidirectional attention interaction optimization processing are recombined and aggregated along the channel dimension. On the one hand, each texture feature global optimization matrix in the texture feature global optimization matrix set is feature aggregated to obtain a texture global optimization feature map with complete and global texture features. On the other hand, each structure feature global optimization matrix in the structure feature global optimization matrix set is feature aggregated to obtain a structure global optimization feature map with complete and global pattern structure features.

[0044] The texture global optimization feature map and the structure global optimization feature map are further fused by weighted summation according to position, and the complementary information of the global texture features and the global structural features of the image to be repaired are integrated to generate a texture-structure joint perception feature map including the global texture features and structural features of the image to be repaired, thereby providing a richer feature basis for the fine restoration of the damaged image.

[0045] Optionally, the texture feature global optimization matrix set is subjected to feature aggregation along the channel dimension to obtain an expression of a texture global optimization feature map as follows: ; in, Optimize feature maps for the entire texture domain; It is the feature aggregation operation along the channel dimension; are the first, second, ..., and third texture feature global optimization matrix sets respectively. , …, A global optimization matrix of texture features.

[0046] Optionally, the structure feature global optimization matrix set is subjected to feature aggregation along the channel dimension to obtain an expression of a structure global optimization feature graph as follows: ; in, Optimize feature maps for the entire domain of the structure; It is the feature aggregation operation along the channel dimension; are the first, second, ..., and third in the global optimization matrix set of structural features. , …, A global optimization matrix of structural features.

[0047] Optionally, the expression for determining the texture-structure joint perception feature map based on the texture global optimization feature map and the structure global optimization feature map is as follows: ; in, It is the texture-structure joint perception feature map; Optimize feature maps for the entire texture domain; Optimize feature maps for the entire domain of the structure; , are different weight coefficients.

[0048] Step 104: input the texture-structure joint perception feature map into an image restoration model to obtain a restoration image output by the image restoration model.

[0049] Specifically, the texture-structure joint perception feature map is input into a pre-trained image restoration model to obtain a restoration image output by the image restoration model, which is obtained by repairing the image to be restored after considering the global texture features and the global structure features.

[0050] Optionally, the image restoration model is constructed based on a diffusion model; the image restoration model is pre-trained based on paired image training samples, wherein each pair of image training samples includes an original lossless image and a damaged image after noise addition processing.

[0051] The diffusion model is a machine learning model for image generation. Its principle is to gradually add noise to the data and learn how to recover the distribution of the original data from the noise. The diffusion model imitates the process of data generation through a gradual denoising process. During the training phase, the random noise in the data is gradually increased, and then the inverse process is learned - that is, how to gradually recover the original data from the noisy state. In the diffusion model, data generation is modeled as a series of small, continuous steps, each step slightly modifies the result of the previous step until a complete data sample is finally generated. The training goal of this model is usually to learn an inverse mapping from the noise distribution to the true data distribution. The model continuously tries to recover the data contaminated by noise until it can generate samples close to the true data distribution.

[0052] The image restoration method provided by the present invention utilizes a bidirectional attention mechanism to capture and measure the correlation between a set of local texture feature matrices and a set of local structural feature matrices of an image to be restored, performs global bidirectional attention interaction optimization on the set of local texture feature matrices and the set of local structural feature matrices, performs semantic interaction between each local feature matrix in one set and all local feature matrices in another set, more comprehensively captures the global bidirectional semantic association information between the texture features and the structural features of the image to be restored, enhances and supplements the relationship between the texture features and the structural features of the image to be restored, and generates a texture-structure joint perception feature map by integrating the complementary information of the enhanced texture features and the structural features of the image to be restored, thereby providing a richer feature basis for the fine restoration of the damaged image, and ultimately achieving image restoration that retains the overall texture structure of the image, accurately restores the image details and texture, and has a more realistic and natural restoration effect.

[0053] Based on the above embodiment, as an optional embodiment, the local texture feature matrix set and the local structure feature matrix set are subjected to global bidirectional attention interaction optimization to obtain a texture feature global optimization matrix set and a structure feature global optimization matrix set, including: Determining a first query feature matrix based on the local texture feature matrices in the local texture feature matrix set; Based on the local structure feature matrix set, determining a first key feature matrix set; Based on the first key feature matrix set, the first query feature matrix is ​​optimized by global attention interaction to obtain a texture feature global optimization matrix, so as to determine a texture feature global optimization matrix set based on the texture feature global optimization matrix.

[0054] Specifically, when performing global bidirectional attention interaction optimization of a local texture feature matrix set and a local structure feature matrix set, and using the local structure feature matrix set to perform global attention optimization on each local texture feature matrix in the local texture feature matrix set, a local texture feature matrix in the local texture feature matrix set is used as the first query feature matrix, and the local structure feature matrix set is used as the first key feature matrix set, and each local structure feature matrix is ​​also a first key feature matrix.

[0055] The first key feature matrix set and the first query feature matrix are input into the first converter structure of the Transformer model, and the first query feature matrix is ​​optimized by the first key feature matrix set through global attention interaction to obtain a texture feature global optimization matrix corresponding to the first query feature matrix output by the first converter structure.

[0056] Repeat the steps of using each local texture feature matrix in the local texture feature matrix set as the first query feature matrix, using the first key feature matrix set to perform global attention interactive optimization on the first query feature matrix, and combining all the obtained texture feature global optimization matrices to construct a texture feature global optimization matrix set.

[0057] The image restoration method provided by the present invention uses the local texture feature matrix as the query matrix and the local structure feature matrix set matrix as the key matrix set, and uses the key matrix set to perform global attention interactive optimization on the query matrix corresponding to each local texture feature matrix, thereby strengthening and supplementing the relationship between the local texture features and the global structure features in the image to be restored, which helps to ultimately achieve accurate restoration of image details and texture, and image restoration with a more realistic and natural restoration effect.

[0058] Based on the above embodiment, as an optional embodiment, based on the first key feature matrix set, the first query feature matrix is ​​subjected to global attention interaction optimization to obtain a texture feature global optimization matrix, including: Determine first matrix product results based on the product of the first query feature matrix and the transposed matrix of each first key feature matrix in the first key feature matrix set; Determine each first attention weight matrix based on the quotient of each first matrix multiplication result divided by the scale square root of the first key feature matrix; The texture feature global optimization matrix is ​​determined based on the product of the first query feature matrix and the first attention weight matrices.

[0059] Specifically, when the global attention interaction optimization is performed on the first query feature matrix corresponding to any local texture feature matrix in the local texture feature matrix set based on the first key feature matrix set, the first matrix product results are determined according to the product of the first query feature matrix and the transposed matrix of each first key feature matrix in the first key feature matrix set. Further, each first attention weight matrix is ​​determined according to the quotient of each first matrix product result divided by the scale square root of the first key feature matrix, and each first attention weight matrix is ​​multiplied with the first query feature matrix to obtain the texture feature global optimization matrix corresponding to the first query feature matrix.

[0060] Optionally, the first attention weight matrices are determined based on the quotient of the product results of the first matrices divided by the scaled square root of the first key feature matrix, including: dividing the product results of the first matrices by the quotient of the scaled square root of the first key feature matrix, and respectively performing normalized exponential function (softmax function) operations to obtain the first attention weight matrices.

[0061] Optionally, based on the first key feature matrix set, the first query feature matrix is ​​subjected to global attention interaction optimization to obtain an expression of a global optimization matrix of texture features as follows: ; in, Based on the local texture feature matrix set A first query feature matrix determined by a local texture feature matrix; is the first query feature matrix The global optimization matrix of texture features after global attention interaction optimization; Based on the local structure feature matrix set The first key feature matrix determined by the local structure feature matrix; is the first key feature matrix Scale of is the normalized exponential function; is the matrix multiplication operation; is the transpose of the matrix.

[0062] The image restoration method provided by the present invention performs global attention interactive optimization on the query matrix corresponding to each local texture feature matrix by using a key matrix set, multiplies the query matrix with the transposed matrix of each key matrix in the key matrix set, and then divides it by the square root of the scale of the key matrix to obtain each attention weight matrix, and then multiplies each attention weight matrix with the query matrix to obtain a global optimization matrix of texture features after global attention optimization, which strengthens and supplements the relationship between local texture features and global structural features in the image to be restored, and helps to ultimately achieve accurate restoration of image details and texture, and image restoration with a more realistic and natural restoration effect.

[0063] Based on the above embodiment, as an optional embodiment, the local texture feature matrix set and the local structure feature matrix set are subjected to global bidirectional attention interaction optimization to obtain a texture feature global optimization matrix set and a structure feature global optimization matrix set, including: Determine a second query feature matrix based on the local structure feature matrices in the local structure feature matrix set; Based on the local texture feature matrix set, determining a second key feature matrix set; Based on the second key feature matrix set, the second query feature matrix is ​​optimized by global attention interaction to obtain a structural feature global optimization matrix, so as to determine the structural feature global optimization matrix set based on the structural feature global optimization matrix.

[0064] Specifically, when performing global bidirectional attention interaction optimization of a local texture feature matrix set and a local structure feature matrix set, and using the local texture feature matrix set to perform global attention optimization on each local structure feature matrix in the local structure feature matrix set, a local structure feature matrix in the local structure feature matrix set is used as the second query feature matrix, and the local texture feature matrix set is used as the second key feature matrix set, and each local texture feature matrix is ​​also a second key feature matrix.

[0065] The second key feature matrix set and the second query feature matrix are input into the second transformer structure of the Transformer model, and the second query feature matrix is ​​optimized through global attention interaction by the second key feature matrix set to obtain a structural feature global optimization matrix corresponding to the second query feature matrix output by the second transformer structure.

[0066] Repeat the steps of using each local structural feature matrix in the local structural feature matrix set as the second query feature matrix, using the second key feature matrix set to perform global attention interaction optimization on the second query feature matrix, and combining all the obtained structural feature global optimization matrices to construct a structural feature global optimization matrix set.

[0067] The image restoration method provided by the present invention uses the local structure feature matrix as the query matrix and the local texture feature matrix set matrix as the key matrix set, and uses the key matrix set to perform global attention interactive optimization on the query matrix corresponding to each local structure feature matrix, thereby strengthening and supplementing the relationship between the local structure features and the global texture features in the image to be restored, which helps to ultimately achieve accurate restoration of image details and texture, and image restoration with a more realistic and natural restoration effect.

[0068] Based on the above embodiment, as an optional embodiment, the second query feature matrix is ​​optimized for global attention interaction based on the second key feature matrix set to obtain a global optimization matrix of structural features, including: Determine each second matrix product result based on the product of the second query feature matrix and the transposed matrix of each second key feature matrix in the second key feature matrix set; Determine each second attention weight matrix based on the quotient of each second matrix multiplication result divided by the scale square root of the second key feature matrix; The structural feature global optimization matrix is ​​determined based on the product of the second query feature matrix and each of the second attention weight matrices.

[0069] Specifically, when the global attention interaction optimization is performed on the second query feature matrix corresponding to any local structural feature matrix in the local structural feature matrix set based on the second key feature matrix set, the product of each second matrix product is determined according to the product of the second query feature matrix and the transposed matrix of each second key feature matrix in the second key feature matrix set. Further, each second attention weight matrix is ​​determined according to the quotient of each second matrix product result divided by the scale square root of the second key feature matrix, and each second attention weight matrix is ​​multiplied by the second query feature matrix to obtain the global optimization matrix of the structural feature corresponding to the second query feature matrix.

[0070] Optionally, the second attention weight matrices are determined based on the quotient of the product results of the second matrices divided by the scaled square root of the second key feature matrix, including: dividing the product results of the second matrices by the quotient of the scaled square root of the second key feature matrix, and respectively performing normalized exponential function (softmax function) operations to obtain each second attention weight matrix.

[0071] Optionally, based on the second key feature matrix set, the second query feature matrix is ​​subjected to global attention interaction optimization, and an expression of the structural feature global optimization matrix is ​​obtained as follows: ; in, Based on the local structure feature matrix set A second query feature matrix determined by a local structure feature matrix; is the second query feature matrix The global optimization matrix of structural features after global attention interaction optimization; Based on the local texture feature matrix set A second key feature matrix determined by a local texture feature matrix; is the second key feature matrix Scale of is the normalized exponential function; is the matrix multiplication operation; is the transpose of the matrix.

[0072] In one embodiment, the scale of the first key feature matrix and the second key feature matrix is the same, the square root of the scale The same is true.

[0073] The image restoration method provided by the present invention performs global attention interactive optimization on the query matrix corresponding to each local structural feature matrix by utilizing a key matrix set, multiplies the query matrix with the transposed matrix of each key matrix in the key matrix set, and then divides it by the square root of the scale of the key matrix to obtain each attention weight matrix, and then multiplies each attention weight matrix with the query matrix to obtain a global optimization matrix of the structural features after global attention optimization, which strengthens and supplements the relationship between the local structural features and the global texture features in the image to be restored, and helps to ultimately achieve accurate restoration of image details and texture, and image restoration with a more realistic and natural restoration effect.

[0074] Based on the above embodiment, as an optional embodiment, the step of obtaining a local texture feature matrix set and a local structure feature matrix set of the image to be restored includes: Extracting a first texture feature map and a structure feature map from the image to be repaired; Performing fine-grained deconstruction on the first texture feature map to obtain the local texture feature matrix set; The structural feature graph is fine-grainedly deconstructed to obtain the local structural feature matrix set.

[0075] Specifically, when obtaining a local texture feature matrix set and a local structure feature matrix set of an image to be repaired, the image to be repaired is first obtained, and texture features of the image are extracted from the image to be repaired to obtain a first texture feature map, and structural features of the image are also extracted from the image to be repaired to obtain a structure feature map. The first texture feature map and the structure feature map are respectively subjected to fine-grained deconstruction processing along the channel dimension, and the first texture feature map is deconstructed into a local texture feature matrix set, and the structure feature map is deconstructed into a local structure feature matrix set.

[0076] Optionally, the first texture feature map is subjected to fine-grained deconstruction to obtain an expression for the local texture feature matrix set as follows: ; in, is a first texture feature map extracted from the image to be repaired; For fine-grained deconstruction operations; is a set of local texture feature matrices; , , …, , …, is the first, second, ..., first local texture feature matrix set , …, A local texture feature matrix.

[0077] Optionally, the fine-grained deconstruction of the structural feature graph to obtain the expression of the local structural feature matrix set is as follows: ; in, is the structural feature map extracted from the image to be repaired; For fine-grained deconstruction operations; is the set of local structure feature matrices; , , …, , …, is the first, second, ..., and third local structure feature matrix set. , …, A local structure feature matrix.

[0078] The image restoration method provided by the present invention extracts a texture feature map and a structure feature map from the image to be restored, and performs fine-grained feature deconstruction on the texture feature map and the structure feature map to obtain a local texture feature matrix set and a local structure feature matrix set. This can more carefully analyze the local information in the feature map of the image to be restored during the image restoration process, which helps to ultimately achieve accurate restoration of image details and texture, and image restoration with a more realistic and natural restoration effect.

[0079] Based on the above embodiment, as an optional embodiment, extracting a texture feature map and a structure feature map from the image to be repaired includes: Inputting the image to be repaired into a first texture feature extractor to obtain the first texture feature map output by the first texture feature extractor; Inputting the image to be repaired into a structural feature extractor to obtain the structural feature map output by the structural feature extractor; The loss function of the first texture feature extractor is determined based on texture consistency loss and perceptual loss; the loss function of the structural feature extractor is determined based on structural similarity index loss, shape preservation metric loss, boundary recall rate loss and precision loss.

[0080] Optionally, both the first texture feature extractor and the structural feature extractor are constructed based on a Depthwise Separable Convolutional Neural Network Model (DSCNN).

[0081] As an efficient convolution operation mode, DSCNN adopts a separated design, which decomposes the traditional convolution operation into two independent steps: depthwise convolution and pointwise convolution. The depthwise convolution extracts texture features by independently applying convolution kernels to each channel of the image to be repaired, and the pointwise convolution is used to combine the feature outputs of the depthwise convolution layer to enhance the interaction of texture features between channels, thereby accurately capturing the subtle texture information in the image to be repaired, providing an important data basis for subsequent repair work, while effectively reducing the amount of calculation, maintaining good feature extraction capabilities and improving the efficiency of feature extraction.

[0082] Specifically, before extracting the texture feature map and the structure feature map from the image to be repaired, it is necessary to pre-build a first texture feature extractor and a structure feature extractor and perform pre-training.

[0083] For the first texture feature extractor used to extract texture features of the image to be repaired, the first texture feature extractor is first constructed based on the DSCNN model. Considering that the performance of the first texture feature extractor requires that the image repair can maintain the original texture features, the loss function of the first texture feature extractor is determined based on texture consistency loss and perceptual loss. Among them, texture consistency loss reflects the similarity of textures before and after image repair, and perceptual loss compares the difference between the images before and after repair in the high-level feature space through a pre-trained deep network to evaluate the quality of texture restoration.

[0084] For the structural feature extractor used to extract the structural features of the image to be repaired, the structural feature extractor is first built based on the DSCNN model. Considering that the performance of the structural feature extractor requires that the image repair can ensure the fidelity of the overall structure and the accuracy of the boundary information, the loss function of the structural feature extractor is determined based on the structural similarity index (SSIM) loss, shape preservation metric loss, boundary recall loss and precision loss. Among them, the structural similarity index loss can reflect the structural similarity between the repaired image and the image to be repaired, the shape preservation metric loss is used to ensure that the repair process does not cause image distortion, and the boundary recall loss and precision loss are used to evaluate the ability of the structural feature extractor to capture the boundary of the object.

[0085] After constructing a first texture feature extractor and a structural feature extractor and their loss functions and completing pre-training using training samples, when obtaining the texture feature map and the structural feature map of the image to be repaired, on the one hand, the image to be repaired is input into the first texture feature extractor and the first texture feature map output by the first texture feature extractor is obtained; on the other hand, the image to be repaired is input into the structural feature extractor and the structural feature map output by the structural feature extractor is obtained.

[0086] Optionally, the loss function of the first texture feature extractor is determined based on the weighted sum of texture consistency loss and perceptual loss; the loss function of the structural feature extractor is determined based on the weighted sum of structural similarity index loss, shape preservation metric loss, boundary recall rate loss and precision loss.

[0087] The image restoration method provided by the present invention utilizes a first texture feature extractor and a structural feature extractor to extract a texture feature map and a structural feature map of an image to be restored, and determines the loss function of the first texture feature extractor based on two indicators, texture consistency and perceptual loss, and determines the loss function of the structural feature extractor based on structural similarity index loss, shape preservation metric loss, boundary recall rate loss and precision loss. The method can more fully mine the texture features and structural features in the image to be restored, and obtain more refined texture feature maps and structural feature maps, so that the local texture feature matrix set and the local structural feature matrix set obtained after fine-grained deconstruction can perform more effective feature fusion of global bidirectional attention interaction optimization.

[0088] In one embodiment, the convolution kernel size of the first texture feature extractor is smaller than the convolution kernel size of the structural feature extractor, and the downsampling operation of the first texture feature extractor is less than the downsampling operation of the structural feature extractor.

[0089] For example, the convolution kernel of the first texture feature extractor is 3×3, and the convolution kernel size of the structural feature extractor is 5×5 or 7×7.

[0090] The texture features of the image to be repaired usually involve local details. Related image repair methods rely on simple copying or expansion of local neighborhood information and lack understanding of the overall structure of the image. Especially when dealing with non-uniform distribution or complex structures, it is difficult to maintain the consistency and coherence of details, resulting in insufficient sensitivity to local details. Using a smaller convolution kernel size and fewer downsampling operations to construct the first texture feature extractor can maintain sensitivity to local details, so that when processing complex texture areas, it can also effectively capture and restore subtle texture changes, ensuring that the repaired image is more realistic.

[0091] The structural features of the image to be repaired usually involve edges, contours and overall shapes. Related image repair methods often cannot completely maintain these key structural features for large-area missing or areas containing important edge and contour information, resulting in the repaired image not looking coherent or real enough, and poor performance in image repair scenarios with large-scale vacancies. By adopting a larger convolution kernel size and more downsampling operations to construct a structural feature extractor, it is possible to capture a wider range of contextual information and effectively supplement the key structural information lost in the original image or the original image that has undergone preliminary image repair processing, ensuring that the repaired image retains the original visual characteristics and enhances naturalness.

[0092] Based on the above embodiment, as an optional embodiment, before obtaining the local texture feature matrix set and the local structure feature matrix set of the image to be repaired, the following steps are included: Get the original image; Inputting the original image into a second texture feature extractor to obtain a second texture feature map output by the second texture feature extractor; The second texture feature map is input into a texture restoration model to obtain the image to be restored output by the texture restoration model.

[0093] The original image is an image that has not been repaired and has damaged or missing parts.

[0094] Optionally, the second texture feature extractor is constructed based on the DSCNN model. Further, the construction and pre-training method of the second texture feature extractor is the same as that of the first texture feature extractor, which will not be described in detail.

[0095] Optionally, the texture restoration model is constructed based on the diffusion model. Further, the construction and pre-training method of the texture restoration model is the same as that of the image restoration model, which will not be described in detail.

[0096] Texture is a basic attribute of an image and is used to reflect the subtle structure of an object's surface. In image restoration, it is crucial to accurately extract and restore the texture information of an image. Before using the global bidirectional attention interaction optimization mechanism to effectively fuse the local texture features and local structural features of the image to be restored, the original image is initially textured to roughly restore the basic structure, contour, and missing parts of the image, thus laying the foundation for subsequent fine restoration.

[0097] Specifically, after obtaining an original image that has not been processed by image restoration and has damaged or missing parts, the original image is input into a pre-trained second texture feature extractor to obtain a second texture feature map output by the second texture feature extractor. The second texture feature map is input into a texture restoration model to obtain an image to be restored that has undergone initial texture restoration and is output by the texture restoration model.

[0098] The image restoration method provided by the present invention fully extracts the texture features of the original missing and damaged image by using a texture feature extractor constructed based on a DSCNN model, etc., and initially restores the original missing and damaged image by using a texture restoration model constructed based on a diffusion model, etc., and preliminarily restores the missing parts, basic structures and contours in the original missing and damaged image, which helps to more fully extract the texture features and structural features of the image to be restored when effectively fusing the local texture features and local structural features of the image to be restored by using a global bidirectional attention interaction optimization mechanism, so as to ultimately obtain an image with a better restoration effect.

[0099] In order to better illustrate the image restoration method provided by the present invention, an embodiment is provided below to elaborate on the complete process of image restoration.

[0100] Figure 2 FIG. 2 is a flow chart of the image restoration method provided by the present invention. Figure 2 As shown, in order to further improve the quality of image restoration, a strategy of performing secondary restoration on the basis of the initial restoration of the image is adopted.

[0101] In the initial restoration, the original image that has not been restored and has damaged or missing parts is obtained by reading it from a local storage device or downloading it from a network server. The original image is input into a second texture feature extractor built based on the DSCNN model and using a 3*3 small convolution kernel. The second texture feature map output by the second texture feature extractor is obtained, which effectively maintains the sensitivity to local details. The second texture feature map is input into a texture restoration model built based on a diffusion model, and the image to be restored is output by the texture restoration model.

[0102] Although the initial restoration of the image restores the texture detail information of the image to a certain extent and effectively fills the blank areas in the image, the integrity of its overall structure and pattern is still insufficient. When processing complex textures and structures, the following problems may occur: some complex textures in the texture restoration image are not accurately restored, especially in places where the texture changes are rich or irregular; for large areas of missing or containing important edge and contour information, the initial restoration may not be able to completely maintain these key structural features, resulting in the restored image not looking coherent or real enough. In addition, the initial restoration focuses more on the matching of local textures, so when processing scenes with global consistency requirements (such as the sky, water surface, etc.), there may be situations that are inconsistent with the surrounding environment. Therefore, in order to further improve the quality of image restoration, this embodiment adopts a strategy of performing secondary texture and structure restoration on the basis of the initial texture restoration of the image. At the same time, the initial restoration of the image can quickly fill the missing parts of the image and roughly restore the basic structure and contour of the image, thereby providing richer features for the secondary fine restoration.

[0103] In the secondary restoration, the image to be restored is input into the first texture feature extractor and the structural feature extractor constructed based on the DSCNN model, and the first texture feature map output by the first texture feature extractor using a smaller convolution kernel and fewer downsampling operations, as well as the structural feature map output by the structural feature extractor using a larger convolution kernel and more downsampling operations, are obtained. The texture features and structural features of the image that have been initially restored are fully extracted and mined, and a richer feature representation is generated, ensuring that the consistency and coherence of the overall structure are retained while restoring the details, thereby enhancing the realism of the restoration result.

[0104] The first texture feature map is fine-grainedly deconstructed to obtain a set of local texture feature matrices, and the structure feature map is fine-grainedly deconstructed to obtain a set of local structure feature matrices, so as to analyze the local information in the feature map in more detail.

[0105] A global bidirectional attention interaction optimization is performed on the local texture feature matrix set and the local structure feature matrix set. That is, on the one hand, the local texture feature matrix in the local texture feature matrix set is used as the first query feature matrix, and the local structure feature matrix set is used as the first key feature matrix set. The first query feature matrix and the first key feature matrix set are input into the first converter structure based on the Transformer model. The first key feature matrix set is used to perform global attention interaction optimization on each first query feature matrix to obtain the texture feature global optimization matrix corresponding to each first query feature matrix, and the texture feature global optimization matrix set is determined according to the texture feature global optimization matrix. On the other hand, the local structure feature matrix in the local structure feature matrix set is used as the second query feature matrix, and the local texture feature matrix set is used as the second key feature matrix set. The second query feature matrix and the second key feature matrix set are input into the second converter structure based on the Transformer model. The second key feature matrix set is used to perform global attention interaction optimization on each second query feature matrix to obtain the structure feature global optimization matrix corresponding to each second query feature matrix, and the structure feature global optimization matrix set is determined according to the structure feature global optimization matrix. This attention joint perception method, which uses a bidirectional attention mechanism to perform multi-level feature extraction and capture the relationship between texture features and structural features, more comprehensively and intelligently captures the global bidirectional semantic correlation information between the first texture feature map and the structural feature map, realizes the mutual enhancement, complementation and effective fusion of texture features and structural features, better understands the overall structure and local details of the image, maintains the overall consistency and coherence of the image, ensures that the restored image is more visually harmonious and unified, and significantly improves the restoration quality.

[0106] The texture feature global optimization matrix set is feature aggregated along the channel dimension to obtain a complete texture global optimization feature map; the structure feature global optimization matrix set is feature aggregated along the channel dimension to obtain a complete structure global optimization feature map, and based on the weighted summation of the texture global optimization feature map and the structure global optimization feature map, a texture-structure joint perception feature map of texture and structure feature interaction is obtained. The texture-structure joint perception feature map is input into the image restoration model built based on the diffusion model to generate a more refined restoration image.

[0107] Overall, the texture features of the image to be repaired are enriched by the initial restoration of the original image; further, the texture features and structural features of the image to be repaired with rich texture features are extracted at multiple levels, and the texture features and structural features of the image to be repaired are jointly perceived by the global two-way attention, and the texture features and structural features of the image to be repaired are comprehensively considered and deeply mined, and then the texture-structure joint perception feature map is obtained, so as to perform secondary restoration of the image, generate a more refined restoration image, and more accurately restore the details and texture of the image while retaining the overall structure of the image, so as to obtain a more realistic and natural restoration effect. That is, the image method provided in this embodiment solves the problem that traditional image restoration methods such as those based on interpolation and texture synthesis perform poorly when dealing with large-area missing or complex texture structures, are prone to produce obvious splicing marks, and cannot effectively maintain the overall structure and details of the image. The image method provided in this embodiment can not only handle various complex image restoration tasks, but also accurately restore the details and texture of the image while retaining the overall structure of the image. This is of great significance to many fields such as repairing old photos, restoring artwork images, improving the quality of medical imaging, and improving the quality of video materials.

[0108] Figure 3 Schematic diagram of the structure of the image restoration device provided by the present invention. Figure 3 As shown, the image restoration device includes but is not limited to a matrix set acquisition module 301, a global attention optimization module 302, a joint perception feature map acquisition module 303 and a joint perception feature map restoration module 304.

[0109] The matrix set acquisition module 301 is used to acquire a local texture feature matrix set and a local structure feature matrix set of the image to be restored.

[0110] The global attention optimization module 302 is used to perform global bidirectional attention interactive optimization on the local texture feature matrix set and the local structural feature matrix set to obtain a texture feature global optimization matrix set and a structural feature global optimization matrix set.

[0111] The joint perceptual feature map acquisition module 303 is used to determine a texture-structure joint perceptual feature map based on the texture feature global optimization matrix set and the structure feature global optimization matrix set.

[0112] The joint perceptual feature map restoration module 304 is used to input the texture-structure joint perceptual feature map into the image restoration model to obtain the restored image output by the image restoration model.

[0113] It should be noted that the image restoration device provided by the present invention can execute the image restoration method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0114] The image restoration device provided by the present invention captures and measures the correlation between the local texture feature matrix set and the local structure feature matrix set of the image to be restored by utilizing the bidirectional attention mechanism, performs global bidirectional attention interaction optimization on the local texture feature matrix set and the local structure feature matrix set, and semantically interacts each local feature matrix in one set with all local feature matrices in another set, so as to more comprehensively capture the global bidirectional semantic association information between the texture features and the structure features of the image to be restored, realize the enhancement and supplement of the mutual relationship between the texture features and the structure features of the image to be restored, and generate a texture-structure joint perception feature map by integrating the complementary information of the enhanced texture features and the structure features of the image to be restored, thereby providing a richer feature basis for the fine restoration of the damaged image, and finally realizing image restoration with retaining the overall texture structure of the image, accurately restoring the image details and texture, and having a more realistic and natural restoration effect.

[0115] Figure 4 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 4 As shown, the electronic device may include: a processor (Processor) 410, a communication interface (Communications Interface) 420, a memory (Memory) 430 and a communication bus 440, wherein the processor 410, the communication interface 420, and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the image restoration method provided in any of the above embodiments, and the image restoration method includes but is not limited to the following steps: obtaining a local texture feature matrix set and a local structure feature matrix set of the image to be restored; performing global bidirectional attention interaction optimization on the local texture feature matrix set and the local structure feature matrix set to obtain a texture feature global optimization matrix set and a structure feature global optimization matrix set; determining a texture-structure joint perception feature map based on the texture feature global optimization matrix set and the structure feature global optimization matrix set; inputting the texture-structure joint perception feature map into the image restoration model to obtain a restored image output by the image restoration model.

[0116] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0117] On the other hand, the present invention also provides a computer program product, which includes a computer program, and the computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image restoration method provided in any of the above embodiments, and the image restoration method includes but is not limited to the following steps: obtaining a local texture feature matrix set and a local structural feature matrix set of the image to be restored; performing global bidirectional attention interactive optimization on the local texture feature matrix set and the local structural feature matrix set to obtain a texture feature global optimization matrix set and a structural feature global optimization matrix set; determining a texture-structure joint perception feature map based on the texture feature global optimization matrix set and the structural feature global optimization matrix set; inputting the texture-structure joint perception feature map into an image restoration model to obtain a restored image output by the image restoration model.

[0118] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the image restoration method provided by any of the above embodiments is implemented. The image restoration method includes but is not limited to the following steps: obtaining a local texture feature matrix set and a local structural feature matrix set of the image to be restored; performing global bidirectional attention interactive optimization on the local texture feature matrix set and the local structural feature matrix set to obtain a texture feature global optimization matrix set and a structural feature global optimization matrix set; determining a texture-structure joint perception feature map based on the texture feature global optimization matrix set and the structural feature global optimization matrix set; inputting the texture-structure joint perception feature map into an image restoration model to obtain a restored image output by the image restoration model.

[0119] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0120] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image restoration method, characterized in that: include: Obtaining a local texture feature matrix set and a local structure feature matrix set of the image to be repaired; Performing global bidirectional attention interaction optimization on the local texture feature matrix set and the local structural feature matrix set to obtain a texture feature global optimization matrix set and a structural feature global optimization matrix set; Determining a texture-structure joint perception feature map based on the texture feature global optimization matrix set and the structure feature global optimization matrix set; The texture-structure joint perception feature map is input into an image restoration model to obtain a restoration image output by the image restoration model.

2. The image restoration method according to claim 1, characterized in that: The performing global bidirectional attention interaction optimization on the local texture feature matrix set and the local structure feature matrix set to obtain a texture feature global optimization matrix set and a structure feature global optimization matrix set includes: Determining a first query feature matrix based on the local texture feature matrices in the local texture feature matrix set; Based on the local structure feature matrix set, determining a first key feature matrix set; Based on the first key feature matrix set, the first query feature matrix is ​​optimized by global attention interaction to obtain a texture feature global optimization matrix, so as to determine a texture feature global optimization matrix set based on the texture feature global optimization matrix.

3. The image restoration method according to claim 2, characterized in that: The method of performing global attention interaction optimization on the first query feature matrix based on the first key feature matrix set to obtain a global optimization matrix of texture features includes: Determine first matrix product results based on the product of the first query feature matrix and the transposed matrix of each first key feature matrix in the first key feature matrix set; Determine each first attention weight matrix based on the quotient of each first matrix multiplication result divided by the scale square root of the first key feature matrix; The texture feature global optimization matrix is ​​determined based on the product of the first query feature matrix and the first attention weight matrices.

4. The image restoration method according to claim 1, characterized in that: The performing global bidirectional attention interaction optimization on the local texture feature matrix set and the local structure feature matrix set to obtain a texture feature global optimization matrix set and a structure feature global optimization matrix set includes: Determine a second query feature matrix based on the local structure feature matrices in the local structure feature matrix set; Based on the local texture feature matrix set, determining a second key feature matrix set; Based on the second key feature matrix set, the second query feature matrix is ​​optimized by global attention interaction to obtain a structural feature global optimization matrix, so as to determine the structural feature global optimization matrix set based on the structural feature global optimization matrix.

5. The image restoration method according to claim 4, characterized in that: The method of performing global attention interaction optimization on the second query feature matrix based on the second key feature matrix set to obtain a global optimization matrix of structural features includes: Determine each second matrix product result based on the product of the second query feature matrix and the transposed matrix of each second key feature matrix in the second key feature matrix set; Determine each second attention weight matrix based on the quotient of each second matrix multiplication result divided by the scale square root of the second key feature matrix; The structural feature global optimization matrix is ​​determined based on the product of the second query feature matrix and each of the second attention weight matrices.

6. The image restoration method according to claim 1, characterized in that: The determining of the texture-structure joint perceptual feature map based on the texture feature global optimization matrix set and the structure feature global optimization matrix set comprises: Performing feature aggregation on the texture feature global optimization matrix set along the channel dimension to obtain a texture global optimization feature map; Performing feature aggregation on the structural feature global optimization matrix set along the channel dimension to obtain a structural global optimization feature graph; Based on the texture global optimization feature map and the structure global optimization feature map, the texture-structure joint perception feature map is determined.

7. The image restoration method according to claim 1, characterized in that: The step of obtaining a local texture feature matrix set and a local structure feature matrix set of the image to be repaired includes: Extracting a first texture feature map and a structure feature map from the image to be repaired; Performing fine-grained deconstruction on the first texture feature map to obtain the local texture feature matrix set; The structural feature graph is fine-grainedly deconstructed to obtain the local structural feature matrix set.

8. The image restoration method according to claim 7, characterized in that: The step of extracting a texture feature map and a structure feature map from the image to be repaired comprises: Inputting the image to be repaired into a first texture feature extractor to obtain the first texture feature map output by the first texture feature extractor; Inputting the image to be repaired into a structural feature extractor to obtain the structural feature map output by the structural feature extractor; The loss function of the first texture feature extractor is determined based on texture consistency loss and perceptual loss; the loss function of the structural feature extractor is determined based on structural similarity index loss, shape preservation metric loss, boundary recall rate loss and precision loss.

9. The image restoration method according to claim 7, characterized in that: Before obtaining the local texture feature matrix set and the local structure feature matrix set of the image to be repaired, the method includes: Get the original image; Inputting the original image into a second texture feature extractor to obtain a second texture feature map output by the second texture feature extractor; The second texture feature map is input into a texture restoration model to obtain the image to be restored output by the texture restoration model.

10. An image restoration device, characterized in that: include: A matrix set acquisition module is used to acquire a local texture feature matrix set and a local structure feature matrix set of the image to be repaired; A global attention optimization module, used for performing global bidirectional attention interaction optimization on the local texture feature matrix set and the local structure feature matrix set to obtain a texture feature global optimization matrix set and a structure feature global optimization matrix set; A joint perception feature map acquisition module, used to determine a texture-structure joint perception feature map based on the texture feature global optimization matrix set and the structure feature global optimization matrix set; The joint perception feature map restoration module is used to input the texture-structure joint perception feature map into the image restoration model to obtain the restored image output by the image restoration model.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the image restoration method according to any one of claims 1 to 9 is implemented.

12. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the image restoration method according to any one of claims 1 to 9 is implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image restoration method according to any one of claims 1 to 9 is implemented.

Images