Image optimization method, image optimization device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses an image optimization method, an image optimization device, computer equipment and a computer readable storage medium. The method comprises the following steps that a double-domain attention network is constructed, and the double-domain attention network comprises a plurality of pairs of encoders and decoders; the encoder and the decoder process the features through a double-domain attention residual block; the system specifically comprises a space attention module and a frequency attention module, the space attention module is used for determining an important area in an image, and the frequency attention module is used for determining a fuzzy area in the image; designing a joint loss function, and making a data set for training; the trained double-domain attention network can process an input image into a high-definition image. Therefore, the space attention module and the frequency attention module are introduced to construct the network to process the image, image details and texture information can be kept relatively completely while the image resolution and definition are ensured, and the image processing efficiency is relatively high.

Description

Technical Field

[0001] This application belongs to the field of remote sensing image technology, and in particular relates to an image optimization method, an image optimization device, a computer device, and a computer-readable storage medium. Background Technology

[0002] Transmission lines, with their power transmission function, are a crucial component of the smart grid. As the lifeline of the power supply system, transmission lines bear a vital responsibility for power transmission. Due to long-term exposure to wind, rain, sunlight, and penetrating ultraviolet radiation, transmission lines are highly susceptible to aging and corrosion. Abnormalities in these lines can cause serious grid system failures, potentially leading to the complete grid shutdown and impacting public safety and electricity supply. To ensure the safe operation of the power system and eliminate potential hazards, grid inspections are essential. To improve the quality and efficiency of transmission line inspections, thereby alleviating the pressure on power line operation and maintenance and ensuring the safe and stable operation of the grid, an increasing number of power supply companies are using drones as a common tool for transmission line inspections.

[0003] However, during drone inspections, adverse weather conditions such as wind and rain may occur, inevitably causing drones to vibrate during image acquisition. Furthermore, the image acquisition sensors carried by drones are limited by weight and size, resulting in low image resolution and blurry images. This substandard data can severely interfere with subsequent detection of power line defects and potential hazards, failing to provide strong assurance for the safe operation of the power system. Therefore, how to optimize data blurring caused by drone flight vibration is a technical problem that urgently needs to be solved by those skilled in the art.

[0004] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0005] Based on this, it is necessary to propose an image optimization method, an image optimization device, a computer device, and a computer-readable storage medium to address the above problems, which can improve the readability of UAV disaster inspection image data.

[0006] The technical problem solved by this application is achieved by the following technical solution:

[0007] This application provides an image optimization method, comprising the following steps: constructing a dual-domain attention network, which includes n encoders and decoders, where n is an integer greater than or equal to 1; the encoders and decoders are connected sequentially, with the encoder scale decreasing stepwise and the decoder scale increasing stepwise, forming a U-shaped architecture; the encoders and decoders include convolutional layers and dual-domain attention residual blocks, the convolutional layers being used to extract shallow features from the input feature image; the dual-domain attention residual blocks including a spatial attention module and a frequency attention module, the spatial attention module being used to determine important regions within the image, and the frequency attention module being used to determine important regions within the image. The algorithm identifies blurred regions; an encoder extracts deep features from the input feature image based on the dual-domain attention residual block within the encoder, with deep features extracted from important regions and / or blurred regions; a decoder converts the deep features of the input into a high-resolution image based on the dual-domain attention residual block of the decoder and outputs it; a joint loss function is designed, and a dataset is created; a dual-domain attention network is trained based on the joint loss function and the dataset; the image to be processed is input into the trained dual-domain attention network for processing to obtain an output image, the clarity of which is higher than that of the image to be processed, and the resolution of the output image is higher than or equal to the resolution of the image to be processed.

[0008] In an optional embodiment of this application, the image to be processed is input into a trained dual-domain attention network for processing, including: a spatial attention module acquiring the feature image output by the convolutional layer; obtaining a query vector, a key vector, and a numerical vector by performing a linear transformation on the feature image; calculating a first attention weight between each pixel of the feature image based on the query vector and the key vector, the first attention weight being used to represent the similarity between pixels; and calculating a second attention weight based on the first attention weight and the numerical vector, the second attention weight being used to represent the important part of the feature image in the spatial domain.

[0009] In an optional embodiment of this application, the spatial attention module includes a spatial pooling layer, a spatial convolutional layer, an activation function layer, and a first output layer. The image to be processed is input into a trained dual-domain attention network for processing, including: acquiring the feature image output by the convolutional layer and inputting it into the spatial pooling layer; the spatial pooling layer generates global features by performing global average pooling on the feature image and inputting these features into the spatial convolutional layer; the spatial convolutional layer adjusts the channel dimension based on the global features to obtain a parameter matrix; the spatial convolutional layer inputs the global features and parameter matrix into the activation function layer; the activation function layer processes the global features and parameter matrix using a hyperbolic tangent activation function to obtain a third attention weight; the first output layer divides the feature image into several groups, each group sharing the same third attention weight in both the channel and spatial dimensions; acquiring the group features of each group; obtaining the local self-attention description of each group based on the group features and the third attention weight; and combining all local self-attention descriptions to obtain a second attention weight, which represents the important part of the feature image in the spatial domain.

[0010] In an optional embodiment of this application, the image to be processed is input into a trained dual-domain attention network for processing, including: a frequency attention module comprising a global frequency modulation layer, a local frequency modulation layer, and a second output layer; obtaining the feature image output by the convolutional layer and inputting it into the global frequency modulation layer, which processes the feature image through global average pooling to obtain a global low-frequency component; processing the feature image and the global low-frequency component to obtain a global high-frequency component; inputting the feature image into the local frequency modulation layer, which processes the feature image through local average pooling to obtain a local low-frequency component; processing the feature image and the local low-frequency component to obtain a local high-frequency component; inputting the global low-frequency component, the global high-frequency component, the local low-frequency component, and the local high-frequency component into the second output layer, which obtains channel weights, modulates the global low-frequency component, the global high-frequency component, the local low-frequency component, and the local high-frequency component according to the channel weights to obtain a fourth attention weight and outputting it, the fourth attention weight being used to determine blurred regions within the image.

[0011] In an optional embodiment of this application, a joint loss function is designed, including: acquiring real image information, predicted image information, multi-scale index, and normalization factor, wherein the real image information is included in the dataset, and the predicted image information and multi-scale index are acquired during the processing of the input image by the dual-domain attention network; constructing a spatial domain loss function based on the real image information, predicted image information, and multi-scale index; constructing a frequency domain loss function based on the real image information, predicted image information, multi-scale index, and normalization factor; and constructing a joint loss function based on the spatial domain loss function, frequency domain loss function, and hyperparameters.

[0012] In an optional embodiment of this application, training a dual-domain attention network based on a joint loss function and a dataset includes: obtaining learning requirements, which include a learning rate and / or the number of iterations; obtaining a training set from the dataset, which includes low-resolution blurred images and high-resolution sharp images; substituting the low-resolution blurred images from the training set into the dual-domain attention network for processing; substituting the pseudo-high-resolution sharp images and high-resolution sharp images output by the dual-domain attention network into the joint loss function to calculate the loss; and optimizing and iterating the dual-domain attention network based on the loss, iterating repeatedly until the learning requirements are met.

[0013] In an optional embodiment of this application, after iterating repeatedly until the learning requirements are met, the process includes: processing the dataset according to a preset method to obtain a test set, the test set including multiple pairs of low-resolution blurred images; inputting the low-resolution blurred images into a dual-domain attention network for processing to obtain pseudo-high-resolution clear images; determining whether the pseudo-high-resolution clear images meet a preset termination condition; if the preset termination condition is not met, obtaining the learning requirements again to train the dual-domain attention network; if the preset termination condition is met, marking the current round of the dual-domain attention network as having completed training, so as to be used to process the images to be processed.

[0014] This application also provides an image optimization apparatus, comprising: a construction module for constructing a dual-domain attention network, the dual-domain attention network including n encoders and decoders, where n is an integer greater than or equal to 1; the encoders and decoders are connected sequentially, with the encoder scale decreasing step by step and the decoder scale increasing step by step, the dual-domain attention network having a U-shaped architecture; the encoders and decoders include convolutional layers and dual-domain attention residual blocks, the convolutional layers being used to extract shallow features of the input feature image; the dual-domain attention residual blocks including a spatial attention module and a frequency attention module, the spatial attention module being used to determine important regions within the image, and the frequency attention module being used to determine blurred regions within the image. The encoder extracts deep features from the input feature image based on the dual-domain attention residual block within the encoder. These deep features are extracted from important regions and / or blurred regions. The decoder converts the deep features of the input into a high-resolution image based on the dual-domain attention residual block within the decoder and outputs it. The training module designs the joint loss function and creates the dataset. It trains the dual-domain attention network based on the joint loss function and the dataset. The processing module inputs the image to be processed into the trained dual-domain attention network to obtain the output image. The clarity of the output image is higher than that of the image to be processed, and the resolution of the output image is higher than or equal to the resolution of the image to be processed.

[0015] This application also provides a computer device including a processor and a memory: the processor is used to execute a computer program stored in the memory to implement the method as described above.

[0016] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described above.

[0017] The embodiments of this application have the following beneficial effects:

[0018] This application introduces a spatial attention module and a frequency attention module to construct a dual-domain attention network for improved image processing performance. The spatial attention module identifies important regions within an image, while the frequency attention module identifies blurred regions. Together, they optimize the input image. The trained dual-domain attention network preserves image details and texture information while maintaining image resolution and sharpness, and boasts fast real-time inference speed, resulting in high image processing efficiency.

[0019] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it according to the contents of the specification, and to make the above and other objects, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and do not limit this application. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] in:

[0022] Figure 1 This is a flowchart illustrating an image optimization method provided in one embodiment;

[0023] Figure 2 A schematic diagram of the overall network architecture of a dual-domain attention network provided in one embodiment;

[0024] Figure 3 A schematic diagram of experimental results for a dual-domain attention network provided in one embodiment;

[0025] Figure 4 A schematic diagram of the network architecture of a spatial attention module provided in one embodiment;

[0026] Figure 5 A schematic diagram of the network architecture of a frequency attention module provided in one embodiment;

[0027] Figure 6 A diagram showing the internal functional modules of an image optimization device according to one embodiment;

[0028] Figure 7 This is a schematic block diagram of the structure of a computer device provided in one embodiment. Detailed Implementation

[0029] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0030] Transmission lines, with their power transmission function, are a crucial component of the smart grid. As the lifeline of the power supply system, transmission lines bear a vital responsibility for power transmission. Due to long-term exposure to wind, rain, sunlight, and penetrating ultraviolet radiation, transmission lines are highly susceptible to aging and corrosion. Abnormalities in these lines can cause serious grid system failures, potentially leading to the complete grid shutdown and impacting public safety and electricity supply. To ensure the safe operation of the power system and eliminate potential safety hazards, regular grid inspections are essential.

[0031] In recent years, with the increase in voltage levels of overhead high-voltage transmission lines, coupled with the growing height and complexity of high-voltage towers, the length of lines requiring maintenance has increased significantly, greatly increasing the pressure on transmission line maintenance units. To improve the quality and efficiency of transmission line inspections, thereby alleviating the pressure on power line operation and maintenance and ensuring the safe and stable operation of the power grid, more and more power supply companies are using drones as a common tool for transmission line inspections. This new drone inspection technology can not only significantly reduce various costs associated with line maintenance but also lower the probability of accidents involving inspection personnel, which will drive the development of intelligent transmission line maintenance.

[0032] Manual inspections are mostly conducted by experienced inspectors with knowledge of the lines, who observe and assess the operational status of the lines visually and with binoculars. Manual inspections are sometimes affected by external environmental factors, and in most cases, only the basic infrastructure and some towers are inspected. Drones, however, offer many advantages and are significant for improving the quality and efficiency of transmission line inspections: First, they are not limited by terrain. Drones can fly from any position in the air, observing the high-voltage lines to be inspected from multiple angles. Simultaneously, drones can transmit monitoring data back to inspectors for real-time analysis, greatly reducing the operational risks of manual inspections. Second, they are not affected by weather conditions. In severe weather such as heavy rain and strong winds, the quality and efficiency of manual inspections will be significantly reduced. Modern drones have strong wind and rain resistance capabilities, allowing them to complete line inspections even in adverse weather conditions. Third, they are relatively easy to operate. Small drones can be carried by inspectors. Larger drones can be transported by vehicle. Furthermore, drone operation has a low barrier to entry; inspectors can use them after simple training. After a few tens of seconds of preparation, the drone can take off immediately and conduct a comprehensive inspection and photography of the power transmission line in a short period of time, with relatively stable inspection efficiency.

[0033] However, due to adverse weather conditions such as wind and rain that may occur during drone inspections, drones inevitably experience shaking during image acquisition. Furthermore, the image acquisition sensors carried by drones are limited by weight and size, resulting in low image resolution and blurry images. This substandard data can severely interfere with subsequent detection of power line defects and potential hazards, failing to provide a strong guarantee for the safe operation of the power system.

[0034] Therefore, existing technologies for optimizing data blur caused by drone flight jitter include manual detection, infrared imaging, and ultraviolet imaging. Each of these methods has its own advantages and disadvantages, and their application scope is limited. To solve this technical problem, this application proposes an image optimization method, including steps S110 to S130. For a clear description of the image optimization method provided in this embodiment, please refer to... Figures 1-6 .

[0035] Step S110: Construct a dual-domain attention network, which includes n encoders and decoders, where n is an integer greater than or equal to 1; the encoders and decoders are connected sequentially, with the encoder scale decreasing step by step and the decoder scale increasing step by step, forming a U-shaped architecture; the encoders and decoders include convolutional layers and dual-domain attention residual blocks, with the convolutional layers used to extract shallow features from the input feature image; the dual-domain attention residual blocks include a spatial attention module and a frequency attention module, with the spatial attention module used to determine important regions within the image and the frequency attention module used to determine blurred regions within the image; the encoder extracts deep features from the input feature image based on the dual-domain attention residual blocks within the encoder, with deep features extracted from important regions and / or blurred regions; the decoder converts the input deep features into a high-resolution image based on the dual-domain attention residual blocks within the decoder and outputs it.

[0036] In one embodiment, for ease of explanation, the overall network architecture of the dual-domain attention network constructed in this application can be referred to Figure 2 A dual-domain attention network consists of n encoders and decoders, where n is an integer greater than or equal to 1. For clarity, please refer to [link to documentation / reference]. Figure 2 In this application, the encoder and decoder can each have three units, connected sequentially, each including three scales. The encoder scale decreases progressively, meaning the dimension or size of the output feature vector decreases with each encoder scale, thus gradually extracting deep features from the feature image. Conversely, the decoder scale increases progressively, meaning the dimension or size of the output feature vector increases with each decoder scale, enabling the decoder to reconstruct the input feature image into a higher-resolution, clearer image. Due to this scale relationship, a dual-domain attention network, such as... Figure 2 As shown, the overall shape is U-shaped.

[0037] Furthermore, the encoder and decoder architectures are identical at each scale, including convolutional layers and dual-domain attention residual blocks. The convolutional layers extract shallow features from the input feature image to facilitate processing by the dual-domain attention residual blocks. To simultaneously bridge the gap between spatial and frequency domain degradation and high-resolution sharp image pairs, this application introduces a dual-domain attention mechanism that can simultaneously satisfy image super-resolution and deblurring, enhancing representation learning capabilities through spatial and frequency attention modules. Specifically, the Spatial Attention Module (SAM) efficiently simulates self-attention, thereby reducing computational complexity. SAM generates selective attention weights from global features and focuses on local regions. In addition, a simple and effective Frequency Attention Module (FAM) is proposed, which highlights useful frequency subbands by decoupling and modulating frequency components. Finally, the two attention modules are integrated into the dual-domain attention residual block to embed them within the dual-domain attention network.

[0038] Step S120: Design a joint loss function and create a dataset; train a dual-domain attention network based on the joint loss function and the dataset.

[0039] In one embodiment, step S120: designing a joint loss function includes: acquiring real image information, predicted image information, multi-scale index, and normalization factor, wherein the real image information is included in the dataset, and the predicted image information and multi-scale index are acquired during the processing of the input image by the dual-domain attention network; constructing a spatial domain loss function based on the real image information, predicted image information, and multi-scale index; constructing a frequency domain loss function based on the real image information, predicted image information, multi-scale index, and normalization factor; and constructing a joint loss function based on the spatial domain loss function, frequency domain loss function, and hyperparameters.

[0040] In one embodiment, a loss function is designed based on the dual-domain attention mechanism proposed in this application to efficiently promote dual-domain learning and ensure good image processing performance. Specifically, it can obtain real image information y within the dataset. r And the predicted image information obtained during the dual-domain attention network processing of the input image. and multi-scale index R Therefore, loss functions are designed for the spatial domain where the spatial attention module resides and the frequency domain where the frequency attention module resides, respectively. For the spatial domain loss function L... spa It can be represented as:

[0041]

[0042] For the frequency domain loss function L freThen it can be calculated using the following formula:

[0043]

[0044] In the above formula, S r It means The normalization factor for the total number of elements. Obtaining the hyperparameter λ, and combining equations (1) and (2), we can obtain the joint loss function L used to train the dual-domain attention network, which can be specifically expressed as:

[0045] L = L spa +λL fre (3)

[0046] In one implementation, the hyperparameter λ can be set to 0.1 empirically to balance dual-domain learning.

[0047] In one embodiment, step S120: training a dual-domain attention network based on a joint loss function and a dataset includes: obtaining learning requirements, which include a learning rate and / or the number of iterations; obtaining a training set from the dataset, which includes low-resolution blurred images and high-resolution sharp images; substituting the low-resolution blurred images from the training set into the dual-domain attention network for processing; substituting the pseudo-high-resolution sharp images and high-resolution sharp images output by the dual-domain attention network into the joint loss function to calculate the loss; and optimizing and iterating the dual-domain attention network based on the loss, iterating repeatedly until the learning requirements are met.

[0048] In one embodiment, an arbitrarily set learning requirement is obtained. In this embodiment, for ease of explanation, the learning requirement can be set as follows: DDANet is trained using the Adam optimizer with an initial learning rate of 1×10⁻⁶. -4 The cosine annealing strategy was used to reduce it to 1×10. -6 The network was trained on 256×256 patches with a batch size of 4 for 300 epochs and tested at full resolution. Data augmentation was applied randomly with a probability of 0.5 for horizontal flipping. The kernel size of the SAM module was set to 3×3. Furthermore, low-resolution blurred images and high-resolution clear images were generated from images collected during drone inspections to form a training set, for example, a training set consisting of 600 pairs of low-resolution blurred and high-resolution clear drone inspection images. The low-resolution blurred images from the training set were then fed into a dual-domain attention network for processing. The pseudo-high-resolution clear images and high-resolution clear images output by the dual-domain attention network were then fed into a joint loss function to calculate the loss. The dual-domain attention network was then optimized iteratively based on the loss, iterating repeatedly until the learning requirements were met.

[0049] In one embodiment, after iterating repeatedly until the learning requirements are met, the process includes: processing the dataset according to a preset method to obtain a test set, the test set including multiple pairs of low-resolution blurred images; inputting the low-resolution blurred images into a dual-domain attention network for processing to obtain pseudo-high-resolution clear images; determining whether the pseudo-high-resolution clear images meet a preset termination condition; if the preset termination condition is not met, obtaining the learning requirements again to train the dual-domain attention network; if the preset termination condition is met, marking the current round of the dual-domain attention network as having completed training, so as to be used to process the images to be processed.

[0050] In one implementation, the test set can be obtained using the same method as the training set, for example, by acquiring 24 low-resolution, blurry drone inspection images as the test set. The images in the test set are then fed into the trained dual-domain attention network for processing to obtain pseudo-high-resolution clear images. The network's training success is determined by whether the pseudo-high-resolution clear images meet a preset termination condition. This preset termination condition can be evaluated using two metrics: PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index). Whether the PSNR and SSIM values ​​meet the preset termination condition is used as an indicator of whether the model has completed training. If either the PSNR or SSIM value fails to meet the preset termination condition, the learning requirements are obtained again to train the dual-domain attention network; if both the PSNR and SSIM values ​​meet the preset termination condition, the dual-domain attention network training is considered complete.

[0051] After training, on the entire test dataset, the PSNR and SSIM scores of the image preprocessing results of the algorithm proposed in this application are 34.05 and 0.975, respectively. For specific resolution restoration results, please refer to... Figure 3 , Figure 3 This is a schematic diagram illustrating the experimental results of the dual-domain attention network. Through comparison with actual images, it can be seen that the super-resolution and deblurring effects of the proposed algorithm are very close to the target image, ensuring high resolution and deblurring of the restored image, while preserving image details and texture information relatively completely. Furthermore, the proposed method guarantees the structural similarity between the pre-processed and optimized image and the target image. Moreover, the optimized image does not exhibit artifact overlap or other phenomena. The algorithm designed in this application has a fast real-time inference speed and high image deblurring efficiency.

[0052] Step S130: Input the image to be processed into the trained dual-domain attention network to obtain an output image. The sharpness of the output image is higher than that of the image to be processed, and the resolution of the output image is higher than or equal to that of the image to be processed.

[0053] In one embodiment, step S130: inputting the image to be processed into a trained dual-domain attention network for processing includes: the spatial attention module acquiring the feature image output by the convolutional layer, and obtaining a query vector, a key vector, and a numerical vector by performing a linear transformation on the feature image; calculating a first attention weight between each pixel of the feature image based on the query vector and the key vector, the first attention weight being used to represent the similarity between pixels; and calculating a second attention weight based on the first attention weight and the numerical vector, the second attention weight being used to represent the important part of the feature image in the spatial domain.

[0054] In one embodiment, to improve the representation capability of the extracted features in the spatial domain, the spatial attention module proposed in this application simulates the operation of MHSA in a convolutional manner. That is, the spatial attention module acquires the feature image output by the convolutional layer, and the attention function can be described as mapping the query vector and a set of key-value pairs to the output, and obtaining the query vector Q, key vector K and value vector V by linear transformation of the feature image.

[0055] MHSA(Q,K,V)=Concat(head1,...,head h W O (4)

[0056]

[0057] Among them W i Q W i K , This is the parameter matrix that projects Q, K, V onto different representation subspaces. The core component of Q, K, V is the scaled dot product attention:

[0058]

[0059] Where, d k The dimension of the key is represented by H×W×C. Here, H (Height) represents the height of the image, i.e., the number of pixels in the vertical direction. W (Width) represents the width of the image, i.e., the number of pixels in the horizontal direction. C (Channels) represents the number of channels in the image, i.e., how many color channels the image consists of. The size and result of the attention map are HW×HW and HW×C, respectively. Assuming that Q, K, V in formula (6) have the same number of channels as the input, therefore,

[0060] Furthermore, the weights of other pixels relative to the target pixel are calculated locally. Specifically, for a single output pixel of equation (6), it can be calculated using the following formula:

[0061]

[0062] Where i, p, j represent pixel coordinates on the image, i is the coordinate of the current pixel, p is the coordinate of the center of the group to which the current pixel belongs, and j is the coordinate of other pixels in the group to which the current pixel belongs. A i,p This refers to the first attention weight in the spatial attention module, used to represent the similarity between pixels. For y i,j That is, the output value, specifically the second attention weight, which is used to represent the important part of the feature image in the spatial domain. A i,p The value can be calculated from the query vector and the key vector. The specific calculation process can be given by the following formula:

[0063]

[0064] In one embodiment, the spatial attention module includes a spatial pooling layer, a spatial convolutional layer, an activation function layer, and a first output layer. Step S130: The image to be processed is input into the trained dual-domain attention network for processing, including: obtaining the feature image output by the convolutional layer and inputting it into the spatial pooling layer; the spatial pooling layer generates global features by performing global average pooling on the feature image and inputting them into the spatial convolutional layer; the spatial convolutional layer adjusts the channel dimension according to the global features to obtain a parameter matrix; the spatial convolutional layer inputs the global features and the parameter matrix into the activation function layer; the activation function layer processes the global features and the parameter matrix using a hyperbolic tangent activation function to obtain a third attention weight; the first output layer divides the feature image into several groups, each group sharing the same third attention weight in both the channel and spatial dimensions; the group features of each group are obtained, and the local self-attention description of each group is obtained based on the group features and the third attention weight; all local self-attention descriptions are combined to obtain a second attention weight, which is used to represent the important part of the feature image in the spatial domain.

[0065] In one embodiment, and for the purpose of specifically describing the spatial attention module, refer to Figure 4 , Figure 4 This is a schematic diagram of the network architecture for the spatial attention module. For simplicity, the key dimension d in equation (6) is omitted in this application. k The scaling factor represented by the two pixels. And from equation (8), it can be concluded that by considering only the information of these two pixels in the channel dimension and ignoring other pixels, the weight of another pixel on the target pixel can be generated.

[0066] Therefore, in order to obtain the weights for every two pixels from the context information, the spatial attention module provided in this application includes a spatial pooling layer, a spatial convolutional layer, an activation function layer, and a first output layer. The spatial pooling layer uses Global Average Pooling (GAP) to generate a global feature GAP(X), and then the spatial convolutional layer is used to adjust the channel dimension.

[0067] Since propagating all information in the spatial domain can have side effects, this application proposes an activation function layer to reduce the scope of propagation and performs local self-attention. Furthermore, unlike ordinary self-attention, the activation function layer can use a hyperbolic tangent activation function to generate negative weights for harmful pixels to suppress their negative impact. Given an input feature image... The third attention weights for all groups are given by formula (9).

[0068] W = Tanh(W1*GAP(X)) (9)

[0069] Here, W1 is the parameter matrix of the convolutional layer, representing the convolution operation, which is obtained by adjusting the channel dimension based on global features through the spatial convolutional layer. k is the size of the attention kernel, and g is the number of groups.

[0070] After group convolution, the first output layer divides the input X into several groups, but each group shares the same third attention weights in both channel and spatial dimensions. This measure can significantly reduce the number of third attention weights and parameters, thereby alleviating the training difficulty. After obtaining the third attention weights and group features, local self-attention is described as...

[0071] S index =W index *X index (10)

[0072] Here, `index` is the group index, intended to promote interaction between different groups, as in Equation (4) of MHSA. (See reference) Figure 4 In this application, another convolutional module is applied in the first output layer to obtain the second attention weights for the final output:

[0073] SAM(X) = Concat(S1,S2,...,S...) g W2 (11)

[0074] That is, all local self-attention descriptions are combined to obtain the second attention weight, which is used to represent the important part of the feature image in the spatial domain.

[0075] As can be seen, the second attention weights processed by the spatial attention module in this application differ from ordinary self-attention in the following aspects: (i) the attention weights are generated based on contextual information. (ii) the integration range is restricted to a small region to reduce computational complexity. (iii) negative weights are generated through a hyperbolic tangent activation function to suppress the influence of harmful pixels.

[0076] In one embodiment, step S130: inputting the image to be processed into a trained dual-domain attention network for processing includes: a frequency attention module including a global frequency modulation layer, a local frequency modulation layer, and a second output layer; acquiring the feature image output by the convolutional layer and inputting it into the global frequency modulation layer, where the global frequency modulation layer processes the feature image through global average pooling to obtain a global low-frequency component; processing the feature image and the global low-frequency component to obtain a global high-frequency component; inputting the feature image into the local frequency modulation layer, where the local frequency modulation layer processes the feature image through local average pooling to obtain a local low-frequency component; processing the feature image and the local low-frequency component to obtain a local high-frequency component; inputting the global low-frequency component, the global high-frequency component, the local low-frequency component, and the local high-frequency component into the second output layer, where the second output layer acquires channel weights, modulates the global low-frequency component, the global high-frequency component, the local low-frequency component, and the local high-frequency component according to the channel weights to obtain a fourth attention weight and outputting it, whereby the fourth attention weight is used to determine blurred regions within the image.

[0077] In one embodiment, the low-frequency components of a sharp image are similar to those of a blurred image, while there are significant differences between the high-frequency components. Inspired by this idea, this application designs a frequency attention module to process different frequency sub-bands individually to achieve efficient frequency learning. For details on the frequency attention module, please refer to [link / reference needed]. Figure 5 , Figure 5 The frequency attention module includes a global frequency modulation layer, a local frequency modulation layer, and a second output layer.

[0078] The frequency attention module mainly includes two steps in the processing: decomposition and modulation. (Refer to...) Figure 5 The global frequency modulation layer (GAP) represents global average pooling; the local frequency modulation layer (AP) represents average pooling using a 3×3 convolutional kernel. Modulation is achieved through channel-by-channel recalibration, where attention weights are set to directly learnable parameters without introducing any additional sublayers. The simplest decoupling method is to divide the spectrum into two parts: the lowest frequency component and the opposite frequency component. To achieve this, an average pooling-based FAM is constructed, the result of which is proportional to the lowest frequency component in the spectrum.

[0079] In the global frequency modulation layer, given the input feature image is Global low-frequency components can be extracted using global average pooling. Furthermore, global high-frequency components can be obtained by processing the feature image and the global low-frequency components; specifically, high frequencies can be obtained by subtracting the obtained low frequencies from X. This process can be described by the following formula:

[0080]

[0081] in, For global low-frequency components, This refers to the global high-frequency components. Furthermore, due to the crucial role of the receptive field in image deblurring, besides the global frequency modulation layer implementing global frequency modulation, a fine-grained pooling operation can be added to the local branches through a local frequency modulation layer. This utilizes a 3×3 average pooling kernel to extract the local lowest frequency, which is then used as the local low-frequency component. Approximately, for the local low-frequency component... and local high-frequency components The process of obtaining can be represented as:

[0082]

[0083] Finally, the second output layer acquires the previously acquired global low-frequency components. For global high-frequency components Local low-frequency components and local high-frequency components The output is modulated. Specifically, the second output layer modulates these values ​​using channel attention weights. The final fourth attention weight output by the frequency attention module is obtained by summing these elements; this fourth attention weight is used to determine blurred regions within the image. The process of obtaining the fourth attention weight can be calculated using the following formula:

[0084]

[0085] The channel weights W are learnable parameters that are directly optimized through backpropagation. For simplicity, no additional sublayers are needed to obtain the attention weights.

[0086] Furthermore, the specific settings for the two attention modules can be referenced as follows. There are a total of 20 residual blocks per scale. Local frequency attention modules are used in the last 4 blocks, while global frequency attention modules are used in the last 8 blocks. The specific architecture and functions of the local and global frequency attention modules will be explained later. The spatial attention module is only used in the last residual block. The total number of residual blocks is the same at each scale. Starting from the first image, the encoder network gradually reduces the spatial size while doubling the number of channels. The decoder performs in the opposite manner to the encoder. In addition, feature-level and image-level skip connections are applied. Finally, degraded images are added via global skip connections to generate a sharp image.

[0087] Therefore, this application introduces a spatial attention module and a frequency attention module to construct a dual-domain attention network to achieve better image processing performance. The spatial attention module identifies important regions within the image, while the frequency attention module identifies blurred regions. Together, they optimize the input image. The trained dual-domain attention network can preserve image details and texture information relatively completely while maintaining image resolution and sharpness, and it also boasts fast real-time inference speed, resulting in high image processing efficiency.

[0088] Figure 6 A diagram illustrating the internal functional modules of an image optimization device in one embodiment is shown. The image optimization device 60 includes: a construction module 61, a training module 62, and a processing module 63. The construction module 61 constructs a dual-domain attention network, which includes n encoders and decoders, where n is an integer greater than or equal to 1. The encoders and decoders are connected sequentially, with the encoder scale decreasing progressively and the decoder scale increasing progressively, forming a U-shaped architecture. The encoders and decoders include convolutional layers and dual-domain attention residual blocks. The convolutional layers extract shallow features from the input feature image. The dual-domain attention residual blocks include a spatial attention module and a frequency attention module. The spatial attention module identifies important regions within the image, and the frequency attention module identifies blurred regions. The encoder extracts deep features from the input feature image based on the dual-domain attention residual blocks within the encoder. These deep features are extracted from important regions and / or blurred regions. The decoder converts the deep features of the input image into a high-resolution image based on the dual-domain attention residual blocks within the decoder and outputs it. The training module 62 designs a joint loss function and creates a dataset; it then trains the dual-domain attention network based on the joint loss function and the dataset. The processing module 63 is used to input the image to be processed into the trained dual-domain attention network to obtain an output image. The sharpness of the output image is higher than that of the image to be processed, and the resolution of the output image is higher than or equal to that of the image to be processed.

[0089] Figure 7 An internal structural diagram of a computer device in one embodiment is shown. This computer device can specifically be a terminal or a server. Figure 7 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and may also store a computer program that, when executed by the processor, enables the processor to implement an image optimization method. The internal memory may also store a computer program that, when executed by the processor, enables the processor to implement the image optimization method. Those skilled in the art will understand that… Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0090] In one embodiment, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the steps of the method described above.

[0091] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0092] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0093] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. An image optimization method, characterized in that, Includes the following steps: A dual-domain attention network is constructed, comprising n encoders and decoders, where n is an integer greater than or equal to 1; the encoders and decoders are connected sequentially, with the encoder scale decreasing step by step and the decoder scale increasing step by step, and the dual-domain attention network having a U-shaped architecture; the encoders and decoders include convolutional layers and dual-domain attention residual blocks, and the convolutional layers are used to extract shallow features from the input feature image; The dual-domain attention residual block includes a spatial attention module and a frequency attention module. The spatial attention module is used to determine important regions within the image, and the frequency attention module is used to determine blurred regions within the image. The encoder extracts deep features of the input feature image based on the dual-domain attention residual block within the encoder. The deep features are extracted from the important regions and / or the blurred regions. The decoder converts the input deep features into a high-definition image and outputs it based on the dual-domain attention residual block within the decoder. Design a joint loss function and create a dataset; train the dual-domain attention network based on the joint loss function and the dataset; The image to be processed is input into the trained dual-domain attention network to obtain an output image. The sharpness of the output image is higher than that of the image to be processed, and the resolution of the output image is higher than or equal to that of the image to be processed.

2. The image optimization method as described in claim 1, characterized in that, The step of inputting the image to be processed into the trained dual-domain attention network includes: The spatial attention module acquires the feature image output by the convolutional layer, and obtains the query vector, key vector, and numerical vector by linear transformation of the feature image. The first attention weight between each pixel of the feature image is calculated based on the query vector and the key vector, and the first attention weight is used to represent the similarity between pixels. A second attention weight is calculated based on the first attention weight and the numerical vector. The second attention weight is used to represent the important part of the feature image in the spatial domain.

3. The image optimization method as described in claim 1, characterized in that, The spatial attention module includes a spatial pooling layer, a spatial convolutional layer, an activation function layer, and a first output layer; The step of inputting the image to be processed into the trained dual-domain attention network includes: The feature image output by the convolutional layer is obtained and input into the spatial pooling layer. The spatial pooling layer generates global features by performing global average pooling on the feature image and then inputs them into the spatial convolutional layer. The spatial convolutional layer adjusts the channel dimension based on the global features to obtain a parameter matrix; The spatial convolutional layer inputs the global features and the parameter matrix into the activation function layer, and the activation function layer uses the hyperbolic tangent activation function to process the global features and the parameter matrix to obtain the third attention weights. The first output layer divides the feature image into several groups, each group sharing the same third attention weight in both channel and spatial dimensions; it obtains the group features of each group, and calculates the local self-attention description of each group based on the group features and the third attention weight; it combines all the local self-attention descriptions to obtain the second attention weight, which is used to represent the important part of the feature image in the spatial domain.

4. The image optimization method as described in claim 1, characterized in that, The step of inputting the image to be processed into the trained dual-domain attention network includes: The frequency attention module includes a global frequency modulation layer, a local frequency modulation layer, and a second output layer. The feature image output by the convolutional layer is obtained and input into the global frequency modulation layer. The global frequency modulation layer processes the feature image through global average pooling to obtain the global low-frequency component. The global high-frequency component is obtained by processing the feature image and the global low-frequency component. The feature image is input into the local frequency modulation layer, which processes the feature image through local average pooling to obtain local low-frequency components; local high-frequency components are obtained based on the feature image and the local low-frequency components. The global low-frequency component, the global high-frequency component, the local low-frequency component, and the local high-frequency component are input into the second output layer. The second output layer obtains the channel weights and modulates the global low-frequency component, the global high-frequency component, the local low-frequency component, and the local high-frequency component according to the channel weights to obtain the fourth attention weights and output them. The fourth attention weights are used to determine the blurred regions in the image.

5. The image optimization method as described in claim 1, characterized in that, The design of the joint loss function includes: The system acquires real image information, predicted image information, multi-scale index, and normalization factor. The real image information is included in the dataset. The predicted image information and the multi-scale index are acquired by the dual-domain attention network during the processing of the input image. A spatial domain loss function is constructed based on the real image information, the predicted image information, and the multi-scale index; a frequency domain loss function is constructed based on the real image information, the predicted image information, the multi-scale index, and the normalization factor. The joint loss function is constructed based on the spatial domain loss function, the frequency domain loss function, and the hyperparameters.

6. The image optimization method according to any one of claims 1 to 5, characterized in that, Training the dual-domain attention network based on the joint loss function and the dataset includes: Obtain the learning requirements, which include the learning rate and / or the number of iterations; Obtain the training set from the dataset, which includes low-resolution blurred images and high-resolution clear images; The low-resolution blurred images in the training set are substituted into the dual-domain attention network for processing. The pseudo-high-resolution clear image and the high-resolution clear image output by the dual-domain attention network are substituted into the joint loss function to calculate the loss. The dual-domain attention network is then optimized and iterated based on the loss, and the iteration is repeated until the learning requirement is met.

7. The image blur removal method as described in claim 6, characterized in that, The iterative process until the learning requirement is met includes: The dataset is processed according to a preset method to obtain a test set, which includes multiple pairs of low-resolution blurred images. The low-resolution blurred image is input into the dual-domain attention network for processing to obtain a pseudo-high-resolution clear image; Determine whether the pseudo-high resolution clear image meets the preset termination condition; If the preset termination condition is not met, the learning requirements are obtained again to train the dual-domain attention network. If the preset termination condition is met, the dual-domain attention network in the current round is marked as having completed training, and can then be used to process the image to be processed.

8. An image optimization device, characterized in that, include: A construction module is used to construct a dual-domain attention network, which includes n encoders and decoders, where n is an integer greater than or equal to 1; the encoders and decoders are connected sequentially, the encoder scale decreases step by step, and the decoder scale increases step by step, and the dual-domain attention network has a U-shaped architecture; the encoders and decoders include convolutional layers and dual-domain attention residual blocks, and the convolutional layers are used to extract shallow features of the input feature image; The dual-domain attention residual block includes a spatial attention module and a frequency attention module. The spatial attention module is used to determine important regions within the image, and the frequency attention module is used to determine blurred regions within the image. The encoder extracts deep features of the input feature image based on the dual-domain attention residual block within the encoder. The deep features are extracted from the important regions and / or the blurred regions. The decoder converts the input deep features into a high-definition image and outputs it based on the dual-domain attention residual block within the decoder. The training module is used to design the joint loss function and create the dataset; and to train the dual-domain attention network based on the joint loss function and the dataset. The processing module is used to input the image to be processed into the trained dual-domain attention network to obtain an output image. The sharpness of the output image is higher than that of the image to be processed, and the resolution of the output image is higher than or equal to that of the image to be processed.

9. A computer device, characterized in that, Including processor and memory; The processor is used to execute a computer program stored in the memory to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images