A structure perception based image enhancement method and device

Abstract

The application relates to a structure perception-based image enhancement method and device. A low-quality image and a corresponding high-quality clear image are collected to form a data set; a structure perception Transformer network is constructed, including an U-shaped main network and a learnable Sobel convolution branch, the U-shaped main network comprising a plurality of structure guiding cascaded Transformer blocks and a structure guiding dynamic upsampling module; a joint loss function is designed to train the network; and a low-quality image to be processed is input into the network to generate an enhanced image. The learnable Sobel convolution branch extracts robust structure priori, the structure guiding cascaded attention mechanism and the structure guiding gated feedforward network realize deep fusion of global context and structure information, and the structure guiding dynamic upsampling module improves the recovery precision of structure regions. The structure integrity and the detail recovery quality of image enhancement are effectively improved, and clear and natural enhancement results are obtained.

Description

Technical Field

[0001] This application relates to the fields of digital image processing and computer vision technology, and in particular to an image enhancement method and device based on structure perception. Background Technology

[0002] Image enhancement technology is a key technology that processes degraded images to improve their visual quality, structural integrity, and information representation capabilities. This technology is widely used in various fields such as intelligent monitoring, autonomous driving, medical imaging, remote sensing imaging, underwater vision, and industrial inspection, and is of great significance for improving image visibility, enhancing detail, and improving the reliability of machine vision systems.

[0003] In recent years, deep learning-based image enhancement methods have made significant progress. In particular, the introduction of convolutional neural networks and visual Transformers has enabled excellent results in color restoration, texture reconstruction, and detail recovery. However, existing methods still have the following shortcomings: First, convolutional neural networks mainly rely on local convolution operations. Although they have good local feature extraction capabilities, their receptive field is limited, making it difficult to fully model the dependencies between distant pixels, thus limiting the utilization of global contextual information.

[0004] Second, although Transformer has strong global modeling capabilities, it often focuses more on high-level semantic information during the modeling process, and pays insufficient attention to low-level structural features such as edges and contours in the image, which can easily lead to insufficient restoration of structural details.

[0005] Third, in deep networks, as the feature layers deepen, structural information and edge textures often gradually weaken or even disappear, which in turn affects the structural integrity and detail realism of the enhancement results.

[0006] Fourth, traditional upsampling methods typically employ fixed strategies such as bilinear interpolation or pixel rearrangement, lacking the ability to adaptively reconstruct structural regions, and are prone to producing blurring, jagged edges, or artifacts in edge regions.

[0007] Therefore, there is an urgent need in related technologies for a way to solve the problem that existing image enhancement methods cannot simultaneously take into account global context modeling and structural detail preservation. Summary of the Invention

[0008] Therefore, it is necessary to provide a structure-aware image enhancement method and device that can solve the problem that existing image enhancement methods cannot simultaneously take into account global context modeling and structural detail preservation.

[0009] Firstly, this application provides a structure-aware image enhancement method. The method includes: A dataset is constructed by collecting low-quality images and their corresponding high-quality, clear images. A structure-aware Transformer network is constructed, which includes a U-shaped backbone network and a learnable Sobel convolutional branch for extracting structural features; the U-shaped backbone network contains multiple structure-guided cascaded Transformer blocks and a structure-guided dynamic upsampling module. Design a joint loss function and use the dataset to train the structure-aware Transformer network end-to-end. The joint loss function includes L1 loss, perceptual loss and image gradient loss. The low-quality image to be processed is input into the trained structure-aware Transformer network to generate an enhanced image.

[0010] Optionally, in one embodiment of this application, the learnable Sobel convolution branch for extracting structural features includes: The input image is convolved using convolution kernels initialized with Sobel operators in the horizontal and vertical directions, respectively, to obtain gradient response features in the corresponding directions; The gradient response features are input into the parameterized mapping unit, and the gradient responses in different directions are weighted and nonlinearly transformed by learnable parameters. Adaptive fusion of gradient responses in each direction is performed to generate structural features.

[0011] Optionally, in one embodiment of this application, the structure-guided cascaded Transformer block includes a channel-aware adaptive layer normalization module, a structure-guided cascaded attention module, a channel-aware adaptive layer normalization module, and a structure-guided gated feedforward network connected in sequence, with the output and input of each module added together through a residual connection.

[0012] Optionally, in one embodiment of this application, the channel-aware adaptive layer normalization module includes: Calculate the mean and standard deviation of the input features along the channel dimension to obtain statistical features; The statistical features are input into a lightweight multilayer perceptron for mapping, generating corresponding channel scaling factors and channel offset factors. The input features are normalized and channel-level modulated using the channel scaling factor and channel offset factor.

[0013] Optionally, in one embodiment of this application, the structure-guided cascaded attention module includes two stages of attention computation: In the first stage, deep convolution operations are performed on the input features and the initial query matrix, key matrix and value matrix are calculated. Self-attention calculation is performed to aggregate the global context and obtain the output features of the first stage. In the second stage, structural features from the learnable Sobel convolution branch are injected, and the output features of the first stage are modulated by structure-guided dynamic convolution. The key matrix and value matrix are obtained by depth convolution, and self-attention is calculated with the query matrix obtained by nonlinear transformation of the output features of the first stage. The calculation results are used to reconstruct features using convolutional layers and fused with the output features of the first stage through residual connections.

[0014] Optionally, in one embodiment of this application, the structure-guided gating feedforward network includes: The main features are passed through convolution, depthwise separable convolution, and GELU activation sequentially through the content branches to obtain the content features; Structural features are passed through gated branches, convolutional mapping, and Sigmoid activation to generate structure-aware gate weights with values ​​ranging from 0 to 1. The enhanced features are obtained by multiplying the gating weights element-wise with the content features; The enhanced features are added element-wise to the backbone features to obtain the final output features.

[0015] Optionally, in one embodiment of this application, the structure-guided dynamic upsampling module includes a structure-guided dynamic sampling branch and a pixel rearrangement branch, wherein: In the structure-guided dynamic sampling branch, the spatial offset is predicted using the upsampled structural features, and a sampling grid is constructed based on the spatial offset. The input features are then sampled using the grid to achieve adaptive upsampling. In the pixel rearrangement branch, the input features are upsampled through convolution mapping and pixel rearrangement operations; The outputs of the two branches are concatenated and fused along the channel dimension to obtain the final upsampling result.

[0016] Optionally, in one embodiment of this application, the joint loss function is composed of a weighted sum of L1 loss, perceptual loss, and image gradient loss, and the L1 loss is further used to supervise the training of the structural features predicted by the learnable Sobel convolutional module.

[0017] Secondly, this application also provides a structure-aware image enhancement device. The device includes a memory and a processor, the memory storing a computer program, and the processor executing the steps of the methods described in the various embodiments above.

[0018] Compared with existing technologies, the above-mentioned structure-aware image enhancement method and device have the following advantages: First, by introducing a learnable Sobel convolution branch, this invention can adaptively extract robust structural edge information from the input image, providing accurate and reliable structural priors for subsequent feature learning.

[0019] Second, the structure-guided cascaded attention mechanism proposed in this invention achieves deep fusion of structural and semantic information during global context modeling, effectively improving the recovery capability of edge regions and texture details.

[0020] Third, the structure-guided gating feedforward network proposed in this invention achieves adaptive enhancement of key structural regions through a structure-aware gating mechanism, thereby improving the selectivity and effectiveness of feature representation.

[0021] Fourth, the structure-guided dynamic upsampling module proposed in this invention can adaptively adjust the sampling position according to structural features, effectively reducing edge blurring and artifacts, and significantly improving the quality of high-resolution reconstruction.

[0022] Fifth, this invention combines the advantages of global modeling of Transformer with the ability of explicit structural modeling, enabling it to achieve superior visual quality, structural integrity, and detail restoration in image enhancement tasks, and has promising application prospects. Attached Figure Description

[0023] Figure 1 This is a flowchart illustrating a structure-aware image enhancement method in one embodiment; Figure 2 This is an overall architecture diagram of a structure-aware Transformer network in one embodiment; Figure 3 This is a schematic diagram of the structure of a learnable Sobel convolution module in one embodiment; Figure 4 This is a schematic diagram of the structure of a structure-guided cascaded Transformer block in one embodiment; Figure 5 This is a schematic diagram of the structure of the channel-aware adaptive layer normalization module in one embodiment; Figure 6 This is a schematic diagram of the structure of a structure-guided cascaded attention module in one embodiment; Figure 7 This is a schematic diagram of the structure of a structure-guided gated feedforward network module in one embodiment; Figure 8 This is a schematic diagram of the structure-guided dynamic upsampling module in one embodiment; Figure 9This is an internal structural diagram of a structure-aware image enhancement device in one embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] In one embodiment, such as Figure 1 As shown, a structure-aware image enhancement method is provided, including the following steps: S101: Collect low-quality images and their corresponding high-quality, clear images to form a dataset.

[0026] In this embodiment, firstly, a dataset is constructed by collecting low-quality images and their corresponding high-quality, clear images; the dataset is then divided into a training set, a validation set, and a test set. Specifically, the UIEB underwater image enhancement benchmark dataset is used, with 800 pairs of images in the dataset assigned to the training set and the remaining 90 pairs assigned to the test set.

[0027] S102: Construct a structure-aware Transformer network, which includes a U-shaped backbone network and a learnable Sobel convolutional branch for extracting structural features; the U-shaped backbone network contains multiple structure-guided cascaded Transformer blocks and a structure-guided dynamic upsampling module.

[0028] In this embodiment of the application, a construction is performed as follows: Figure 2 The Structure-Aware Transformer Network (SATNet) shown is used for image enhancement. The Structure-Aware Transformer Network mainly consists of a U-shaped backbone network and learnable Sobel Convolution (LSConv) branches for extracting structural features. The U-shaped backbone network contains multiple Structure-guided Cascaded Transformer Blocks (SGCTB) and Structure-guided Dynamic Upsample (SGDU) modules.

[0029] Specifically, Sobel convolution branches can be learned to extract robust structural representations, and the extracted structural features S representations are injected into each stage of the U-shaped backbone network to guide feature learning. The U-shaped backbone network includes an encoder, a bottleneck layer, and a decoder. Both the encoder and decoder are set as three-layer structures, and each layer of the backbone network is constructed by several structure-guided cascaded Transformer blocks. The input image X first passes through a convolutional mapping layer to extract shallow features and map them to a high-dimensional feature space, obtaining an initial feature representation F0. Subsequently, the initial features F0 are input to the encoder stage. The encoder consists of a three-level hierarchical structure, with each level containing an SGCTB and a downsampling operation. After each SGCTB level, the features, guided by the structural features S, complete local detail enhancement and global dependency modeling; then, downsampling gradually reduces the feature resolution, thereby obtaining a multi-scale semantic representation. The features output from each encoding stage are not only passed to the next layer but also retained through skip connections for use in the decoding stage. In the bottleneck layer, the lowest-resolution features are further processed by SGCTB for deep feature modeling. This stage focuses on capturing long-range dependencies and high-level semantic information in the image, while continuously enhancing the response to key edges and contour regions under the guidance of the structural features S. The decoder also employs a three-level hierarchical structure to progressively restore the spatial resolution of the features. Each decoder level first upsamples the input features through an SGDU module. This module uses the structural features S to predict spatial offsets and combines a dynamic sampling mechanism to achieve more accurate feature reconstruction, especially effectively recovering edge regions and detail information. Subsequently, the upsampled features are fused with the features of the corresponding layer of the encoder via skip connections in the channel dimension to fully integrate shallow detail information and deep semantic information. The fused features are then input into SGCTB for further optimization and reconstruction. After the final decoding stage, the features are mapped back to the image space through a tail convolutional layer to generate a residual image. Finally, this residual image is added pixel-by-pixel to the input image X to obtain the enhanced output image.

[0030] In one embodiment of this application, the learnable Sobel convolution branch for extracting structural features includes: S201: Perform convolution operations on the input image using convolution kernels initialized with Sobel operators in the horizontal and vertical directions respectively, to obtain gradient response features in the corresponding directions.

[0031] S202: Input the gradient response features into the parameterized mapping unit, and perform weighted modulation and nonlinear transformation on the gradient responses in different directions through learnable parameters.

[0032] S203: Adaptively fuse gradient responses in all directions to generate structural features.

[0033] In one embodiment of this application, such as Figure 3 As shown, the learnable Sobel convolution branch, given an input image X, first extracts gradient information in the horizontal and vertical directions through two parallel convolution branches. The two convolution kernels are initialized with the Sobel operator to enhance the module's prior perception of edge structures. After convolution, gradient response features Gx and Gy in the corresponding directions are obtained. Subsequently, the horizontal gradient response features Gx and Gy are input to the parameterized mapping unit. This unit uses learnable convolutional layers to adaptively weight and nonlinearly transform the gradient information in different directions, effectively suppressing noise interference and enhancing the response to weak edge regions and fine-grained structural information. Finally, the enhanced multi-directional gradient features are fused to obtain structural features S. This structural feature map has good edge continuity and structural consistency, and can more accurately represent the contour, boundary, and texture information in the image, providing reliable prior support for subsequent structure-guided feature learning.

[0034] In one embodiment of this application, the structure-guided cascaded Transformer block includes a channel-aware adaptive layer normalization module, a structure-guided cascaded attention module, a channel-aware adaptive layer normalization module, and a structure-guided gated feedforward network connected in sequence. The output and input of each module are added together through a residual connection.

[0035] In one embodiment of this application, such as Figure 4 As shown, the Structure-Guided Cascaded Transformer block jointly optimizes the global context and structural features through a cascaded attention mechanism. Specifically, the input features are processed sequentially through the Channel-aware Adaptive Layer Normalization (CALN) module and the Structure-Guided Cascaded Attention (SGCA) module, and the results are added to the residual connections. Subsequently, the features are processed again through the Channel-aware Adaptive Layer Normalization module and the Structure-guided Gated Feed-forward Network (SGFN), and the results are added to the residual connections.

[0036] In one embodiment of this application, the channel-aware adaptive layer normalization module includes: S301: Calculate the mean and standard deviation of the input features along the channel dimension to obtain statistical features.

[0037] S302: Input the statistical features into a lightweight multilayer perceptron for mapping to generate corresponding channel scaling factors and channel offset factors.

[0038] S303: Perform normalization processing and channel-level modulation operation on the input features using the channel scaling factor and channel offset factor.

[0039] In one embodiment of this application, such as Figure 5 As shown, the channel-aware adaptive layer normalization module is used to perform adaptive normalization and modulation of the input features along the channel dimension. Specifically, given the input features, the mean and standard deviation along the channel dimension are first calculated to obtain statistical features. Then, the statistical features are input into a lightweight multilayer perceptron for mapping to generate corresponding channel scaling factors and channel offset factors. Finally, the input features are normalized and modulated using the channel scaling factors and channel offset factors.

[0040] In one embodiment of this application, the structure-guided cascaded attention module includes two stages of attention computation: S401: In the first stage, deep convolution operation is performed on the input features and the initial query matrix, key matrix and value matrix are calculated. Self-attention calculation is performed to aggregate the global context and obtain the output features of the first stage.

[0041] S402: In the second stage, structural features from the learnable Sobel convolution branch are injected, and the output features of the first stage are modulated by structure-guided dynamic convolution. The key matrix and value matrix are obtained by depth convolution. Self-attention calculation is performed with the query matrix obtained by nonlinear transformation of the output features of the first stage. The calculation results are used to reconstruct features using convolutional layers and fused with the output features of the first stage through residual connections.

[0042] In one embodiment of this application, such as Figure 6As shown, the structure-guided cascaded attention mechanism comprises two stages of attention computation. In the first stage, deep convolution is performed on the input features to compute the initial query matrix, key matrix, and value matrix. Standard self-attention computation is then performed to aggregate the global context, yielding the output feature F1 of the first stage features. In the second stage, structural features S from the learnable Sobel convolution module are injected. The output feature F1 of the first stage is modulated through structure-guided dynamic convolution, and the key and value matrices are obtained through deep convolution. Self-attention computation is then performed with the query matrix obtained by nonlinear transformation of F1 to further integrate structural information with global context information and enhance the feature's responsiveness to edge regions and texture details. Subsequently, the attention computation results of the second stage are reconstructed using convolutional layers and fused with the output feature F1 of the first stage through residual connections to obtain the final attention output feature. This approach improves the accuracy and stability of structural information representation while maintaining global dependency modeling capabilities.

[0043] In one embodiment of this application, the structure-guided gating feedforward network includes: S501: The main features are passed through convolution, depthwise separable convolution and GELU activation in sequence through the content branches to obtain the content features.

[0044] S502: Structural features are generated by gating branches through convolution mapping and Sigmoid activation, producing structure-aware gating weights with values ​​ranging from 0 to 1.

[0045] S503: Multiply the gating weights element-wise with the content features to obtain the enhanced features.

[0046] S504: Add the enhanced features to the backbone features element by element to obtain the final output features.

[0047] In one embodiment of this application, such as Figure 7As shown, a structure-guided gated feedforward network is used to introduce a structure-guided mechanism during the feedforward feature transformation process to achieve feature selection and enhancement. Specifically, the backbone feature Fx is sequentially passed through convolution, depthwise separable convolution, and GELU activation via the content branch to extract rich local details and contextual information, resulting in content feature Fc. Simultaneously, the structural feature S is mapped through convolution via the gated branch and activated using Sigmoid to generate structure-aware gate weights Gs with values ​​ranging from [0,1], representing the importance of each spatial location. Subsequently, the gate weights Gs are multiplied element-wise with the content feature Fc to achieve adaptive filtering and enhancement of the content feature, thereby highlighting structurally relevant regions and suppressing irrelevant or redundant information, resulting in enhanced feature F'. Finally, the enhanced feature F' is added element-wise to the original input feature Fx to form the final output feature Fy. This process guides content feature selection through structural information, achieving an effective unification of structure preservation and feature enhancement.

[0048] In one embodiment of this application, the structure-guided dynamic upsampling module includes a structure-guided dynamic sampling branch and a pixel rearrangement branch, wherein: S601: In the structure-guided dynamic sampling branch, the spatial offset is predicted using the upsampled structural features. Based on the spatial offset, a sampling grid is constructed, and the input features are sampled using the grid to achieve adaptive upsampling.

[0049] S602: In the pixel rearrangement branch, the input features are upsampled through convolution mapping and pixel rearrangement operations.

[0050] S603: The outputs of the two branches are concatenated and fused along the channel dimension to obtain the final upsampling result.

[0051] In one embodiment of this application, such as Figure 8 As shown, the structure-guided dynamic upsampling module is used to improve the recovery accuracy of structural regions during feature reconstruction. Specifically, the module includes a structure-guided dynamic sampling branch and a pixel rearrangement branch. In the structure-guided dynamic sampling branch, the spatial offset is predicted using the upsampled structural features, and a sampling grid is constructed based on the spatial offset to perform grid sampling on the input features to achieve adaptive upsampling. In the pixel rearrangement branch, the input features are upsampled through convolution mapping and pixel rearrangement operations. Finally, the outputs of the two branches are concatenated and fused in the channel dimension to obtain the final upsampling result.

[0052] S103: Design a joint loss function and use the dataset to train the structure-aware Transformer network end-to-end. The joint loss function includes L1 loss, perceptual loss, and image gradient loss.

[0053] In one embodiment of this application, the joint loss function is composed of a weighted sum of L1 loss, perceptual loss and image gradient loss, and the L1 loss is also used for supervised training of the structural features predicted by the learnable Sobel convolutional module.

[0054] In one embodiment of this application, the joint loss function mainly consists of three parts: L1 loss, perceptual loss, and image gradient loss, as shown in Equation (1). The L1 loss measures the absolute difference between the predicted and actual values. It takes the absolute value of the error for each pixel and then calculates the average of these absolute errors, as shown in Equation (2). The perceptual loss is based on the intermediate layer features of the pre-trained VGG network and measures the difference between the generated image and the target image in the feature space, rather than the pixel-level difference. It emphasizes the high-level semantics and details of the image, making the generated image visually closer to the real image, as shown in Equation (3). The image gradient loss constrains the consistency of the generated image and the target image in terms of structural and edge information. By calculating the second-order gradient and measuring the difference between the two in the gradient space, it enhances edge details and reduces over-smoothing, as shown in Equation (4).

[0055] (1) (2) (3) (4) in, For the joint loss function, For L1 loss, In order to perceive loss, Image gradient loss; This is a VGG16 network pre-trained on the ImageNet dataset; C, H, and W represent the number of feature channels, height, and width of the feature map, respectively. For reconstructing the image, I is the truth label; and These are the pixel values ​​of the predicted image and the actual ground truth, respectively. and These are the values ​​of the corresponding pixels in the pre-trained VGG16 network for the predicted image and the ground truth, respectively. Represents the discrete Laplace operator; , as well as These represent the weights of L1 loss, perceptual loss, and image gradient loss in the joint loss function, respectively. Furthermore, the L1 loss is additionally used to supervise the training of the structural features S predicted by the learnable Sobel convolutional modules.

[0056] S104: Input the low-quality image to be processed into the trained structure-aware Transformer network to generate the enhanced image.

[0057] In one embodiment of this application, the performance of the trained structure-aware Transformer network is tested using a test set. Specifically, training and testing are performed on the Underwater Image Enhancement Benchmark (UIEB) dataset, with 800 images in the training set and 90 images in the test set. PSNR and SSIM are used as objective evaluation metrics. The test results show that the proposed method achieves enhancement performance of 25.926 dB PSNR and 0.931 SSIM on the test set, indicating that the proposed method can effectively improve the overall quality of underwater images. The enhancement results show good performance in terms of structural feature integrity, and the structural information such as edge contours and texture details of the image are accurately restored and effectively preserved.

[0058] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0059] In one embodiment, a structure-aware image enhancement device is provided. This device can be a terminal, and its internal structure diagram can be as follows: Figure 9As shown, the device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, it implements a structure-aware image enhancement method. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the device casing, or an external keyboard, touchpad, or mouse.

[0060] Those skilled in the art will understand that Figure 9 The structure shown is merely a block diagram of a portion of the structure related to the solution of this application, and does not constitute a limitation on a structure-aware image enhancement device on which the solution of this application is applied. A specific device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0061] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0062] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0063] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0064] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A structure-aware image enhancement method, characterized in that, The method includes: A dataset is constructed by collecting low-quality images and their corresponding high-quality, clear images. A structure-aware Transformer network is constructed, which includes a U-shaped backbone network and a learnable Sobel convolutional branch for extracting structural features; the U-shaped backbone network contains multiple structure-guided cascaded Transformer blocks and a structure-guided dynamic upsampling module. Design a joint loss function and use the dataset to train the structure-aware Transformer network end-to-end. The joint loss function includes L1 loss, perceptual loss and image gradient loss. The low-quality image to be processed is input into the trained structure-aware Transformer network to generate an enhanced image.

2. The structure-aware image enhancement method according to claim 1, characterized in that, The learnable Sobel convolution branch used to extract structural features includes: The input image is convolved using convolution kernels initialized with Sobel operators in the horizontal and vertical directions, respectively, to obtain gradient response features in the corresponding directions; The gradient response features are input into the parameterized mapping unit, and the gradient responses in different directions are weighted and nonlinearly transformed by learnable parameters. Adaptive fusion of gradient responses in each direction is performed to generate structural features.

3. The structure-aware image enhancement method according to claim 1, characterized in that, The structure-guided cascaded Transformer block includes a channel-aware adaptive layer normalization module, a structure-guided cascaded attention module, a channel-aware adaptive layer normalization module, and a structure-guided gated feedforward network connected in sequence. The output and input of each module are added together through residual connections.

4. The structure-aware image enhancement method according to claim 3, characterized in that, The channel-aware adaptive layer normalization module includes: Calculate the mean and standard deviation of the input features along the channel dimension to obtain statistical features; The statistical features are input into a lightweight multilayer perceptron for mapping, generating corresponding channel scaling factors and channel offset factors. The input features are normalized and channel-level modulated using the channel scaling factor and channel offset factor.

5. The structure-aware image enhancement method according to claim 3, characterized in that, The structure-guided cascaded attention module includes two stages of attention computation: In the first stage, deep convolution operations are performed on the input features and the initial query matrix, key matrix and value matrix are calculated. Self-attention calculation is performed to aggregate the global context and obtain the output features of the first stage. In the second stage, structural features from the learnable Sobel convolution branch are injected, and the output features of the first stage are modulated by structure-guided dynamic convolution. The key matrix and value matrix are obtained by depth convolution, and self-attention is calculated with the query matrix obtained by nonlinear transformation of the output features of the first stage. The calculation results are used to reconstruct features using convolutional layers and fused with the output features of the first stage through residual connections.

6. The structure-aware image enhancement method according to claim 3, characterized in that, The structure-guided gated feedforward network includes: The main features are passed through convolution, depthwise separable convolution, and GELU activation sequentially through the content branches to obtain the content features; Structural features are passed through gated branches, convolutional mapping, and Sigmoid activation to generate structure-aware gate weights with values ​​ranging from 0 to 1. The enhanced features are obtained by multiplying the gating weights element-wise with the content features; The enhanced features are added element-wise to the backbone features to obtain the final output features.

7. The structure-aware image enhancement method according to claim 1, characterized in that, The structure-guided dynamic upsampling module includes a structure-guided dynamic sampling branch and a pixel rearrangement branch, wherein: In the structure-guided dynamic sampling branch, the spatial offset is predicted using the upsampled structural features, and a sampling grid is constructed based on the spatial offset. The input features are then sampled using the grid to achieve adaptive upsampling. In the pixel rearrangement branch, the input features are upsampled through convolution mapping and pixel rearrangement operations; The outputs of the two branches are concatenated and fused along the channel dimension to obtain the final upsampling result.

8. The structure-aware image enhancement method according to claim 1, characterized in that, The joint loss function consists of a weighted sum of L1 loss, perceptual loss, and image gradient loss, and the L1 loss is also used to supervise the training of the structural features predicted by the learnable Sobel convolutional module.

9. A structure-aware image enhancement device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images